How to chain the commands 'date -d @xxxxxx' and 'find ./'?

Question

I have directories whose names are timestamps, given in milliseconds since 1970-01-01:

1439715011728
1439793321429
1439879712214
.
.

And I need an output like:

1442039711    Sat Sep 12 08:35:11 CEST 2015
1442134211    Sun Sep 13 10:50:11 CEST 2015
1442212521    Mon Sep 14 08:35:21 CEST 2015
.
.

I can list all directories by command:

find ./ -type d | cut -c 3-12

But I cannot put the output to the next command: date -d @xxxxxx and manipulate the output.

How can I do this?

Solution 1

I'd avoid running several commands per file in a loop. Since you're already using GNUisms:

find . ! -name . -prune -type d |
  awk '{t = substr($0, 3, 10); print t, strftime("%a %b %d %T %Z %Y", t)}'

Which just runs two commands. strftime() is GNU-specific, like date -d.

Solution 2

You are on the right track (for a simpler solution, running only 2 or 3 commands, see below). You should use * instead of ./ to get rid of the current directory¹ and this simplifies cutting of the milliseconds somewhat, then just pipe the result into GNU parallel or xargs²:

find * -type d | cut -c 1-10 | parallel date --date=@{} +%c

to get

Sat 12 Sep 2015 08:35:11 CEST
Sun 13 Sep 2015 10:50:11 CEST
Mon 14 Sep 2015 08:35:21 CEST

and to add the seconds offset before that as your example indicates:

find * -type d | cut -c 1-10 | parallel 'echo "{} "  $(date --date=@{} +%c)'

or:

find * -type d | cut -c 1-10 | xargs -I{} bash -c 'echo "{} "  $(date --date=@{} +%c)'

to get:

1442039711  Sat 12 Sep 2015 08:35:11 CEST
1442134211  Sun 13 Sep 2015 10:50:11 CEST
1442212521  Mon 14 Sep 2015 08:35:21 CEST

However it is simpler to do³:

find * -type d -printf "@%.10f\n" | date -f - +'%s  %c'

which gets you the same requested output once more.

The disadvantage of using * is that you are limited by your commandline for its expansion, the advantage however is that you get your directories sorted by timestamp value. If the number of directories is a problem use -mindepth 1, but lose the ordering:

find ./ -mindepth 1 -type d -printf "@%.10f\n" | date -f - +'%s  %c'

and insert sort if needed:

find ./ -mindepth 1 -type d -printf "@%.10f\n" | sort | date -f - +'%s  %c'

¹ _{This assumes there are no nested subdirectories, as seems to be the case from your example. You can also use ./ -mindepth 1 instead of *}
²_{You can replace parallel with xargs -I{} here as @hobbs and @don_crissti suggested, it just more verbose.} ³_{based on Gilles' answer to use dates file reading capabilities}

Solution 3

You already have:

find ./ -type d | cut -c 3-12

which presumably gets you the timestamps in epoch format. Now add a while loop:

find ./ -type d | cut -c 3-12 | while read datestamp
do
    printf %s "$datestamp"
    date -d "@$datestamp"
done

Note though that in some shells, that syntax gets the while loop in a subshell, which means that if you try to set a variable there, it won't be visible once you've left the loop. To fix that, you need to turn things on their head slightly:

while read datestamp
do
    printf %s "$datestamp"
    date -d "@$datestamp"
done < <(find ./ -type d | cut -c 3-12)

which puts the find in the subshell, and keeps the while loop in the main shell. That syntax (AT&T ksh, zsh and bash specific) is only needed if you're looking to reuse a result from inside the loop, though.

Solution 4

If you have GNU date, it can convert dates read from an input file. You just need to massage the timestamps a little so that it can recognize them. The input syntax for a timestamp based on the Unix epoch is @ followed by the number of seconds, which can contain a decimal point.

find ./ -type d ! -name '*[!0-9]*' |
sed -e 's~.*/~@~' -e 's~[0-9][0-9][0-9]$~.&~' |
date -f - +'%s  %c'

Solution 5

I would do it perlishly - feed in a list of timestamps:

#!/usr/bin/perl
use strict;
use warnings;
use Time::Piece;

while ( my $ts = <DATA> ) { 
   chomp ( $ts );
   my $t = Time::Piece->new();
   print $t->epoch, " ", $t,"\n";
}

__DATA__
1442039711  
1442134211  
1442212521

This outputs:

1442039711 Sat Sep 12 07:35:11 2015
1442134211 Sun Sep 13 09:50:11 2015
1442212521 Mon Sep 14 07:35:21 2015

If you want a specific output format, you can use strftime e.g.:

print $t->epoch, " ", $t->strftime("%Y-%m-%d %H:%M:%S"),"\n";

Which to turn this into a one liner in your pipe:

 perl -MTime::Piece -nle '$t=Time::Piece->new($_); print $t->epoch, "  ", $t, "\n";'

But I'd probably suggest instead looking at using the File::Find module and doing the whole thing in perl instead. If you give an example of your directory structure before cutting it, I'll give you an example. But it would be something like:

#!/usr/bin/env perl

use strict;
use warnings;
use Time::Piece;
use File::Find; 

sub print_timestamp_if_dir {
   #skip if 'current' item is not a directory. 
   next unless -d; 
   #extract timestamp (replicating your cut command - I think?)
   my ( $timestamp ) = m/.{3}(\d{9})/; #like cut -c 3-12;

   #parse date
   my $t = Time::Piece->new($timestamp);
   #print file full path, epoch time and formatted time; 
   print $File::Find::name, " ", $t->epoch, " ", $t->strftime("%Y-%m-%d %H:%M:%S"),"\n";
}

find ( \&print_timestamp_if_dir, "." ); 

View more solutions

Comments

[Gilles 'SO- stop being evil']

@Sobrique Clearly milliseconds since the epoch.

[Wouter Verhelst]

regardless, saying that it is bash-specific isn't correct :)

[Stéphane Chazelas]

Actually, as you had initially written it, done <(find) instead of done < <(find), it was correct for yash (where <(...) is process redirection, not process substitution), so my edit was a bit cavalier as it could have been the shell you meant that for.

[Stéphane Chazelas]

Note that parallel is written in perl. This seems overkill considering that perl has a strftime() operator. Like perl -MPOSIX -lpe '$_.=strftime(" %c", localtime substr $_, 2, 10)'

[hobbs]

Or xargs if you don't have parallel, which many people probably don't.

[don_crissti]

It does: find ./ -type d | cut -c 3-12 | xargs -I{} date --d @{} +'%Y-%m-%d'

[hobbs]

@Anthon it does if you use the -I option.

[Ole Tange]

1. It is shorter. 2. You do not need to learn Perl.

[Anthon]

@don_crissti thanks. But your --d option for date has a dash too many I think.

[Anthon]

+1 for using dates file reading. This will give a date: invalid date ‘@’ because of the translation of the current directory (./). And since you can throw away the milliseconds you can simplify the second sed edit to just drop the last 3 characters. Or remove all of that and use find * -type d -printf "@%.10f" | date ...

[Anthon]

This doesn't cut the milliseconds of the directory names but displays the full 13 characters instead of the requested first 10

[Stéphane Chazelas]

@Anthon, ah yes, missed that requirement. Should be OK now.

[Stéphane Chazelas]

@Anthon, GNU long options can be abbreviated as long as they're not ambiguous. --d or --da would work with current versions of GNU date, but it could stop working the day date introduces a --dalek option (for dates in the Dalek calendar).

[Anthon]

@StéphaneChazelas Ok. And --dalek would give different results depending on them having rewritten time or not, would be useful for the Doctor.

[Stéphane Chazelas]

It's 27% shorter, but it's several orders of magnitude less efficient (about 800 times as slow in the test I made; consider it needs to spawn a shell (your shell, not /bin/sh) and a date command for each line) and unfriendly to the system as it burdens all CPUs at once. And you still need to learn parallel. IMO, parallel is a great tool to parallelize CPU intensive tasks, but not really appropriate for this kind of task here.

[Stéphane Chazelas]

There are plenty of contexts where efficiency is not a concern, so it's still an acceptable solution, but it's still worth mentioning the performance issue, especially when considering that parallel usually rhymes with high performance in people's mind.

[Sobrique]

Why not do it all in python? Rather than chaining a bunch of pipes?

[lukaz]

It would make better sense. I agree.