How to replace epoch timestamps in a file with other formats?
Solution 1
Assuming consistent file format, with bash
you can read the file line by line, test if it's in given format and then do the conversion:
while IFS= read -r i; do [[ $i =~ ^#([0-9]{10})$ ]] && \
date -d@"${BASH_REMATCH[1]}"; done <file.txt
BASH_REMATCH
is an array whose first element is the first captured group in Regex matching, =~
, in this case the epoch.
If you want to keep the file structure:
while IFS= read -r i; do if [[ $i =~ ^#([0-9]{10})$ ]]; then printf '#%s\n' \
"$(date -d@"${BASH_REMATCH[1]}")"; else printf '%s\n' "$i"; fi; done <file.txt
this will output the modified contents to STDOUT, to save it in a file e.g. out.txt
:
while ...; do ...; done >out.txt
Now if you wish, you can replace the original file:
mv out.txt file.txt
Example:
$ cat file.txt
#1472047795
ll /data/holding/email
#1472047906
cat /etc/rsyslog.conf
#1472048038
ll /data/holding/web
$ while IFS= read -r i; do [[ $i =~ ^#([0-9]{10})$ ]] && date -d@"${BASH_REMATCH[1]}"; done <file.txt
Wed Aug 24 20:09:55 BDT 2016
Wed Aug 24 20:11:46 BDT 2016
Wed Aug 24 20:13:58 BDT 2016
$ while IFS= read -r i; do if [[ $i =~ ^#([0-9]{10})$ ]]; then printf '#%s\n' "$(date -d@"${BASH_REMATCH[1]}")"; else printf '%s\n' "$i"; fi; done <file.txt
#Wed Aug 24 20:09:55 BDT 2016
ll /data/holding/email
#Wed Aug 24 20:11:46 BDT 2016
cat /etc/rsyslog.conf
#Wed Aug 24 20:13:58 BDT 2016
ll /data/holding/web
Solution 2
While it's possible with GNU sed
with things like:
sed -E 's/^#([0-9]+).*$/date -d @\1/e'
That would be terribly inefficient (and is easy to introduce arbitrary command injection vulnerabilities1) as that would mean running one shell and one date
command for each #xxxx
line, virtually as bad as a shell while read
loop. Here, it would be better to use things like perl
or gawk
, that is text processing utilities that have date conversion capabilities built-in:
perl -MPOSIX -pe 's/^#(\d+).*/ctime $1/se'
Or:
gawk '/^#/{$0 = strftime("%c", substr($0, 2))};1'
1 If we had written ^#([0-9]).*
instead of ^#([0-9]).*$
(as I did in an earlier version of this answer), then in multi-byte locales like UTF-8 ones (the norm nowadays), with an input like #1472047795<0x80>;reboot
, where that <0x80>
is the byte value 0x80 which does not form a valid character, that s
command would have ended up running date -d@1472047795<0x80>; reboot
for instance. While with the extra $
, those lines would not be substituted. An alternative approach would be: s/^#([0-9])/date -d @\1 #/e
, that is leave the part after the #xxx
date as a shell comment
Solution 3
All the other answers spawn a new date
process for every epoch date that needs to be converted. This could potentially add performance overhead if your input is large.
However GNU date has a handy -f
option that allows a single process instance of date
to continuously read input dates without the need for a new fork. So we can use sed
, paste
and date
in this manner such that each one only gets spawned once (2x for sed
) regardless of how large the input is:
$ paste -d '\n' <( sed '2~2d;y/#/@/' epoch.txt | date -f - ) <( sed '1~2d' epoch.txt )
Wed Aug 24 07:09:55 PDT 2016
ll /data/holding/email
Wed Aug 24 07:11:46 PDT 2016
cat /etc/rsyslog.conf
Wed Aug 24 07:13:58 PDT 2016
ll /data/holding/web
$
- The two
sed
commands respectively basically delete even and odd lines of the input; the first one also replaces#
with@
to give the correct epoch timestamp format. - The first
sed
output is then piped todate -f
which does the required date conversion, for every line of input that it receives. - These two streams are then interlaced into the single required output using
paste
. The<( )
constructs are bash process substitutions that effectively trick paste into thinking it is reading from given filenames when it is in fact reading the output piped from the command inside.-d '\n'
tellspaste
to separate odd and even output lines with a newline. You could change (or remove) this if for example you want the timestamp on the same line as the other text.
Note that there are several GNUisms and Bashisms in this command. This is not Posix-compliant and should not be expected to be portable outside of the GNU/Linux world. For example date -f
does something else on OSXes BSD date
variant.
Solution 4
Assuming the date format you have in your post is what you want, the following regex should fit your needs.
sed -E 's/\#(1[0-9]{9})(.*)/echo \1 $(date -d @\1)/e' log.file
Be mindful of the fact this will only replace one epoch per line.
Related videos on Youtube
machinist
Updated on September 18, 2022Comments
-
machinist over 1 year
I have a file that contains epoch dates which I need converting to human-readable. I already know how to do the date conversion, eg:
[server01 ~]$ date -d@1472200700 Fri 26 Aug 09:38:20 BST 2016
..but I'm struggling to figure out how to get
sed
to walk through the file and convert all the entries. The file format looks like this:#1472047795 ll /data/holding/email #1472047906 cat /etc/rsyslog.conf #1472048038 ll /data/holding/web
-
Toby Speight over 7 yearsFor future reference (assuming this is a Bash history file; it looks like one), look to the
HISTTIMEFORMAT
shell variable to control the format at time of writing. -
dave_thompson_085 over 7 years@Toby the value of HISTTIMEFORMAT is used when displaying (to stdout), but only its status (set to anything even null vs unset) matters when writing HISTFILE.
-
Toby Speight over 7 yearsThanks @dave, I didn't know that (not being a user of history times myself).
-
Gert van den Berg over 7 years
date -d
is not portable to say Solaris... I'm assuming this is on a system with mostly GNU tools? (GNU AWK / Perl tend to be the more portable methods to deal with date conversions).gawk '{ if ($0 ~ /^#[0-9]*$/) {print strftime("%c",substr($0,2)); } else {print} }' < file
(strftime
seems non-portable...)
-
-
machinist over 7 yearsI'm getting the following error with that command:
sed: -e expression #1, char 48: invalid reference \3 on 's' command's RHS
-
machinist over 7 yearsNice....that prints the converted date to screen, now how do I get that command to replace the entries in the file?
-
heemayl over 7 years@machinist Check my edits..
-
Hatclock over 7 yearsMy mistake, edited the post.
-
chepner over 7 yearsIf you are using a recent version of
bash
,printf
can do the conversion itself:printf '#%(%F %H)T\n' "${BASH_REMATCH[1]}"
. -
Gert van den Berg over 7 years
date -d
(from the question) is also non-portable... (On FreeBSD it will try to mess with DST settings, on Solaris it will give an error...) The question does not specify an OS though... -
Digital Trauma over 7 years@GertvandenBerg yes, this is addressed in the last paragraph of this answer.
-
Gert van den Berg over 7 yearsI mean that the asker's sample also has portability issues... (They should probably have tagged an OS...)
-
Alex Harvey almost 7 yearsThe perl command seems to add a new line after ctime $1 and I can't find any way to remove it.
-
Stéphane Chazelas almost 7 years@Alex. Right. See edit. Adding the
s
flag makes so that.*
also includes the newline on input. You can also usestrftime "%c", localtime $1
. -
Alex Harvey almost 7 years@StéphaneChazelas thanks so much. It's a great answer.
-
zylstra over 2 yearsWhere does the perl command reference the file it is using as input and output?
-
Stéphane Chazelas over 2 years@zylstra,
-p
withperl
enables ased
-like mode, so input comes from files given as arguments (though see also Security implications of running perl -ne '...' *) or stdin if no argument and output goes to stdout like insed
. Seeperldoc perlrun
for details.