How to add a timestamp to bash script log?

Question

I have a constantly running script that I output to a log file:

script.sh >> /var/log/logfile

I'd like to add a timestamp before each line that is appended to the log. Like:

Sat Sep 10 21:33:06 UTC 2011 The server has booted up.  Hmmph.

Is there any jujitsu I can use?

Solution 1

You can pipe the script's output through a loop that prefixes the current date and time:

./script.sh | while IFS= read -r line; do printf '%s %s\n' "$(date)" "$line"; done >>/var/log/logfile

If you'll be using this a lot, it's easy to make a bash function to handle the loop:

adddate() {
    while IFS= read -r line; do
        printf '%s %s\n' "$(date)" "$line";
    done
}

./thisscript.sh | adddate >>/var/log/logfile
./thatscript.sh | adddate >>/var/log/logfile
./theotherscript.sh | adddate >>/var/log/logfile

Solution 2

See ts from the Ubuntu moreutils package:

command | ts

Or, if $command does automatic buffering (requires expect-dev package):

unbuffer command | ts

Solution 3

The date command will provide that information

date -u
Sat Sep 10 22:39:24 UTC 2011

so you can

echo $(date -u) "Some message or other"

is that what you wanted ?

Solution 4

Make a config.sh file

#!/usr/bin/env bash
LOGFILE="/path/to/log.log"
TIMESTAMP=`date "+%Y-%m-%d %H:%M:%S"`

When you need to send to log file use

#!/usr/bin/env bash
source /path/to/config.sh

echo "$TIMESTAMP Say what you are doing" >> $LOGFILE

do_what_you_want >> $LOGFILE

Log file will looks like

2013-02-03 18:22:30 Say what you are doing

So it will be easy to sort by date

Solution 5

You can simply echo the command outputs to the logfile. ie,

echo "`date -u` `./script.sh`" >> /var/log/logfile

It really works :)

Example:

[sparx@E1]$ ./script.sh 
Hello Worldy
[sparx@E1]$ echo "`date -u` `./script.sh`" >> logfile.txt
[sparx@E1]$ cat logfile.txt 
Mon Sep 12 20:18:28 UTC 2011 Hello Worldy
[sparx@E1]$ 

View more solutions

Comments

[Zoredache]

See this questin. serverfault.com/questions/80749/…. A couple answer would apply here.

[User]

For an awk/gawk solution see: stackoverflow.com/questions/21564/…

[codeforester]

Here is a comprehensive implementation of logging for bash: github.com/codeforester/base/blob/master/lib/stdlib.sh

[Stéphane Gourichon]

Best local answer is @ChuckCottrill's serverfault.com/a/835534/137665 . script.sh | gawk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }' >> /var/log/logfile

[Antonius Bloch]

Using the date command was kind of what I had in mind, but I can't really add that to the script itself, so what I'm looking for is a way to change this line: "script.sh >> /var/log/logfile" to append the date.

[user9517]

In that case redirect the output of your script to a named pipe and have a daemon listening for output which takes the script output and adds a date before writing it to a log file. You can probably modify the script I wrote here to do this. I would do it because it interests me but it's late in the UK and I have an early start tomorrow.

[SparX]

What are you getting when you execute the command ?

[cjc]

My god, people, everyone is doing back tick replacement, named pipes, etc. Just enclose both the date command and the script in parens! The guy who has the function has a legit case if there's multi-line output and the log needs to look pretty with the date on every line, but most of these solutions are overkill that doesn't use shell semantics.

[Nils]

What is "IFS= " for?

[Gordon Davisson]

@Nils it's a trick to prevent read from trimming whitespace at the beginning and and of the line. It sets IFS (bash's Internal Field Separator, basically a list of whitespace characters) to empty for the read command.

[Nils]

...and -r ignores the escape-character "\". This should really work in all cases - great pice of scripting.

[Gordon Davisson]

@Nils it's not completely bulletproof, since some implementations of echo interpret escape sequences. If you really want it not to mess with the content (other than adding dates), replace the echo command with printf "%s %s\n" "$(date)" "$line"

[Dave Forgac]

That will only add the timestamp once per execution of script.sh. The OP needs a timestamp per line.

[User]

although this doesn't answer the OP question I still found it useful info.

[stackersunited]

That puts a timestamp before the entire output of ''./script.sh'', not before each line.

[stackersunited]

Your ''config.sh'' will run ''date'' exactly once, on ''source .../config.sh''.

[Sanjay Yadav]

@ shazbot:Thanks for editing, that was a typo error, i didnt notice.

[Univers3]

This is so cool!

[Pablo A]

You might be interested on a ISO-8601 compliant date/timestamp: date -u +"%Y-%m-%dT%H:%M:%SZ" or maybe more pretty date +"%Y-%m-%d %T".

[Michael Schaefers]

while this script works as expected, it spawns a new process (execute date) for each log line, which can be a massive drawback depending on your machine and the amount of logs. I would rather suggest to use ts if available, see the answer from @willem

[Evan Benn]

this is because it is bash evaluating "s/^/$(date -R) /" and running date once before sed. Sed is passed a static string.

[Evan Benn]

as for the other answer, this is just bash running date once. sed is not updating the time per line.

[Jason Harrison]

your first date command should use single quotes, not double quotes.

[ND Geek]

@PabloBianchi date [-u] -Iseconds or date [-u] --iso-8601=seconds will accomplish the same thing without manually recreating the format. Per this question/answer, this has been a documented option since v8.15 of coreutils. It doesn't appear to use a 'Z' for UTC, however, which is mildly annoying IMO.

[akhan]

Plain bash: yes | head -5000000 | while read line; do echo $((SECONDS)); done | uniq -c which is much slower than gawk. ts utility has similar performance as bash loop.

[akhan]

Perl: yes |head -5000000 |perl -ne 'print localtime."\t".$_' |uniq -c which is slightly slower than awk.

[Stéphane Gourichon]

This answer deserves more attention. Even though using external program date might now be such a performance problem on nowadays machines, when a tool has everything inside, better not complicate with more tools. Plus this answers provide a simple and locally reproducible performance measurement (on my machine I get 700-1000 per second and 150k-170k per second). More answers should be of such quality.

[kdb]

@StéphaneGourichon There are actually cases where it can make a huge performance difference still. E.g. I use this construct for adding timing information to stdout with someprogram | while true; do date; done | uniq -c . If a program produces hundreds of lines per second, calling date for each slows the performance down, while using the builtin does not. Though curiously, right now even the builtin produces only about 4000 lines per uniq -c for me... Could be that I tested on Linux back then.

[Stéphane Gourichon]

yes we agree, no repeated external program will be faster. When performance is really important, no repeated bash construct will be even faster. GNU awk based solution may provide the best, see @ChuckCottrill's answer serverfault.com/a/835534/137665 On my machine I get 3.5x speed gain by replacing yes |head -5000000 | while true; do printf '%(%F %T)T\n'; done | uniq -c with yes |head -5000000 | gawk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }' >> /var/log/logfile .

[OneCricketeer]

You're missing a close paren

[mcdave]

FYI, unbuffer is in the "expect" package on Arch Linux.

[Christian]

This gives a weird result on subcommands with the time bash-3.2$ PS4='\t ' bash-3.2$ echo $(echo hi) hi bash-3.2$ set -x bash-3.2$ echo $(echo hi) 113:18:04 echo hi 13:18:04 echo hi hi

[Sohail Si]

How to preserve color?

[Gordon Davisson]

@SohailSi This shouldn't remove color (i.e. any color-setting terminal control escape sequences), but many commands only output color-setting escape sequences when their output is going directly to a terminal. You may need to tell the command producing the initial output to include color sequences even though its output is going to a pipe (e.g. grep --color=always).

[Josh Williams]

Thanks @OneCricketeer missed that

[Admin]

This is great, but it is not available on AL2.