Universal non-bash `time` benchmark alternative?

Question

For comparing run times of scripts between different shells, some SE answers suggest using bash's built-in time command, like so:

time bash -c 'foo.sh'
time dash -c 'foo.sh'

...etc, for every shell to test. Such benchmarks fail to eliminate the time taken for each shell to load and initialize itself. For example, suppose both of the above commands were stored on a slow device with the read speed of an early floppy disk, (124KB/s), dash (a ~150K executable) would load about 7x faster than bash (~1M), the shell loading time would skew the time numbers -- the pre-loading times of those shells being irrelevant to measuring the run times of foo.sh under each shell after the shells were loaded.

What's the best portable and general util to run for script timing that can be run from within each shell? So the above code would look something like:

bash -c 'general_timer_util foo.sh'
dash -c 'general_timer_util foo.sh'

NB: no shell built-in time commands, since none are portable or general.

---

Better yet if the util is also able to benchmark the time taken by a shell's internal commands and pipelines, without the user having to first wrap them in a script. Artificial syntax like this would help:

general_timer_util "while read x ; do echo x ; done < foo"

Some shells' time can manage this. For example bash -c "time while false ; do : ; done" works. To see what works, (and doesn't), on your system try:

tail +2 /etc/shells | 
while read s ; do 
    echo $s ; $s -c "time while false ; do : ; done" ; echo ----
done

Solution 1

You should note that time is specified by POSIX, and AFAICT the only option that POSIX mentions (-p) is supported correctly by various shells:

$ bash -c 'time -p echo'

real 0.00
user 0.00
sys 0.00
$ dash -c 'time -p echo'

real 0.01
user 0.00
sys 0.00
$ busybox sh -c 'time -p echo'

real 0.00
user 0.00
sys 0.00
$ ksh -c 'time -p echo'       

real 0.00
user 0.00
sys 0.00

Solution 2

The time utility is usually built into the shell, as you have noticed, which makes it useless as a "neutral" timer.

However, the utility is usually also available as an external utility, /usr/bin/time, that may well be used to perform the timing experiments that you propose.

$ bash -c '/usr/bin/time foo.sh'

Solution 3

I use the GNU date command, which supports a high resolution timer:

START=$(date +%s.%N)
# do something #######################

"$@" &> /dev/null

#######################################
END=$(date +%s.%N)
DIFF=$( echo "scale=3; (${END} - ${START})*1000/1" | bc )
echo "${DIFF}"

And then I call the script like this:

/usr/local/bin/timing dig +short unix.stackexchange.com
141.835

The output unit is in milliseconds.

Solution 4

Here is a solution that:

1. eliminate[s] the time taken for each shell to load and initialize itself

2. can be run from within each shell

3. Uses

   no shell built-in time commands, since none are portable or general

4. Works in all POSIX-compatible shells.
5. Works on all POSIX-compatible and XSI-conforming systems with a C compiler, or where you can compile a C executable in advance.
6. Uses the same timing implementation on all shells.

There are two parts: a short C program that wraps up gettimeofday, which is deprecated but still more portable than clock_gettime, and a short shell script that uses that program to get a microsecond-precision clock reading both sides of sourcing a script. The C program is the only portable and minimal-overhead way to get a sub-second precision on a timestamp.

Here is the C program epoch.c:

#include <sys/time.h>
#include <stdio.h>
int main(int argc, char **argv) {
    struct timeval time;
    gettimeofday(&time, NULL);
    printf("%li.%06i", time.tv_sec, time.tv_usec);
}

And the shell script timer:

#!/bin/echo Run this in the shell you want to test

START=$(./epoch)
. "$1"
END=$(./epoch)
echo "$END - $START" | bc

This is standard shell command language and bc and should work as a script under any POSIX-compatible shell.

You can use this as:

$ bash timer ./test.sh
.002052
$ dash timer ./test.sh
.000895
$ zsh timer ./test.sh
.000662

It doesn't measure system or user time, only non-monotonic wall-clock elapsed time. If the system clock changes during the execution of the script, this will give incorrect results. If the system is under load, the result will be unreliable. I don't think anything better can be portable between shells.

A modified timer script could use eval instead to run commands outside of a script.

Solution 5

Multiple times revised solution using /proc/uptime and dc/bc/awk in large parts thanks to the input by agc:

#!/bin/sh

read -r before _ < /proc/uptime

sleep 2s # do something...

read -r after _ < /proc/uptime

duration=$(dc -e "${after} ${before} - n")
# Alternative using bc:
#   duration=$(echo "${after} - ${before}" | bc)
# Alternative using awk:
#   duration=$(echo "${after} ${before}" | awk '{print $1 - $2}')

echo "It took $duration seconds."

Assumes obviously that /proc/uptime exists and has a certain form.

View more solutions

Comments

[phk]

Concerning the portability, how about simply subtracting before and after value of /proc/uptime? (Also works across time changes.)

[Michael Homer]

I don't understand how any non-builtin could possibly both "eliminate the time taken for each shell to load and initialize itself" and execute a standalone script while being "portable and general".

[agc]

@MichaelHomer, if you know, (or believe), that certain constraints are necessarilly mutually exclusive, please put post those impossible combos in an answer.

[Michael Homer]

That's not an answer to the question, it's a prompt for you to clarify what you want.

[agc]

@MichaelHomer, various ways I'd suppose. For example, here's a crude method: a util might on the fly compute a good average of how much time a given shell takes to load with a null script, then subtract that time from the time the same shell takes to run the user's test script.

[Michael Homer]

I've posted my best attempt, but I think the question is still underspecified about exactly what it's actually trying to achieve.

[agc]

@Gilles, just curious, various problems come up now and then, it's not one thing. See my comment to muru's answer for an example of inconsistency among time builtins...

[peterh]

Also you have a reopen vote regarding your question.

[mosvy]

Both bash and dash are loaded on demand (so were they in 2017 ;-). Your assumption that they will have to be read whole from disk before being started, and starting a big executable is necessarily slower than a fast one is completely wrong. Especially since they're both dynamically linked, and the size of the libraries they're using may be bigger than their own size.

[mosvy]

s/than a fast one/than a small one/ above

[agc]

@mosvy, Well that's commonly true, but it's generally incorrect -- it all depends on the environment. An embedded (or a minimalist system) system might have a minimal amount of memory, and not preload anything because it hasn't enough buffer memory.

[mosvy]

Nothing to do with "preload": demand loading means that only the stuff actually used from the executable will be read from the disk: you can have a huge multi-giga executable, and have it start and exit instantly, because 99% of it will never be paged-in.

[agc]

@mosvy, And of course not all shells are dynamically linked.

[mosvy]

bash and dash are both dynamically linked.

[Kusalananda]

This would make it portable between shells, but not portable between Unix implementations since some simply lack the /proc filesystem. If this is a concern or not, I don't know.

[phk]

Assuming the time (epoch time) does not change in-between. Can't think of a case in practise where it would cause a problem but still worth mentioning.

[Kusalananda]

You should add that this requires GNU date specifically.

[Kusalananda]

The problem is that to be able to compare timings, the result has to be timed by the same implementation of time. It's comparable to letting sprinters measure their own time on 100m, individually, instead of doing it with one clock at the same time. This is obviously nitpicking, but still...

[Rabin]

@phk please explain ?

[muru]

@Kusalananda I thought the problem was that OP thought time isn't portable; it seems to be portable. (I agree with your point on comparability, though)

[Rabin]

@phk Thanks, you are right, small change but would be considered.

[agc]

@muru, on my system dash -c 'time -p while false ; do : ; done' returns "time: cannot run while: No such file or directory<cr> Command exited with non-zero status 127 " errors.

[muru]

@agc POSIX also says: "The term utility is used, rather than command, to highlight the fact that shell compound commands, pipelines, special built-ins, and so on, cannot be used directly. However, utility includes user application programs and shell scripts, not just the standard utilities." (see section RATIONALE)

[Jörg W Mittag]

Also, don't some NTP clients slow down and speed up the clock, rather than making "jumps" in the system time? If you have an NTP client like that, and you made some timings yesterday evening, they are probably skewed by the NTP client "anticipating" the leap second. (Or does the system clock simply run to 61 in that case?)

[agc]

@muru, what's the source of that POSIX quote?

[muru]

@agc manpages.ubuntu.com/manpages/wily/en/man1/time.1posix.html, section RATIONALE

[Michael Homer]

How does this "eliminate the time taken for each shell to load and initialize itself"?

[Kusalananda]

@Kevin Very true. I only took "no shell built-in time" into consideration it seems. The shell startup time might have to be measured separately.

[agc]

Coincidentally, just before reading this, (and the last line about eval), I was tweaking the script in Rabin's answer to include eval "$@", so it could run shell builtins on the fly.

[phk]

@JörgWMittag This wasn't about the leap second which do not affect UNIX epoch timestamps AFAIK and while the adaption method you talk about would make sense it is not what I was seeing. I believe the client was busybox's ntpd.

[Jonas Schäfer]

NTP clients will prefer slowing/accelerating the clock if the time is slightly off instead of changing it directly. This prevents jumps and clock discontinuities, which is generally nicer to applications working with timestamps. However, leap seconds are handled as additional seconds, unless you are using googles smearing NTP servers.

[Stéphane Chazelas]

I don't know of any shell that has a time builtin command. However many shells including bash have a time keyword that can be used to time pipelines. To disable that keyword so the time command (in the file system) be used, you can quote it like "time" foo.sh. See also unix.stackexchange.com/search?q=user%3A22565+time+keyword

[agc]

@muru, Reviewing RATIONALE: that section just seems to describe two sides of a compromise made because apparently nobody at the time which that was written had arrived at a satisfactory answer to this very question. Rather POSIX settled for both shell-based time builtins and a external time util, because even two non-general time methods were better than no time standards at all.

[phk]

@agc Good input, thanks, I added some alternative solutions accordingly.