Why `journalctl --list-boots` doesn't match what `uptime` and `who -b` report?

Question

Here's a test script I used:

last_reboot=$(last reboot | grep 'still running' | awk '{for (i=5; i<=NF; i++) printf $i FS}' | awk '{for (i=1; i<=NF - 2; i++) printf $i FS}')
if [ "$last_reboot" ]; then
    date -d "$last_reboot" '+last reboot: %Y-%m-%d'
fi

days=$(uptime | awk '{print $3}')
hours=$(uptime | awk '{print $5}' | sed -E 's/,$//')
h=$(echo "$hours" | cut -d: -f 1)
m=$(echo "$hours" | cut -d: -f 2)
date -d "- $days days - $h hours - $m minutes" '+uptime: %Y-%m-%d'

who -b | awk '{print "who: " $3}'

journalctl --list-boots | awk '$1 == "0" {print "journalctl: " $4}'

Locally, all four dates match.

I ran it on about 10 servers. last reboot doesn't report anything (probably, because wtmp gets rotated by logrotate). uptime and who -b match. And journalctl doesn't. What exactly does journalctl --list-boots report? Why can it not match what other tools report?

Solution

The new binary logs on Linux operating systems do not work in the way that the old binary logs did.

The old binary logs were /var/log/wtmp and /var/log/btmp. At system bootstrap an entry would be written to wtmp with the username reboot, and at shutdown an entry would be written to wtmp with the username shutdown. Finding the times that the system was rebooted was a matter of using the last reboot and last shutdown commands to print out these entries.

The new binary logs are the systemd journal, and they do not have such entries.

Instead, every journal record has a field named the boot ID. You can see this with the -o verbose option to journalctl. A boot ID is generated by the kernel at bootstrap, and systemd-journald applies the current boot ID, taken from the kernel, to every log record as it is adding it to the journal.

To implement the list-boots functionality, journalctl scans the entire journal, reading the timestamps and boot IDs of every record, and noting the earliest and latest timestamps associated with each unique boot ID.

Thus if parts of the journal are purged, or conversely stick around overlong, the apparent boot and shutdown times reported by journalctl will differ wildly from the actual boot and shutdown times.

/run/utmp is a table of terminal login records, with special entries for bootup and shutdown. These entries are read by uptime and who -b. They are written by programs such as systemd-update-utmp, an analogue of the FreeBSD utx command, which are run as parts of the startup and shutdown procedures. They are not run first or last, as the relevant services are not (and indeed cannot be) ordered absolutely first or last. There may be journal entries with the relevant boot ID that precede the time that systemd-update-utmp reboot is run, and similar journal entries that postdate the time that systemd-update-utmp shutdown is run.

Further reading

• https://unix.stackexchange.com/a/383575/5132
• https://unix.stackexchange.com/a/294206/5132

Comments

[Raman Sailopal]

systemd collates and stores information reported by syslog

[Rfraile]

Maybe you can use tuptime, it is designed exactly for track and report the system life.

[x-yuri]

Can you elaborate on "stick around overlong" part? "Thus if parts of the journal are purged, or conversely stick around overlong, the apparent boot and shutdown times reported by journalctl will differ wildly from the actual boot and shutdown times." How can it affect boot and shutdown times reported by journalctl?

[JdeBP]

Read the very first item of further reading.

[x-yuri]

I'd already done so, when I was asking my question. Let me try and explain my point. When logs doesn't get purged, we get more or less accurate boot and shutdown times. Since at those moments something is bound to get to the log. When some parts of the log get purged, boot and shutdown times change. Could you possibly mean this particular thing? That depending on how much of the log you have, boot and shutdown times may differ drastically? If that wasn't your point, I'd appreciate it to hear at least a brief explanation.