How do view older journalctl logs (after a rotation maybe?)

Question

I am running docker on ubuntu 16.04 and would like to view the logs. However, I am unable to view logs after what I am guessing is some sort of rotation or the logs grow to a certain size.

I have not made any changes to my journald.conf, so I am using defaults there. There are containers running so the docker log outputs quite a lot of data.

Examples of what I am seeing:

systemctl docker status confirms service has been active:

since Thu 2016-10-13 18:56:28 UTC

However, when I run something like:

journalctl -u docker.service --since "2016-10-13 22:00"

The only output I get is:

-- Logs begin at Fri 2016-10-14 01:18:49 UTC, end at Fri 2016-10-14 16:18:25 UTC. --

I can view the logs in that range as expected.

My question is: why can I not view older logs with journalctl, and how can I fix this issue so I can view the logs?

Solution 1

The reason this happens is because of defaults on the size of journald files stored. There is more detail about this in the docs. It's worth reading the whole section I have linked to, but the defaults work like so:

journald will use 10% of the disk or 4G, whichever is smaller.

journald will leave free 15% of the disk or 4G, whichever is larger.

For viewing logs from the last boot, assuming you have Storage=persistent in your journald.conf, as the other answer notes, you can use the --boot=-1 flag on journalctl commands to get logs from just the previous boot.

In the case of the OP where they were sure the host had not been rebooted, the loss of logs was simply caused by the SystemMaxUse and/or SystemKeepFree defaults.

Note: I'm the OP and this question still has upvotes trickling in, so since I've gained more experience with journald (and rtfm) I am posting this here in the hopes it helps others.

Solution 2

It could be because you are trying to review the journal since the last boot, which seems likely to be the case inside a docker image.

On Ubuntu 16.04, the journal storage defaults to being in-memory. You can change the default to be persistent by opening /etc/systemd/journald.conf and changing the Storage= line from auto to persistent. You may need to restart journald by systemctl restart systemd-journald after the config file edit.

I think the journal should be persistent-by-default, so I opened a bug about that.

Comments

[jdf]

I agree with you it should be persistent by default (and will change that setting), but I am certain this machine has not been rebooted, so I am not 100% convinced that alone will resolve my issue. Still glad you filed that issue.

[Mark Stosberg]

Under certain conditions, logs can land in the journal with incomplete metadata, so that they can be found when reviewing the full journalctl output but not the unit logs. This has been known to happen to logs issues just before a unit shuts down. The logs are sent to journald, and then journald issues a query for the metadata. Sometimes in the meantime the process exists. That also doesn't sound like it is a perfect fit for your case, though.

[Dmitry Markin]

I'd like to emphasize that with in-memory log storage "10% of the disk" refers to 10% of tmpfs on /run, which itself typically defaults to 10% of RAM. So you can reach this 1% of RAM limit pretty fast

[jgomo3]

But in any case: How to see those old logs?

[Mark Stosberg]

@jgomo3 It's not clear if you are talking about a system where the the logs are persistent vs those in memory. If they were in-memory, they are lost with each boot.