How to make journald increase logs storage capacity?

Question

I would like to have persistent logs via journald. I created /var/log/journal and reloaded the service. Now logs are saved on the disk. But the log size limit is low.

I added /etc/systemd/journald.conf. (Storage=persistent/auto doesn't matter, I tried both).

[Journal]
Storage=persistent
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
SystemKeepFree=10G
SystemMaxFileSize=1G
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

As you can see I only changed each journal file size to 1Gb and told that I want 10Gb free on the disk.

But journald tells me that he only has 4Gb log storage capacity.

$ sudo systemctl status systemd-journald
...
jan 20 15:44:26 host systemd-journald[1218]: System journal (/var/log/journal/) is 4.5G, max 4.0G, 0B free.
jan 20 15:44:26 host systemd-journald[1218]: Journal started

What did I miss?

$ systemctl --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

Solution 1

It's necessary to setup SystemMaxUse=100G e.g.

Solution 2

You may need to also raise SystemMaxFiles since systemd will start removing journal files from /var/log/journal/*/ folder before it reaches the System{MaxUse/KeepFree} limit because it's ran out of the number of journal files.

On my system, each file in /var/log/journal/*/systemd@*.journal was capped at 129Mb and with a default limit of only 100 SystemMaxFiles, the amount of logs stored was effectively capped at ~13Gb despite setting SystemMaxUse to 50G.

Solution 3

From the man page:

SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=

[...]

The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G. If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

So yes, it maxes out at 4G, however, the following is interesting in your case:

journalctl and systemd-journald ignore all files with names not ending with ".journal" or ".journal~", so only such files, located in the appropriate directories, are taken into account when calculating current disk usage.

So, moving the log files regularly allows you to circumvent the limitation. Note that you will have to then ensure the logs do not fill up your file system somehow, removing old log files! You could, say have 1 file max 1G, then check regularly and move the file to ${filename}.n where n is 1-9, thus having max 10G ... provided you move the files when they reach 1G in size to the oldest ${filename}.n. To read the logs, you disregard the name, just oldest to youngest ...

Comments

[don_crissti]

The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G

[4xy]

Thank you. Does it mean that 4Gb is the maximum space can be occupied by the log?

[4xy]

Eventually the deal in SystemMaxUse=, thanks

[thecarpy]

I think if that is the case, then the man page is confusing.

[nicocesar]

Is this true? can it really go beyond 4G?

[ptman]

Seems to be able to go over 4G