Big /var/log/journal?

Question

I don't understand why the /var/log/journal/ folder is so big.

For example, by executing the command systemctl -f, i see the fill. If I click on an email on Thunderbird, it generates dozens of lines that I consider useless.

Currently, I have more than 1.5GB (du -h /var/log/journal/) generated in 1 day!

Is there a method to reduce this log considerably without stopping logging?

Solution 1

You can diminish the size of the journal by means of these commands:

journalctl --vacuum-size=100M

This will retain the most recent 100M of data.

journalctl --vacuum-time=10d

will delete everything but the last 10 days.

From man journalctl:

--vacuum-size=, --vacuum-time=, --vacuum-files= Removes the oldest archived journal files until the disk space they use falls below the specified size (specified with the usual "K", "M", "G" and "T" suffixes), or all archived journal files contain no data older than the specified timespan (specified with the usual "s", "m", "h", "days", "months", "weeks" and "years" suffixes), or no more than the specified number of separate journal files remain. Note that running --vacuum-size= has only an indirect effect on the output shown by --disk-usage, as the latter includes active journal files, while the vacuuming operation only operates on archived journal files. Similarly, --vacuum-files= might not actually reduce the number of journal files to below the specified number, as it will not remove active journal files.

--vacuum-size=, --vacuum-time= and --vacuum-files= may be combined in a single invocation to enforce any combination of a size, a time and a number of files limit on the archived journal files. Specifying any of these three parameters as zero is equivalent to not enforcing the specific limit, and is thus redundant.

These three switches may also be combined with --rotate into one command. If so, all active files are rotated first, and the requested vacuuming operation is executed right after. The rotation has the effect that all currently active files are archived (and potentially new, empty journal files opened as replacement), and hence the vacuuming operation has the greatest effect as it can take all log data written so far into account.

Solution 2

As @kurt-fitzner wrote:

Teach them to edit /etc/systemd/journald.conf and you teach them how to solve the problem permanently.

More specifically: Activate the SystemMaxUse= option there, e.g. as SystemMaxUse=100M to only use 100 MB.

After editing, use service systemd-journald restart to activate the changed configuration. This will remove the excess logs.

journald.conf also has other options that might be useful.

Solution 3

You also need to set this in /etc/systemd/journald.conf:

SystemMaxFileSize=100M

See: https://got-tty.org/journalctl-via-journald-conf-die-loggroesse-definieren (in German)

Comments

[0xC0000022L]

For me it's 2.5G on one 18.04 machine and 4.0G on another. What's the issue with the size?

[Rinzwind]

"I don't understand why the /var/log/journal/ folder is so big." because it logs everything.

[Rinzwind]

Might be sufficient for most but if I got 10 dollars for every time some co-worker asks me what happen on a specific date where that date was more than 10 days (heck 1 here dared to ask about something happend 6 years ago :D ) I'd be rich. Log rotate (last sentence) might be a better method for most and it might be nice to focus more on that part :)

[Tomachi]

Vacuuming done, freed 3.7G of archived journals from /var/log/journal !

[Kurt Fitzner]

Perhaps this can be edited to show how to set journald growth limits. Teach people to vacuum and they mitigate the problem once. Teach them to edit /etc/systemd/journald.conf and you teach them how to solve the problem permanently. Also, while not asked, a little bit about why journald is even necessary vs. text syslogs. Rinzwind's comment might otherwise prevent people from controlling journald when they actually have most everything journald offers in another source.

[user199710]

After spending the afternoon carefully looking at documentation before proceeding with it, I run journalctl --vacum-size=100M thinking it would delete 100M of files in /var/log/journal ... I then see "Vacuuming done, freed 2.0G of archived journals", I'm an idiot! Oh well, not that I needed log files from March 2020 (I hope)

[jeffmjack]

MaxRetentionSec is the parameter in /etc/systemd/journald.conf that controls how long to keep journal files. This setting takes time values which may be suffixed with the units "year", "month", "week", "day", "h" or "m" to override the default time unit of seconds.

[Levente]

Just a remark: sudo seems to be necessary.

[slv]

After editing, sudo service systemd-journald restart takes the changes into account and vacuums.

[eric.green]

One last thing -- in a professional environment, you should be running a centralized logging system with sufficient space to store a year's worth of logs (possibly compressed) for every system in the company (or at least six months worth) so that if your network is compromised, the compromise can be tracked back in time. So journald keeping more than a few days of local logs is literally a waste of local disk space in a professional enviornment.