Where is "journalctl" data stored?

Question

When I issue journalctl I get a massive log of all system services, but where is all this information stored?

Solution 1

From man systemd-journald:

FILES
       /etc/systemd/journald.conf
           Configure systemd-journald behavior. See journald.conf(5).

       /run/log/journal/machine-id/*.journal,
       /run/log/journal/machine-id/*.journal~,
       /var/log/journal/machine-id/*.journal,
       /var/log/journal/machine-id/*.journal~
           systemd-journald writes entries to files in
           /run/log/journal/machine-id/ or /var/log/journal/machine-id/ with
           the ".journal" suffix. If the daemon is stopped uncleanly, or if
           the files are found to be corrupted, they are renamed using the
           ".journal~" suffix, and systemd-journald starts writing to a new
           file.  /run is used when /var/log/journal is not available, or when
           Storage=volatile is set in the journald.conf(5) configuration file.

And as man journalctl says:

journalctl may be used to query the contents of the systemd(1) journal
as written by systemd-journald.service(8).

These logs are managed by the systemd-journald service, so a more appropriate term would be "journald logs".

Solution 2

Note however that Ubuntu is not using a persistent journald log file by default. Only the volatile /run/log/journal/<machine-id>/*.journal[~] is kept until the next boot. All is lost at each reboot.

You may see a list of boot retained in the log with:

journalctl --list-boot

The logs are still kept in a text file under /var/log unless you have activated the use of persistent journald log by creating /var/log/journal directory.

Solution 3

Short answer

Usually the storage directory is /var/log/journal or /run/log/journal, but it doesn't have to necessarily exist in your system.

If you just want to check the amount of space that the journal is currently occupying on your disk, simply type:

$ journalctl --disk-usage

Long answer

The storage directory depends on journald configuration.

Configuration files are:

/etc/systemd/journald.conf
/etc/systemd/journald.conf.d/*.conf
/run/systemd/journald.conf.d/*.conf
/usr/lib/systemd/journald.conf.d/*.conf

There the "Storage=" option controls whether to store journal data or not, and where. Possible values are "volatile", "persistent", "auto" and "none". Defaults to "auto".

If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed).

If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.

"auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes.

"none" turns off all storage, all log data received will be dropped.

Solution 4

In addition to Muru's answer on where data is stored there are other relevant answers.

How to increase journalctl to find previous boot logs

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal

How to keep journalctl file size down

$ sudo journalctl --vacuum-size=200M
Deleted archived journal /var/log/journal/d7b25a27fe064cadb75a2f2f6ca7764e/[email protected]~ (56.0M).
Deleted archived journal /var/log/journal/d7b25a27fe064cadb75a2f2f6ca7764e/[email protected]~ (8.0M).
Deleted archived journal /var/log/journal/d7b25a27fe064cadb75a2f2f6ca7764e/user-1000@1bbb77599cf14c65a18af51646751696-000000000000064f-00056444d58433e1.journal (112.0M).
Vacuuming done, freed 176.0M of archived journals on disk.

View more solutions

Comments

[user2284570]

With the use of Berkley ᴅʙ 4.3 as the file format for logs. I’m thinking journalctl is you only option.

[php_nub_qq]

Thanks for the correction, but suppose a noob like me will search for that too so I guess it is better left this way. A follow-up question - are these logs safe to delete?

[muru]

Well, logs are safe to delete unless you need the information from them later on.

[mattdm]

Note that by default, systemd will delete older logs as they approach a certain percentage of disk space used.

[Mark Stosberg]

However, the journald log arguably should be persistent by default. [bug #1618188] (bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188) has been opened to track the progress of this change. Check there for the latest status.

[Klaas van Schelven]

As per 17.10 the above-mentioned bug is fixed

[Alexis Wilke]

@muru, at the same time, if the logs are under /run/log, they won't survive a reboot...

[Doug Smythies]

I had to use sudo for --vacuum-size=. But thanks.

[WinEunuuchs2Unix]

@DougSmythies Thanks for pointing that out. I have a robot named cron that does all my vacuuming monthly :). So I never had to type command in terminal. I've updated the answer.