systemd-journald Doesn't start at all

Question

After deleting my /var/log/journal by accident, I am unable to recover from the consequences. systemd-journald keeps failing over and over again every time I try to start it. when I run:

$ journalctl --verify
PASS: /run/log/journal/system.journal

which doesn't bring me one step closer to solve it because /run/log/journal/system.journal is created when I run /usr/lib/systemd/systemd-journald directly, which on it self outputs

$ /usr/lib/systemd/systemd-journald
Failed to open runtime journal: Invalid argument

Things I have already tried:

1. cleaning /var/log/journal and/run/log/journal`.
2. copying /var/log/journal from another machine to both paths, running journalctl --verify to check them, and once again with --file flag.
3. changing Storage setting in /etc/systemd/journald.conf to Storage=volatile just to disable /var/log/journald.
4. Rebooting.

Solution 1

After renaming the hostname, deleting & recreating /etc/machine-id by running systemd-machine-id-setup and rebooting to apply these, systemd-journald came back to life!

Solution 2

Try deleting /var/log/journal in case that there are any remainders from previous restoring attempts. Then set Storage=persistent in /etc/systemd/journald.conf and reboot.

Normally systemd should recreate /var/log/journal during reboot.

If this doesn't help try

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

Comments

[U.Z.A.I.R]

I deleted and recreated /etc/machine-id and then ran systemd-tmpfiles --create --prefix /var/log/journal without rebooting and it worked.

[markshep]

My journal service was failing in the same way due to a stray newline in the middle of the /etc/machine-id file. Dunno how that got there! Removing the file, running systemd-machine-id-setup then systemctl start systemd-journald.service fixed it. No reboot necessary.

[Top Sekret]

Then strace will trace only systemctl, not the actual systemd. It uses IPC to communicate with main daemon.