Relationship of rsyslog and journald on Ubuntu 16.04

Question

I am running what is a vanilla Ubuntu 16.04 server, and I'm trying to wrap my head around how logging is set up by default. I can see that both journald and rsyslog are installed and running, but it's not at all clear to me how log messages are being processed.

Most messages seem to show up both in /var/log/syslog and via journalctl, but I can't see any explicit configuration for forwarding between the two in either /etc/systemd/journald.conf (which is basically all commented out by default), /etc/rsyslog.conf or /etc/rsyslog.d/50-default.conf. I tried to look for official documentation, or even a blog post explaining how hese two are set up in Ubuntu, but haven't managed to find anything.

To further add to my confusion, I have executed logger -p local1.info Test on the host, and found that nothing was written to /var/log/syslog, while the message did show up under journalctl.

My questions are:

1. How exactly do journald and rsyslog work together on Ubuntu 16.04 (by default)?
2. How come messages sent from logger seemingly end up in the journal, but not in syslog?

Update: Turns out logger not working as expected was a mistake on my end, so it's not relevant to the main question.

Solution 1

By default rsyslog is using "imuxsock" module, the module provides:

the ability to accept syslog messages via local Unix sockets. Most importantly, this is the mechanism by which the syslog(3) call delivers syslog messages to rsyslogd [1].

It is possible for rsyslog to import structured log messages from systemd-journal using a module named "imjournal" [2].

It can be load like:

module(load="imjournal") 

in:

/etc/rsyslog.conf

In the other hand "systemd-journald" captures all data itself:

man systemd-journald

systemd-journald is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:

   ·   Kernel log messages, via kmsg
   ·   Simple system log messages, via the libc syslog(3) call
   ·   Structured system log messages via the native Journal API, 
       see sd_journal_print(4)
   ·   Standard output and standard error of system services
   ·   Audit records, via the audit subsystem

You can disable rsyslogd while you still have access to system logs using journalctl.

$ sudo systemctl mask rsyslogd
$ sudo systemctl stop syslog.socket
$ sudo systemctl stop rsyslog.service
$ systemctl is-active rsyslog.service 
inactive
$ logger -p mail.info Helllooo
$ journalctl

For example, centos is using "imuxsock" module to capture all "systemd-journald" data via rsyslog while opensuse does not have "syslog" at all.

---

To find out why your message didn't end up to /var/log/syslog, you should check this file:

less /etc/rsyslog.d/50-default.conf

look for *.info, see where they will be stored, it might be an other file like messages.

For me it show up in both journalctl and syslog.

Solution 2

Systemd is an init system used to start the services when the system boots up. Journald is responsible for making logs for services that are started by systemd. By integrating journald with systemd, even the earliest boot process messages are available to journald.

Rsyslog is a daemon specially made for log processing, nothing to do with journald. It can take logs in by many ways and output them in many ways. It is not enabled by default that it takes log messages from journald also. For that, you have to write in /etc/rsyslog.conf file,

$ModLoad imjournal # im -> input module
OR
load(type="imjournal")

Now, it will accept logs fromm journald also. But I suggest you should not change your /etc/rsyslog.conf file.

At the end of /etc/rsyslog.conf file, there is a line written,

$IncludeConfig /etc/rsyslog.d/*.conf

It means that all the files having .conf at there end in the /etc/rsyslog.d/ folder should be included during rsyslog loading. So, all your custom configs should go in these files

I would suggest you to make a file /etc/rsyslog.d/journald.conf and paste the below snippet in that.

Below is the snippet from the rsyslog official page of imjournal

module(load="imjournal" PersistStateInterval="100"
   StateFile="/path/to/file") #load imjournal module
module(load="mmjsonparse") #load mmjsonparse module for structured logs

template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n" ) #template for messages

action(type="mmjsonparse")
action(type="omfile" file="/var/log/ceelog" template="CEETemplate")

Line 1 - it loads the imjournal module for accepting the logs from journald

2 - The mmjsonparse module is loaded used in parsing of the logs

3 - They are structured into the described format in the template

4 - It parses those logs using the mmjsonparse module.

5 - It sends those logs to a file namely /var/log/ceelog according to the structure provided in the given template, using the omfile(output module file - outputs to file) module.

Make changes in the config according to your need.

Comments

[luv.preet]

logger -p .info hello is not valid. You have not given any facility name.

[Ravexina]

Updated the answer ....

[motns]

As I noted in my edit, logger not working was my fault, so that's fixed now (thanks for the tips anyway). As for logs showing up in two places, your note on imuxsock seems to be the key: it seems both rsyslog and journald are listening for local syslog messages, hence why those entries go into two separate logs.

[motns]

Thanks for the explanation, but I think the answer by @Ravexina is closer to helping me understand what's going on with rsyslog and journald, by default (without extra configuration).

[Hackeron]

I followed the above on ubuntu and now I'm not getting any logs from logger, hmm: root@T:~# logger -p mail.info Helllooo root@T:~# journalctl No journal files were found. -- No entries -- any ideas?

[Matthijs Kooijman]

Great answer. What I'm missing,though, is a note about how journald knows to open a syslog socket (or if it does not, how it gets at syslog messages otherwise) when rsyslog is disabled? Or is some manual configuration of journald needed?

[axblount]

The answer explains how both can work, but not what actually happens.

[eichin]

The missing bit of magic in 16.04 is that imuxsock is hard-coded to look at /run/systemd/journal/syslog ( SYSTEMD_PATH_LOG ) and /lib/systemd/system/syslog.socket has ListenDatagram=/run/systemd/journal/syslog and that triggers /etc/systemd/system/syslog.service -> /lib/systemd/system/rsyslog.service and debian/patches/Re-enable-journal-forwarding-to-syslog.patch makes ForwardToSyslog the default, with only the barest documentation update.