init, syslog and stdout/stderr

Question

I'm trying to build an embedded system with linux. (Means very few resources and not much else than busybox.) I'd like to make sure, that everything is going to syslog. I'm using the newest version of busybox (v.1.21.1), with the builtin init and syslogd.

There's a few issues though. First the inittab:

....
null::sysinit:/bin/sh /etc/rc
null::sysinit:/bin/touch /var/log/messages
null::respawn:/sbin/syslogd -n -S -s 12 -b 8
null::respawn:/sbin/klogd -n
....

The rc script is run before syslog, which means that all usefull information from rc is lost. (I'm not sure, what the touch command is for btw.) I could start syslog before rc (null::sysinit:/sbin/syslogd...), but that would remove the very important 'respawn'.

Also when 3rd party daemons are starting up, it could look like this:

...
null::respawn:/bin/modbus
...

If the daemon doesn't support syslog or if something slips (eg. a print to stderr) it is lost. I cannot seem to redirect output to syslog. Eg.

...
console::respawn:/bin/modbus|logger 2>1
...

I've tried a number of different combinations. Nothing seems to work well. Ofc, I could edit all the daemons and make them support syslog. (But what if something slips?) It could be solved if I could write something like this:

log::respawn:/bin/modbus

And actually, there is a /dev/log. But it's a socket, normal redirects won't work. ... so I could make it work through a custom kernel module. Eg. I could create a module that creates a /dev/syslog_link that writes everything to printk. Problem is that the syslog would mark all messages from that one as 'kernel'. Very wrong.

So now I'm thinking that I could create a kernel module that writes to the /dev/log: https://stackoverflow.com/questions/1184274/how-to-read-write-files-within-a-linux-kernel-module https://www.cs.drexel.edu/~jjohnson/2012-13/fall/cs543/project/reading/kernel_fileio.pdf

I don't know if it's possible though and writing to files from kernel space hurts my fragile sence of right and wrong.

Any thoughts?

Solution 1

The rc script is run before syslog

There may be a good reason for that, e.g, because that's where the filesystem containing /var/log gets remounted rw. If not, then you can start syslog first.

I've tried a number of different combinations. Nothing seems to work well.

I haven't played much with inittab in more than a few years. Have you tried putting, e.g., /bin/modbus|logger 2>1 in a short script and then using that with respawn instead?

Ofc, I could edit all the daemons and make them support syslog.

If you can, you probably should. Or you could have them write directly to a file.

there is a /dev/log. But it's a socket, normal redirects won't work. ... so I could make it work through a custom kernel module.

That seems like overkill. It would be easier to use a userspace daemon and start that first; it can read from a fifo and write to whatever. Of course, that might be a bit redundant considering there is already syslog :/ The trick with a fifo reader is to reopen the pipe when read() returns 0. But again, I'd try to get syslog working first. Keeping stuff standard seems simpler.

Solution 2

You might consider /dev/kmsg, for early startup messages. These will be held in a circular buffer in the kernel until userspace (of pretty much all distros) dump it into syslog after the filesystem is made writable and disk checks have run. This will work as long as your kernel buffer is larger than the size of the messages that happen before the logger is started. The size of the kernel log buffer is configurable at compile time. You should be able to redirect things into /dev/kmsg with typical shell scripts.

As for redirecting stdout of arbitrary programs to syslog, there are utilities out there that read STDIN and redirect to syslog, and then you just pipe the output of the daemon to the utility. Here's one I've found useful: http://b0llix.net/perp/site.cgi?page=sissylog.8

All that said, there are lots of people doing lots of clever things with logging. Since you seem interested in the subject, you might be interested in reading the section "The syslog design is flawed form the start" from http://www.skarnet.org/software/s6/s6-log.html