System logs are empty (/var/log/messages; /var/log/secure; etc)

Question

I found that rsyslog stopped writing on logs (messages; secure; cron;etc)

System information: NAME="Red Hat Enterprise Linux Server" VERSION="7.4 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.4"

already restarted rsyslog with no results. Other logs processed by rsyslogd are being writen without issues.

Also restarted systemd-journald.

rsyslog.conf (Omitting commented lines):

$ModLoad imuxsock
$ModLoad imjournal
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local3.*                                                /var/log/sshd-second.log

Journald.conf (Omitting commented lines):

[Journal]

Already deleted /run/log/journal/* and restarted journald Already deleted imjournal.state and restarted rsyslog

output of the command rsyslogd -N 1:

rsyslogd: version 8.24.0, config validation run (level 1), master config /etc/rsyslog.conf

rsyslogd: End of config validation run. Bye.

output of the command rsyslogd -N 6:

rsyslogd: version 8.24.0, config validation run (level 6), master config /etc/rsyslog.conf

rsyslogd: End of config validation run. Bye.

Already tried all the solutions described in https://unix.stackexchange.com/questions/124942/rsyslog-not-logging with no results

Output of lsof -p

COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
rsyslogd 5820 root  cwd       DIR              202,2      236       64 /
rsyslogd 5820 root  rtd       DIR              202,2      236       64 /
rsyslogd 5820 root  txt       REG              202,2   663872   552421 /usr/sbin/rsyslogd
rsyslogd 5820 root  mem       REG               0,18  8388608    27215 /run/log/journal/ca23e130dda846d5b2a30e4ab9461e43/system.journal
rsyslogd 5820 root  mem       REG              202,2    29352 13067688 /usr/lib64/rsyslog/imudp.so
rsyslogd 5820 root  mem       REG              202,2    68192    34595 /usr/lib64/libbz2.so.1.0.6
rsyslogd 5820 root  mem       REG              202,2    99944    34673 /usr/lib64/libelf-0.168.so
rsyslogd 5820 root  mem       REG              202,2   402384    34477 /usr/lib64/libpcre.so.1.2.0
rsyslogd 5820 root  mem       REG              202,2    19888    34705 /usr/lib64/libattr.so.1.1.0
rsyslogd 5820 root  mem       REG              202,2   297328    45575 /usr/lib64/libdw-0.168.so
rsyslogd 5820 root  mem       REG              202,2   111080  1274500 /usr/lib64/libresolv-2.17.so
rsyslogd 5820 root  mem       REG              202,2    19384    34688 /usr/lib64/libgpg-error.so.0.10.0
rsyslogd 5820 root  mem       REG              202,2   535064    34703 /usr/lib64/libgcrypt.so.11.8.2
rsyslogd 5820 root  mem       REG              202,2   157400    34499 /usr/lib64/liblzma.so.5.2.2
rsyslogd 5820 root  mem       REG              202,2   155752    34476 /usr/lib64/libselinux.so.1
rsyslogd 5820 root  mem       REG              202,2  1139680    34473 /usr/lib64/libm-2.17.so
rsyslogd 5820 root  mem       REG              202,2    20032    34709 /usr/lib64/libcap.so.2.22
rsyslogd 5820 root  mem       REG              202,2    24928 13067682 /usr/lib64/rsyslog/imjournal.so
rsyslogd 5820 root  mem       REG              202,2    38032 13067689 /usr/lib64/rsyslog/imuxsock.so
rsyslogd 5820 root  mem       REG              202,2    24416 13067690 /usr/lib64/rsyslog/lmnet.so
rsyslogd 5820 root  mem       REG              202,2  2127336    23375 /usr/lib64/libc-2.17.so
rsyslogd 5820 root  mem       REG              202,2    88720  1233870 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
rsyslogd 5820 root  mem       REG              202,2    20040    35554 /usr/lib64/libuuid.so.1.3.0
rsyslogd 5820 root  mem       REG              202,2    40824   374355 /usr/lib64/libfastjson.so.4.0.0
rsyslogd 5820 root  mem       REG              202,2    15424   392270 /usr/lib64/libestr.so.0.0.0
rsyslogd 5820 root  mem       REG              202,2    44448    23398 /usr/lib64/librt-2.17.so
rsyslogd 5820 root  mem       REG              202,2    19776    34471 /usr/lib64/libdl-2.17.so
rsyslogd 5820 root  mem       REG              202,2   144792  1274481 /usr/lib64/libpthread-2.17.so
rsyslogd 5820 root  mem       REG              202,2    90632    34489 /usr/lib64/libz.so.1.2.7
rsyslogd 5820 root  mem       REG              202,2   164112    23368 /usr/lib64/ld-2.17.so
rsyslogd 5820 root  mem       REG              202,2   162560     3600 /usr/lib64/libsystemd.so.0.6.0
rsyslogd 5820 root    0r      CHR                1,3      0t0     1041 /dev/null
rsyslogd 5820 root    1w      CHR                1,3      0t0     1041 /dev/null
rsyslogd 5820 root    2w      CHR                1,3      0t0     1041 /dev/null
rsyslogd 5820 root    3u     IPv4              28378      0t0      UDP *:syslog
rsyslogd 5820 root    4u     IPv6              28379      0t0      UDP *:syslog
rsyslogd 5820 root    5r  a_inode                0,9        0     5987 inotify
rsyslogd 5820 root    6u     unix 0xffff8800da61a400      0t0    28380 socket
rsyslogd 5820 root    7r      REG               0,18  8388608    27215 /run/log/journal/ca23e130dda846d5b2a30e4ab9461e43/system.journal
rsyslogd 5820 root    8u  a_inode                0,9        0     5987 [eventpoll]
rsyslogd 5820 root    9w      REG              202,2   193240     8457 /var/log/haproxy.log

Does anyone have any clues?

Solution

well... almost 2 years had gone by and finally found the solution for the issue. If anyone has the same issue please try the steps below.

syslogd version rsyslogd 8.24.0-38.el7

The issue was related to the imjournal module. I've remove all the entries of the rsyslog.conf related to imjournal and switched the OmitLocalLogging to off

After that I've restarted the rsyslog service using systemctl restart rsyslog and the log entries started to be populated to the log file that I've configured in rsyslog.conf.

The rsyslog.conf file now looks like this:

$ModLoad imuxsock
#$ModLoad imjournal
$ModLoad immark 
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging off
#$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
local3.*                                                /var/log/sshd-second.log

Comments

[BANJOSA]

That doesn't seem to be the issue. The version that is running is rsyslog.x86_64 0:8.24.0-12.el7 (the same one where you link points to). Nevertheless i upgraded to version rsyslog.x86_64 0:8.24.0-16.el7_5.4 witth no success in solving the issue.

[Satish]

I have same issue and no solution found

[BANJOSA]

I haven´t found a solution yet. if found I'll posted it here.

[BANJOSA]

Solution was found :)

[Jeter-work]

This does not address the underlying issue, which is, "Why did rsyslog stop or why did it not start." And does not address that this is only needed if rsyslog is stopped. Without resolving the underlying issue, upon the next system restart, rsyslog could fail again. Or it could fail again in the mean time.

[Modassir Haider]

I have always faced size issue on messages files. Sometimes size of /var/log/messages goes above 6GB. I archive /var/log/messages and create a new file, provide required permission and restart the rsyslog so that logs starts written to messages. I have done this many times in PROD environment. I don't know how can people disvote my post without knowing anything. Anyway I posted here to spread knowledge not for votes.

[Modassir Haider]

rsyslog already running on PID. If you restart, it will jump to another PID.

[BANJOSA]

you are being downvoted because your answer is not related with the issue presented. If you read the description and the comments that were made you will see that the service is running, was already restarted, and the logs were no written. Nevertheless, the solution was already found fot the presented issue.

[Altimus Prime]

Granted, the OP had some different questions, but the Google overlords serve this page as an answer to my question, which was different from the OP, and this answered helped me.

[Admin]

Just setting OmitLocalLogging off works for me. No need to comment out imjournal. Also works for var/log maillog and var/log/cron etc.