Ubuntu 14.04 - logrotate does not rotate /var/log/* (rsyslog config)

Question

I have a bunch of Ubuntu 14.04 servers I manage where logrotate is not rotating anything under /var/log/*. I'm going to assume its an issue with the /etc/logrotate.d/rsyslog as I see other entries in /etc/logrotate.d/ rotating logs properly.

These servers are deployed using CHEF. However I don't see anything in the recipes, etc. that are referencing any functional changes to logrotate. There are a few scripts added to /etc/logrotate.d and those scripts do run every day as expected. Is it possible that one script under /etc/logrotate.d could affect another (I wouldn't think so)? Similar servers using the same CHEF recipes are configured on Ubuntu 12.04 boxes do not have this symptom of logs not rotating under /var/log/*.

Running /usr/sbin/logrotate -d /etc/logrotate.conf, I get the following (omitting anything not rsyslog related):

rotating pattern: /var/log/syslog
 after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 104
considering log /var/log/syslog
  log does not need rotating
switching euid to 0 and egid to 0

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 weekly (4 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 104
considering log /var/log/mail.info
  log does not need rotating
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
  log does not need rotating
considering log /var/log/daemon.log
  log does not need rotating
considering log /var/log/kern.log
  log does not need rotating
considering log /var/log/auth.log
  log does not need rotating
considering log /var/log/user.log
  log does not need rotating
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  log does not need rotating
considering log /var/log/messages
  log does not need rotating
not running postrotate script, since no logs were rotated
switching euid to 0 and egid to 0

When I look in /var/log, I can see that nothing is being rotated:

$ ls -l /var/log
total 34116
-rw-r--r-- 1 root      root    19512 Jan  9 07:15 alternatives.log
drwxr-xr-x 2 root      root     4096 Jan  7 20:28 apt
-rw-r----- 1 syslog    adm   3725622 Jan 12 19:50 auth.log
-rw-r--r-- 1 root      root     2481 Jan 10 03:46 boot.log
-rw-rw---- 1 root      utmp        0 Apr 11  2013 btmp
-rw-r--r-- 1 syslog    adm      6170 Jan  7 20:11 cloud-init.log
drwxr-xr-x 2 root      root     4096 Nov 18  2011 cron-apt
-rw-r----- 1 root      adm     19724 Jan 12 19:21 daemon.log
-rw-r----- 1 root      adm      5944 Jan 10 03:46 debug
drwxr-xr-x 2 root      root     4096 Oct 10  2012 dist-upgrade
-rw-r--r-- 1 root      adm     15312 Jan 10 03:46 dmesg
-rw-r--r-- 1 root      adm     15312 Jan  8 04:18 dmesg.0
-rw-r--r-- 1 root      adm      5451 Jan  7 20:11 dmesg.1.gz
-rw-r--r-- 1 root      root       28 Jan  7 20:11 dmesg.2.gz
-rw-r--r-- 1 root      root   162648 Jan 12 07:10 dpkg.log
drwxr-xr-x 2 root      root     4096 Apr 11  2013 fsck
-rw-r----- 1 syslog    adm     81900 Jan 10 03:46 kern.log
drwxr-xr-x 2 landscape root     4096 Jan  7 20:11 landscape
-rw-rw-r-- 1 root      utmp   292584 Jan 12 19:20 lastlog
drwxr-xr-x 2 root      root     4096 Jan  7 20:43 logstash
-rw-r----- 1 syslog    adm         0 Jan  7 20:11 mail.err
-rw-r----- 1 root      adm   9433681 Jan 12 19:50 mail.info
-rw-r----- 1 syslog    adm   9433797 Jan 12 19:50 mail.log
-rw-r----- 1 root      adm         0 Jan  7 20:43 mail.warn
-rw-r----- 1 root      adm     92617 Jan 12 07:10 messages
-rw-r----- 1 root      adm       519 Jan  7 20:43 monit.log
drwxr-s--- 2 mysql     adm      4096 Jan  7 20:45 mysql
-rw-r----- 1 mysql     adm         0 Jan  7 20:45 mysql.err
-rw-r----- 1 mysql     adm         0 Jan  7 20:45 mysql.log
drwxr-xr-x 2 root      root     4096 Jan  7 20:11 news
drwxr-xr-x 2 www-data  root     4096 Jan  7 20:41 nginx
-rw-r----- 1 syslog    adm  11460381 Jan 12 19:50 syslog
drwxr-xr-x 3 root      root     4096 Jan 12 19:45 sysstat
-rw-r--r-- 1 root      root    84672 Jan 10 03:46 udev
-rw-r----- 1 syslog    adm         0 Jan  7 20:11 ufw.log
drwxr-xr-x 2 root      root     4096 Jan  8 04:20 upstart
-rw-r----- 1 root      adm     41714 Jan 12 07:10 user.log
-rw-rw-r-- 1 root      utmp   206208 Jan 12 19:20 wtmp

Here is my version of logrotate (Stock Ubuntu 14.04):

$logrotate
logrotate 3.7.8 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License

Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command] [-s|--state=statefile] [-v|--verbose] [-?|--help]
        [--usage] [OPTION...] <configfile>

Logrotate is in cron.daily (default config):

$ ls /etc/cron.daily/logrotate
/etc/cron.daily/logrotate

The default crontab config was never changed:

$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

The /etc/logrotate.conf file:

$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

The /etc/logrotate.d/rsyslog file:

$ cat rsyslog
/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

Solution

Same issue happened and resolved on Ubuntu 14.04 server with below change in /etc/logrotate.d/rsyslog

from:

/var/log/syslog
{
    rotate 7
    daily
    ...
    postrotate
        reload rsyslog >/dev/null 2>&1 || true
    endscript
}

to this:

{
    rotate 7
    daily
    ...
    postrotate
        service rsyslog rotate >/dev/null 2>&1 || true
    endscript
}

for each rotated rsyslog files. I triggered logrotate manually with "/usr/sbin/logrotate /etc/logrotate.conf" and tested it with logger.

Comments

[Admin]

A good friend suggested that a logrotate script under /etc/logrotate.d can affect the others as stated in the logrotate man page (missed that). To verify that custom scripts were not causing problems, I removed them and re-ran sudo /usr/sbin/logrotate -d /etc/logrotate.conf. Still logs under /var/log/* didn't rotate. I then tried removing all configs but the /etc/logrotate.d/rsyslog, re-ran sudo /usr/sbin/logrotate -d /etc/logrotate.conf and still logs under /var/log/* did not rotate. I think I ruled that out as a possibility.

[Admin]

Did you solve this? Try running "logrotate -d /etc/logrotate.conf" and see what it says it's doing. Mine does the same as yours and when I run that it states that it rotates all the /var/log/syslog* files, but nothing actually happens...

[Admin]

If you run logrotate in debug mode (-d parameter), then no changes will be made to the logs (see the logrotate man page).

[Admin]

Your /etc/crontab has these logrotate operations running between 6 and 7AM. Are your computers running then (Wild guess, as I do see you refer to them as "servers")?

[Admin]

Sorry for necroing an old post but I'm facing the same issue. I am running Ubuntu 16.04. I have had this issue for months but didn't notice until a week ago. I tried @Ozgur Batur's solution but that didn't work. I just discovered that that cron and anacron run every day, but it always has the exit code 255. What does this mean? Also, the log says I need to install an MTA to see the exact output of the job. What is an MTA and how do I install one?

[Admin]

Same problem here with Ubuntu 16.04. I couldn't launch logrotate -d because... the logrotate command did not exist!? Checking again, I realized that none of the files in /var/log were rotated. apt-get install logrotate showed cron and logrotate were to be installed. Not sure why these were not part of the default distribution package set, but in case that was the issue, this is a VM installed on Gandi's cloud.