System logs in Mac OSX 10.6

Question

Something is filling up my system log files, to the point that console.app will only show the last half hour of each file. It looks like something is trying to escape its sandbox, but I'm not exactly sure what... I'm getting MANY repeated messages that look like:

Feb  3 00:29:57 Brians-mini sandboxd[16]: syslogd(15) deny file-read-data /private/var/log/asl/StoreData
Feb  3 00:29:57 Brians-mini sandboxd[16]: syslogd(15) deny mach-task-name
Feb  3 00:29:57 Brians-mini sandboxd[16]: syslogd(15) deny file-read-data /private/var/log/asl/StoreData
Feb  3 00:29:57 Brians-mini sandboxd[16]: syslogd(15) deny mach-task-name
Feb  3 00:29:59: --- last message repeated 1 time ---
Feb  3 00:29:57 Brians-mini sandboxd[16]: *** process 16 exceeded 500 log message per second limit  -  remaining messages this second discarded ***
Feb  3 00:29:57 Brians-mini sandboxd[16]: syslogd(15) deny mach-task-name
Feb  3 00:30:00: --- last message repeated 499 times ---
Feb  3 00:29:58 Brians-mini sandboxd[16]: *** process 16 exceeded 500 log message per second limit  -  remaining messages this second discarded ***
Feb  3 00:29:58 Brians-mini sandboxd[16]: syslogd(15) deny mach-task-name

Edit to add:

Activity monitor says that sandboxd is taking up 60% cpu, and syslogd is taking up 120%. Now, I assume that that's PER processor (I'm on a core 2 duo) But that's still an AWFUL lot of cpu for those two processes...

EDIT: var/log/asl as per request:

drwxr-xr-x  25 root     wheel       850 Jan  6 06:41 ./
drwxr-xr-x  47 root     wheel      1598 Feb  4 15:16 ../
-rw-r-----   1 root     admin     11414 Jan  4 11:49 2010.01.04.U0.G80.asl
-rw-------   1 root     wheel       874 Jan  4 09:41 2010.01.04.U0.asl
-rw-------   1 acordex  wheel     43862 Jan  4 17:53 2010.01.04.U501.asl
-rw-r--r--   1 root     wheel     44614 Jan  4 23:42 2010.01.04.asl
-rw-r-----   1 root     admin  10241494 Jan  5 16:53 2010.01.05.U0.G80.asl
-rw-------   1 acordex  wheel    669585 Jan  5 18:11 2010.01.05.U501.asl
-rw-r--r--   1 root     wheel    772889 Jan  5 23:42 2010.01.05.asl
-rw-r-----   1 root     admin      9731 Jan  6 18:54 2010.01.06.U0.G80.asl
-rw-------   1 acordex  wheel    404532 Jan  6 18:50 2010.01.06.U501.asl
-rw-r--r--   1 root     wheel    838013 Jan  6 18:53 2010.01.06.asl
-rw-r-----   1 root     admin     52896 Sep 24 18:20 BB.2010.09.30.U0.G80.asl
-rw-r--r--   1 root     wheel     50908 Sep 29 10:30 BB.2010.09.30.asl
-rw-r-----   1 root     admin     58875 Oct 30 11:18 BB.2010.10.31.U0.G80.asl
-rw-r--r--   1 root     wheel     46188 Oct 30 17:41 BB.2010.10.31.asl
-rw-r-----   1 root     admin     10322 Nov  5 18:21 BB.2010.11.29.U0.G80.asl
-rw-r--r--   1 root     wheel      2159 Nov  4 17:21 BB.2010.11.29.asl
-rw-r-----   1 root     admin      6586 Nov  9 14:06 BB.2010.11.30.U0.G80.asl
-rw-r--r--   1 root     wheel     23147 Nov 25 16:36 BB.2010.11.30.asl
-rw-r-----   1 root     admin     21686 Dec 16 19:06 BB.2010.12.31.U0.G80.asl
-rw-r--r--   1 root     wheel     36951 Dec 23 18:32 BB.2010.12.31.asl
-rw-r--r--   1 root     wheel      2584 Jan  6 16:49 BB.2011.01.31.asl
-rw-r--r--   1 root     wheel        12 Jan  6 18:54 StoreData
-rw-r--r--   1 root     wheel         8 Jan  6 16:59 SweepStore

Solution 1

Looks to me like it's syslogd itself causing the problem -- it's been sandboxed away from its data files, so when it tries to access them it generates a sandbox error, which gets handed to syslogd, which triggers it to try to get at its files again... and this repeats as fast as syslogd and sandboxd can go.

Check the contents of /System/Library/LaunchDaemons/com.apple.syslogd.plist (the launchd item that controls how syslogd is launched). It should have a section like this:

    <key>ProgramArguments</key>
    <array>
<!--
    Un-comment the following lines to run syslogd with a sandbox profile.
    Sandbox profiles restrict processes from performing unauthorized
    operations; so it may be necessary to update the profile
    (/usr/share/sandbox/syslogd.sb) if any changes are made to the syslog
    configuration (/etc/syslog.conf).
-->
<!--
        <string>/usr/bin/sandbox-exec</string>
        <string>-f</string>
        <string>/usr/share/sandbox/syslogd.sb</string>
-->
        <string>/usr/sbin/syslogd</string>
    </array>

Note that in the above example (taken from my Mac), the sandbox wrapper around syslogd is commented out. Is the same true on your Mac? If not, re-add the comment markers, and restart syslogd (you can do this with launchctl, but I'd just reboot the machine).

Another note: I took a look in the sandbox profile, /usr/share/sandbox/syslogd.sb, and it looks (to my inexpert eyes) like it does deny mach-task-name and access to /private/var/log/asl/StoreData -- apparently, Apple hasn't debugged (/updated) the profile to match what syslogd actually needs...

Solution 2

Two different thoughts:

1. Try to identify if there's a particular process spawned by sandboxd which is causing these errors. Run the Activity Monitor app and choose "All Processes, Hierarchically" from the select list at the top of the window. Find sandboxd in the list and see if there are any child processes under it -- child processes will be indented.

2. The log mesages mention /private/var/log/asl. Look at the man pages for aslmanager and asl.conf (config file for aslmanager). There is a setting in the config file for logging level. Perhaps this got bumped to a more detailed level.

Solution 3

Well, the process filling things up is sandboxd (process 16).

As to why it's logging so much, we'd need to see more of its messages to understand the problem. (It looks from this tiny snippet like some program is trying to do something unauthorized -- access the file specified -- but there's not much info here.)

Comments

[Brian Postow]

well, yeah I kind of figured that... and basically those messages (which I need to re-format) are basically the whole thing, repeated OVER AND OVER AND OVER AND OVER.

[Jon Lasser]

Thanks for reformatting the logs -- it helps. Unfortunately, what you have is some application trying to read the system log files in the asl directory that sandboxd is dying permission to -- but sandboxd isn't reporting that specific process ID. If it did, we'd be able to identify this. You might try quitting any applications you're running in your logged in session (including menulets) and see if any of that stops the spew. You could also open the console application and see if any other logs have useful data.

[Brian Postow]

well, the bad news is: No, there's no child process. The good news is that sandboxd is taking up 60% of a processor, which seems ... unlikely...

[ridogi]

Doesn't seem unlikely at all. Something is busy generating a bunch of errors every second, so it stands to reason that whatever is failing will use up some CPU power.

[Brian Postow]

@ridogi: True. it just seems unlikely that this is happening without something else being ... very wrong...

[ridogi]

I wasn't saying it is normal and can be ignored, just that a process that is crashing or constantly trying to write some sort of message to the logs would hog a bunch of cpu. Hard to say what the problem is, but I wrote an answer with some things to look try.

[Brian Postow]

I haven't tried safe mode yet, but rebooting and logging in as another user don't help. ls is posted in the original message so I could format it. I do not believe I have modified either of those files. The files in ASL haven't been updated since Jan 6. and the syslog command doesn't show anything after Jan 6 either.

[Brian Postow]

yeah. asl.conf hasn't been touched in 6 months, and nothing LOOKS weird to me... I'm guessing that the problem is that something is trying to log, but sandboxd is stopping it...

[ridogi]

It seems like ASL is broken on your machine, as opposed to the problem being the mystery process. Leaving the computer on overnight (set not to sleep) may help as the logs should roll over at midnight and the maintenance scripts will run around 3. I don't know what the BB.* files are - I'm not in front of a SL machine but I haven't seen them before. The last one with the 2011 date is disconcerting. Perhaps that one is indicative of the problem. I would move all of the BB.* file out of there and reboot. Did you install anything on the 6th? Perhaps Software Update.log will jog your memory.

[ridogi]

Have you changed habits as far as leaving the computer on all the time vs letting it sleep or turning it off? Reinstalling the 10.6.2 combo updater may help you also. LongTTL.U0.asl and LongTTL.asl are not present in your directory. Can someone else with SL confirm that they should be present still? (I'm on 10.5)

[Brian Postow]

neither rebooting nor logging in as a new user seems to have helped. I quit all user apps... And I don't see anything in the other logs...

[Brian Postow]

oh, and I don't seem to HAVE a console.log in /var/logs.

[ridogi]

It doesn't live there. Can you open it in Console.app? Have you run DiskWarrior?

[Brian Postow]

When I show console in console.app, I see nothing since January 6. I have not tried DiskWarrior, but I have tried DiskUtility, which didn't seem to help

[ridogi]

DiskWarrior is more thorough-it actually rebuilds the directory structure from scratch rather than the basic repair that Disk Utility does. I have seen console become inaccessible from messing with aslmanager. Have you run aslmanager? You can also try running the maintenance scripts with sudo periodic daily weekly monthly, and the application soma-zone.com/BackupLoupe will let you examine the contents of your backups. Perhaps you'll see what you installed on the sixth. Helpfully you're just under a month so the sixth should still be listed as 1 day of backup in Time Machine.

[Brian Postow]

Bing bing bing, we have a winner. yeah, it seems a bit odd that apple would put that commented code in there, and then have it break so badly when you uncomment it... thanks!