rsyslogd using 197% CPU for the past few days

Question

Is there any good way to use svn pre-commit hooks etc. to keep developers from forgetting to add a file? Specifically I would like the commit to fail if the user has a local to them unversioned file whose path/filename matches a certain regexp pattern (in this particular case rails database migrations). User education (always run svn status & svn diff before committing), shaming, etc. has so far proven insufficient to prevent these sorts of problems from occasionally occurring,

Solution 1

I know this is old now but I had this issue on a Virtual private server I am running. It was Ubuntu server and I believe it was 10.10.

I found this page: https://bugs.launchpad.net/ubuntu/lucid/+source/rsyslog/+bug/523610

And it suggested to me a problem with the kernel so changed it to use the latest (through a web interface, the company is running Xen.) and that solved my problem.

This happened back in July 2011 (a couple of months ago) and I can't remember which version of the kernel I was using but now I'm using version 3.0.0...

Solution 2

On OpenVZ, rsyslog has a tendency to use 100%+ CPU. The following commands would fix the problem:

service rsyslog stop
sed -i -e 's/^\$ModLoad imklog/#\$ModLoad imklog/g' /etc/rsyslog.conf
service rsyslog start

Solution 3

Jesse my go-to "WTF?" debugging method for something like this is strace.

Ctrl+C to exit these:

To see what its doing in real time:

sudo strace -p `pidof rsyslogd`

To see a summary of where it has been spending its time:

sudo strace -c -p `pidof rsyslogd`

There are some useful flags to these that will change their output some. man strace for more info.

Solution 4

This sounds like a bug report (that rsyslogd is not behaving as expected), please can you report it to the bug-tracker so that it can be tracked down and investigated with your help:

• launchpad.net/ubuntu/+source/rsyslog/+filebug

View more solutions

Comments

[Albin Sunnanbo]

Doesn't your continuous integration find that out within a couple of minutes and tell the user?

[Tim]

This is a ruby on rails project => dynamic language so there is no static compilation step. Continuous integration tests can find some of these problems eventually, but it doesn't do much good if the developer also forgets to svn add the new test that exercises the new feature etc. Also with db migrations in particular things like ensuring proper indexing so that performance under load is acceptable etc. are very difficult to test for in fast turnaround integration testing.

[Tim]

"That said, your best bet might simply be a pre-commit hook that aborts if there are any files in the working copy that aren't staged into the repository. " => Is there actually a way to do this? Other poster suggests that this information isn't sent to the server, so I presume it would have to be client side. Have you setup a hook like that before?

[Jesse]

Yeah, this turned out to be the problem. I ended up ditching the server and using Debian instead.

[Torben]

Seems that a fix for this specific bug is on its way. See also the related bug report: bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1366829