mod_python with Apache Failures

Question

I'm having an issue with mod_python and Apache, and I'm pretty certain I know what the problem is, but I'd like a bit of reassurance. I currently have a situation where I'm working with a very small VM with just 256M of RAM. Now MySQL and Apache run fairly well, and everything is fine and dandy, until I start hitting swap, which I obviously want to avoid as much as possible. With that little RAM though it's just about impossible. I've opted to cron up restart of MySQL and Apache a few times a day, however, I receive a no disk space left on device after about 4-5 days:

[Mon May 11 06:00:14 2009] [notice] caught SIGTERM, shutting down
[Mon May 11 06:00:20 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon May 11 06:00:20 2009] [notice] Digest: generating secret for digest authentication ...
[Mon May 11 06:00:20 2009] [notice] Digest: done
[Mon May 11 06:00:21 2009] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Mon May 11 06:00:21 2009] [error] (28)No space left on device: mod_python: Failed to create global mutex 1 of 4 (/tmp/mpmtx265021).

My initial though was a lack of semaphores, but that didn't seem to be the case. At least the tricks I've used to circumvent that in the past hasn't worked thus far. I've seen similar issues with people using mod_python, but I haven't found much in the way of actual fixes. Are my restarts causing this, or is this just mod_python being buggy? Thanks!

Solution 1

I've had similar issues when repeatedly restarting Apache on my laptop (for me it took about 40 restarts); the only solution I found was to restart the computer entirely. Of course, my laptop has 2GB of RAM and a lot more disk space than your server probably does, so you might run into this problem more quickly... basically, I'd avoid restarting Apache too many times without rebooting the server entirely.

Solution 2

1. remove leaked semaphores that are no longer used

   ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
   

2. increase the available semaphores

   echo “kernel.sem = 512 32000 100 512″ >> /etc/sysctl.conf; sysctl -p
   

Solution 3

Googling around, I've found this solution:

You actually need to increase the amount of SysV Semaphores on your system.

I use (in /etc/sysctl.conf):
kernel.sem = 512 32000 100 512
(set the options in this file by running sysctl -p).

BTW. mod_python is a bit obsolete, since introduction of WSGI. Most modern applications can be run using mod_wsgi.

Solution 4

I'll assume this is done on a linux system.

I'm just wondering if you might be seeing some shared memory issue as well as the semaphores? I've had that happen to me when apache is killed via kill -9.

When this happens have you checked out the status of the shared memory area using the command: ipcs-a This will show the current use of semaphores, message queues, and shared memory. This may provide you with more information on the problem.

You actually can if you want to remove old semaphores or shared memory regions using the ipcrm command (see man page for specific options you need).

I can't recall at the moment, but I remember it is relatively easy in Linux to modify the semaphore and shared memory parameters.

Well just some thoughts on your issue. Hope it helps.

View more solutions

Comments

[f4nt]

Yeah, I hear ya. I've been rebooting, and I'd like to find a way around that..

[David Z]

Well, I personally would say just don't restart Apache so often ;-) I also run a mod_python-based site on a 256MB server (VPS), I just don't restart Apache except when I update the code (can be every few dsys or weeks), and my site runs fine. As long as the server isn't actively swapping a lot of pages back and forth between disk and RAM, I don't see swap usage as a major problem.