Automatically Kill/Restart Process(es) When Memory is Critically Low
Solution 1
The linux kernel has a so-called OOM Killer built-in. It is the "Out of memory killer". So when your box has exhausted its ram & swap, the kernel will start killing stuff to make the server accessible.
You can tweak the priorities of processes, to determine the "likelihood" of a process being killed. Read more at this link, see section "Configuring the OOM Killer".
Basically, you adjust the likelihood in the /proc/*/oom_adj file. Eg. raise the likelihood of killing any of the currently running apache instances?
pgrep apache2 |sudo xargs -I %PID sh -c 'echo 10 > /proc/%PID/oom_adj'
Or lower the likelihood that SSH will get killed:
pgrep sshd |sudo xargs -I %PID sh -c 'echo -17 > /proc/%PID/oom_adj'
Also i recommend completely disabling swap on a server where you have this issue, because swap is so slow that it can grind the server to a virtual standstill, even though theres still swapspace left, thus never triggering the OOM killer.
Solution 2
First of all I would say restart is not a problem of solution and the better way is to found the offending process and why it's consuming high memory. Like mentioned above linux already have OOM mechanism to find the offending process and to kill it to release memory pressure
Other way to find it out using Kdump,configure this parameter vm.panic_on_oom = 1(/etc/sysctl.conf),this will generate vmcore when system is going out of memory.You can find more info about it here
http://people.redhat.com/anderson/crash_whitepaper/
Also limits.conf has lot of limitation,better Solution is to use cgroups to restrict the memory utilization per process
So in /etc/cgconfig.conf you can define the control group like this(here I am restricting my app to use only 256MB of memory)
group test {
memory {
memory.limit_in_bytes = 256m;
}
}
and then in /etc/cgrules.conf I can define that your app utilzation(in your case django can't exceed beyond 256)
*:django memory test/
For more info about cgroup you can refer
But the idea of restarting the app seems to be bad.
Solution 3
If these apps are running inside an apache2
server, you can tune the server. Consider:
- Limit the MaxRequestWorkers (This limits number of workers using memory).
- Limit the MaxConnectionsPerChild (This recycles servers so that they don't consume to much memory. This is useful if the applications are leaking memory.
If your processes are leaking memory, you can use /etc/security/limits.conf
to limit the amount of memory a server can contain. This will prevent servers from growing too large. The same effect can be achieved on a temporary basis using the ulimit
command. It may be best to use ulimit
to discover an appropriate size, and then set those values in the limits.conf
file. If your server supports it, drop a file into /etc/security/limits.d
rather than editing /etc/security/limits.conf
.
Related videos on Youtube
kramer65
Updated on September 18, 2022Comments
-
kramer65 over 1 year
So I'm editing an existing codebase using Django Rest Framework, and I added a new field to a model:
class MyModel(models.Model): id = models.AutoField(primary_key=True) name = models.CharField(max_length=150, null=True) the_new_field = models.IntegerField(null=True, default=None)
I've got a serializer, which is fairly basic:
class MyModelSerializer(serializers.HyperlinkedModelSerializer): class Meta: model = MyModel fields = ( 'id', 'name', )
So to the serializer I simply added the new field:
class MyModelSerializer(serializers.HyperlinkedModelSerializer): class Meta: model = MyModel fields = ( 'id', 'name', 'the_new_field', )
I can call the endpoint using
name
using?name=awesomename
, which actually filters based onawesomename
, but when I call the endpoint using?the_new_field=123456
it simply returns all the records in the database.What am I missing here? How can I make it filter based on this new field?
-
ryekayo over 9 yearsYou should be able to download
tops
. I believe that monitors CPU usage -
Michael Hampton over 9 yearsIt should not be difficult to tune your app server to limit its memory consumption, but you didn't ask about that (and you should have).
-
ewwhite over 9 yearsPlease use Monit or something like Nagios for this.
-
PersianGulf over 9 yearsuse
free -m
command,vmstat
command andtop
command, This is not your answer , but may you can use from this: at first you can replace some application with some other, for example downapache
and installlighttpd
two: down every service and program what you don't need. -
JWL over 9 years@ewwhite monit would allow me to kill the offending process and then restart it, right?
-
Bernardo Duarte almost 4 yearsI've pointed out below a sollution using
django-filter
, but I'm missing your view on this question, can you provide us with it?
-
-
JWL over 9 yearsThanks, but this like the other suggestions before it, doesn't address the issue of how I would then automatically restart the killed process. My intent isn't to eternally suspend an offending app, but to kill it (free resources), then restart it automatically. Maybe monit might help?
-
BillThor over 9 years@nemesisfixx Apache will restart a new server if an existing server gets killed. You shouldn't have to restart anything. In the case of MaxConnectionsPerChild, the server will server X requests and then be replaced by a new one. This can be very effective in dealing with memory leaks in applications.
-
rob almost 4 yearsI don't often favor answers pointing at 3rd party libraries- but this is probably good advice. At this point I'm not even sure why drf ships without
django-filter
. -
Bernardo Duarte almost 4 years@rob I'm adding an example, just trying to make it work locally, but pointing out first to speed up the process. I do also favor more examples than links, it's also the SO guideline.