Server super slow, RAM & Swap 100% used, ClamAV problem?

Question

My Ubuntu 10.04 server is down since this morning. When I restart it, I can use it for a couple of minutes then the RAM and Swap get full, and SSH doesn't answer. It seems ClamAV is part of the problem. How can I stop it? clamd stop didn't work.
And more globally, how can I stop all email services?

UPDATE: I don't think log files are needed anymore.

UPDATE 2:

wpc2617:~# free -m
                total    used    free  shared    buffers     cached
Mem:           875     868       7       0          0         12
-/+ buffers/cache:     855      20
Swap:          976     867     108
wpc2617:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1 31 971132   8076    696  18988  311 1300  2915  1309  244  200  9  2 11 77

Solution 1

(Converting my comment on question into an answer as it seemed to work - hope that's OK.)

Try killall clamscan, then stop whatever is starting those processes. The "D" under the S column means the processes are in "uninterruptible sleep" which could be due to various reasons but is probably due to paging to/from swap as your memory and swap space are entirely full (from the top and free -m "buffer/cache" line).

Killing the processes may be a challenge (see this answer on uninterruptible processes), so you may need to reconfigure system not to use clamscan, then restart server.

Solution 2

A few comments and things you can check:

• I don't see anything obviously wrong in the log you posted. What makes you think the last line in the log is an issue?
• Near 100% RAM utilization is actually a good thing in general. What you don't want to happen, though, is begin to use swap space which will kill your server's performance pretty quickly and may be the issue hear. Use the top command and ensure that the swap memory is near zero. free -m can also be used to see how your memory is in use along with vmstat.
• Also check your CPU usage with top. If the usage or loads are high that could be another reason.
• If you are being DoS you also may have a fully used network connection (especially on a 10 Mbps line). Check your network usage with ifconfig or your provider's control panel.
• If you cannot seem to catch the issue in real time try installing a basic monitoring package like sar to capture stats that you can given later.

Solution 3

try starting the server and then immediately log in and run top then press M (being actually a capital M to sort by memory usage) and watch your resources. You should see your swap at 0 bytes used and your memory free either fully used as linux can use sometimes or little under but mainly watch the top of the light of processes consuming memory. You should see the pattern of what memory starts to increase if an app has a memory leak. if you see what it is, stop the process/service and see how it goes after 30min if it lasts.

good luck

EDIT: the problem there is that your mail logs will show that amavis cannot find any primary scanners and is most likely fallen back to secondary scanning which in your case is clamscan. clamscan is very slow and starts a process for each mail or file it scans. What you need is clamd and configure amavisd to use it as a primary scanner. If you don't have amavisd being used (though you should), then you are filtering using procmail as this appears to be running as the user account. You need to revise your setup and possibly configure postfix to connect to an amavisd service to best handle scanning.

Comments

[Nabab]

In this screenshot I stopped postfix and the server is still extremely slow. It seems caused by clamscan, but I tried "clamd stop" and it didn't change anything. Apache and MySQL won't start and everything slown down big time

[RichVel]

try killall clamscan then stop whatever is starting those processes. The "D" under the S column means the processes are in "uninterruptible sleep" which could be due to various reasons but is probably due to paging to/from swap (the top also shows all your swap is used.) Killing the processes may be a challenge - see this answer - so you may need to reconfigure system not to use clamscan, then restart server.

[Nabab]

Thanks a million for killall, that's the command I was looking for. I've been able to kill all clamscan, stop postfix, then after uninstalling/reinstalling clamav everything was working fine.

[Nabab]

Thanks, I'll try that as soon as the guys at the hosting company get out of the recovery mode

[Halfgaar]

Kind of depends: near 100% with cache is good, without is not :). If orignal poster does free -m, he/she should look at the "-/+ cache" column.

[BMDan]

@Halfgaar It's a row, not a column.

[Nabab]

So the swap is fully used, that's bad isn't it?

[Nabab]

Thanks, yeah the log was irrelevant in fact. So I have a full swap, I've added the result of free -m and vmstat. I don't know how to use ifconfig but I'd like to find out first how to stop all email services

[Halfgaar]

BMDan: whoops...

[Michael Hampton]

Don't do this. It will only make things worse.