Linux: find out what process is using all the RAM?
Solution 1
On Linux in the top
process you can press <
key to shift the output display sort left. By default it is sorted by the %CPU
so if you press the key 4 times you will sort it by VIRT
which is virtual memory size giving you your answer.
Another way to do this is:
ps -e -o pid,vsz,comm= | sort -n -k 2
should give you and output sorted by processes virtual size.
Here's the long version:
ps --everyone --format=pid,vsz,comm= | sort --numeric-sort --key=2
Solution 2
Show the processes memory in megabytes and the process path.
ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n
Solution 3
Just a side note on a server showing the same symptoms but still showing memory exhaustion. What ended up finding was a sysctl.conf from a box with 32 GB of RAM and setup for a DB with huge pages configured to 12000. This box only has 2 GB of RAM so it was assigning all free RAM to the huge pages (only 960 of them). Setting huge pages to 10, as none were used anyway, freed up all of the memory.
A quick check of /proc/meminfo to look for the HugePages_ settings can be a good start to troubleshooting at least one unexpected memory hog.
Solution 4
Make a script called show-memory-usage.sh
with content:
#!/bin/sh
ps -eo rss,pid,user,command | sort -rn | head -$1 | awk '{ hr[1024**2]="GB"; hr[1024]="MB";
for (x=1024**3; x>=1024; x/=1024) {
if ($1>=x) { printf ("%-6.2f %s ", $1/x, hr[x]); break }
} } { printf ("%-6s %-10s ", $2, $3) }
{ for ( x=4 ; x<=NF ; x++ ) { printf ("%s ",$x) } print ("\n") }
'
Make it executable with chmod +x show-memory-usage.sh
and call it like this ./show-memory-usage.sh 10
(10 => show max 10 lines)
Output Example:
5.54 GB 12783 mysql /usr/sbin/mysqld
1.02 GB 27582 root /usr/local/cpanel/3rdparty/bin/clamd
131.82 MB 1128 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start inventory.mass.update --single-thread --max-messages=10000
131.21 MB 1095 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start product_action_attribute.update --single-thread --max-messages=10000
131.19 MB 1102 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start product_action_attribute.website.update --single-thread --max-messages=10000
130.80 MB 1115 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start exportProcessor --single-thread --max-messages=10000
130.69 MB 1134 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start inventory.reservations.update --single-thread --max-messages=10000
130.69 MB 1131 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start inventory.reservations.cleanup --single-thread --max-messages=10000
130.69 MB 1107 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start codegeneratorProcessor --single-thread --max-messages=10000
130.58 MB 1120 company+ /opt/cpanel/ea-php73/root/usr/bin/php /home/companyde/redesign.company.de/bin/magento queue:consumers:start inventory.source.items.cleanup --single-thread --max-messages=10000
Solution 5
In my case the issue was that the server was a VMware virtual server with vmw_balloon
module enabled:
$ lsmod | grep vmw_balloon
vmw_balloon 20480 0
vmw_vmci 65536 2 vmw_vsock_vmci_transport,vmw_balloon
Running:
$ vmware-toolbox-cmd stat balloon
5189 MB
So around 5 GB of memory was in fact reclaimed by the host. So despite having 8 GB to my VM "officially", in practice it was much less:
$ free
total used free shared buff/cache available
Mem: 8174716 5609592 53200 27480 2511924 2458432
Swap: 8386556 6740 8379816
Related videos on Youtube
Tim
Software Developer and Agile Project Manager with 15 years of experience in Software Engineering.
Updated on September 18, 2022Comments
-
Tim over 1 year
Before actually asking, just to be clear: yes, I know about disk cache, and no, it is not my case :) Sorry, for this preamble :)
I'm using CentOS 5. Every application in the system is swapping heavily, and the system is very slow. When I do
free -m
, here is what I got:total used free shared buffers cached Mem: 3952 3929 22 0 1 18 -/+ buffers/cache: 3909 42 Swap: 16383 46 16337
So, I actually have only 42 Mb to use! As far as I understand,
-/+ buffers/cache
actually doesn't count the disk cache, so I indeed only have 42 Mb, right? I thought, I might be wrong, so I tried to switch off the disk caching and it had no effect - the picture remained the same.So, I decided to find out who is using all my RAM, and I used
top
for that. But, apparently, it reports that no process is using my RAM. The only process in my top is MySQL, but it is using 0.1% of RAM and 400Mb of swap. Same picture when I try to run other services or applications - all go in swap,top
shows that MEM is not used (0.1% maximum for any process).top - 15:09:00 up 2:09, 2 users, load average: 0.02, 0.16, 0.11 Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4046868k total, 4001368k used, 45500k free, 748k buffers Swap: 16777208k total, 68840k used, 16708368k free, 16632k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND 3214 ntp 15 0 23412 5044 3916 S 0.0 0.1 0:00.00 17m ntpd 2319 root 5 -10 12648 4460 3184 S 0.0 0.1 0:00.00 8188 iscsid 2168 root RT 0 22120 3692 2848 S 0.0 0.1 0:00.00 17m multipathd 5113 mysql 18 0 474m 2356 856 S 0.0 0.1 0:00.11 472m mysqld 4106 root 34 19 251m 1944 1360 S 0.0 0.0 0:00.11 249m yum-updatesd 4109 root 15 0 90152 1904 1772 S 0.0 0.0 0:00.18 86m sshd 5175 root 15 0 90156 1896 1772 S 0.0 0.0 0:00.02 86m sshd
Restart doesn't help, and, by they way is very slow, which I wouldn't normally expect on this machine (4 cores, 4Gb RAM, RAID1).
So, with that - I'm pretty sure that this is not a disk cache, who is using the RAM, because normally it should have been reduced and let other processes to use RAM, rather then go to swap.
So, finally, the question is - if someone has any ideas how to find out what process is actually using the memory so heavily?
-
Hackeron over 8 yearsDid you ever find the answer to this?
-
Deltik over 8 years@Hackeron: OP accepted this answer. I know that answer doesn't address your question, though. I was able to reproduce your issue on one of my servers, and I'm currently researching if there is a way to troubleshoot it.
-
Hackeron over 8 years@Deltik Ah, ok. Thank you :) - I have 2 servers here that leak all available memory in the space of around 12 hours, let me know if there is anything I can do to help diagnose this. I'm reachable as the nickname "hackeron" on IRC (irc.freenode.org).
-
Deltik over 8 years@Hackeron: I wasn't able to find you as "hackeron" on
irc.freenode.org
. I did create a chatroom for extended discussion here. -
kqr over 5 yearsWorth noting that the ZFS in-memory ARC (and/or L2ARC) cache does not show in
free -m
, but the size of it can be queried on Linux withcat /proc/spl/kstat/zfs/arcstats | grep data_size
. -
gaoithe over 4 yearsIn top hit "M" to sort by memory used. You want to look at RES and used for used memory and not at VIRT and free which can be deceptive. See the classic linuxatemyram.com
-
-
Karlson about 12 years@OliverSalzburg The issue is
-o
options. RHEL4 this works. RHEL5:ps -e -o pid,vsz,comm= | sort -n -k 2
works. I'll try 11.10 later tonight but if you find the right sort options before please let me know.ps -e -o pid,vsz,comm | sort -n -k 2
might work but I don't have a place to verify at the moment. -
Oliver Salzburg about 12 yearsI'm not really familiar with the
-ef
option. But this seems to produce reasonable output:sudo ps axo pid,vsz,comm=|sort -n -k 2
-
Karlson about 12 years@OliverSalzburg Sorry. Amended (thought I changed it already). It should be
ps -e
orps -a
-
SSH This almost 12 yearsTy, I like the top suggestion of
<
I didn't know that was possible, fedora -
fields almost 10 yearsI recently had another server where this was the problem. If your organization has ex-Oracle employees in it, this setting may be your culprit.
-
KingsInnerSoul over 8 yearsOut of curiosity, what is the correct way to escape this command? It shows END ocne i reach the last line, it does not kill the process when i Ctrl+C it.
-
enobayram over 8 years@KingsInnerSoul press 'q'
-
fixer1234 over 8 yearsWelcome to Super User. Can you expand your answer to explain what this code does and how it addresses the problem? Unexplained code is discouraged, because it doesn't teach the solution. Thanks.
-
John about 8 yearsI'm surprised this answer is downvoted and has a comment asking to explain it.. it's short enough that it should be clear what it does (pipes ps aux into awk and then sort), and in the context of the question, it shows which processes are using the most RAM. I think it's a fine answer.
-
user over 7 yearsWhy? What does this do? How does it work? Don't tell people to run random code; explain its purpose and how it works.
-
Dooley_labs over 7 yearsFigure I'll explain the code for those that don't understand as it appears to be safe to run, but the downvote may ward off those it would be useful towards. It's running the same command as is in above answers, but it's adding formatting with AWK. I've not personally run the script as I have no use for it, but explaining it helps those in need of some formatting.
-
sengs over 5 yearsSlightly modified version to get the processes that occupy RAM and shows the full command:
ps -e --format=pid,rss,args | sort --numeric-sort --key=2
-
Stéphane Gourichon about 5 yearsI've read the code and run it. It aligns fields like a table, and formats consumed resident memory with prefixes (like 1.12 GB, 582.79 MB).
-
gaoithe over 4 yearsYou do not want to sort by vsz but by rss as @sengs shows. rss shows actual USED memory not VIRTUAL memory which will be given back by the system.
-
Black over 3 yearsI added an output example and improved the script to make it accept an arguemnt for head.
-
mgutt over 3 yearsMy system used multiple GB of RAM which was not listed by this command. I checked with
free -h
it was under "used" (and missing under "available") which usually means its not used for caching, but executingsync; echo 3 > /proc/sys/vm/drop_caches
freed it. Are "dentries and inodes" reserving RAM? As far as I know the "page cache" uses only "available" RAM and does not really reserve something. -
roachsinai over 2 years@Black Thanks for your post, and how to remove the blank lines between each item?
-
alper over 2 yearsis it possible to ignore
0 MB
processes? -
Ciprian Tomoiagă over 2 years@alper pipe through
grep -v '^0 MB'
.^0 MB
selects lines starting with0 MB
and-v
inverses the matching -
michaelkrieger about 2 yearsFor ease of reading, doing some number formatting sure makes this easier to use
ps aux | awk '{printf "%8.3f MB\t\t%s\n", $6/1024, $11}' | sort -n | grep -v '^ 0.000 MB'
Explanation: * process list including all processes on the system * displays the process memory, in MB, rounding to a consistent 3 decimal places and left-padding spaces so that the string is 8 characters so the decimals all line up and the process name. * sorting by the first key, numerically * removing lines starting with " 0.000 MB" (including the padded spaces)