How to figure out memory leaks?

linux process memory nginx

8,691

Nothing to see with what you have provided of information. Your free -m just shows that you have only used around half of your memory. Have you checked your HHVM and nginx logs to see if this is the actual problem?

You could make a small script like this that will take the output of ps aux and append it to a file every 30 second (or more). Let it run in the background in a period and analyse the memory output after. Various UNIX tools can be used on the file with the output afterwards for great analysis opportunities i think.

The script could look like this:

while [ true ]
do
  printf "\n\n" >> /root/psaux_analysis
  echo "--------------------------------------" >> /root/psaux_analysis
  date >> /root/psaux_analysis
  echo "--------------------------------------" >> /root/psaux_analysis
  printf "\n" >> /root/psaux_analysis
  ps aux >> /root/psaux_analysis
  sleep 30;
done

This will give you output similar to the following:

--------------------------------------
Mon Jun 16 07:44:03 UTC 2014
--------------------------------------

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2028   584 ?        Ss   Mar12   1:08 init [2]      
root         2  0.0  0.0      0     0 ?        S    Mar12   0:00 [kthreadd/146]
root         3  0.0  0.0      0     0 ?        S    Mar12   0:00 [khelper/146]
daemon     313  0.0  0.0   1804   512 ?        Ss   Mar12   0:00 /sbin/portmap
daemon     528  0.0  0.0   2156   304 ?        Ss   Mar12   0:00 /usr/sbin/atd
104        551  0.0  0.0   2580   240 ?        Ss   Mar12   0:00 /usr/bin/dbus-daemon --system
root       560  0.0  0.0   2288   732 ?        Ss   Mar12   0:28 /usr/sbin/cron
root       727  0.0  0.0  29576  4128 ?        Sl   Mar12  38:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
...

8,691

Slevin

Updated on September 18, 2022

Comments

Slevin over 1 year

I'm running Ubuntu 14.04 on Digitalocean to host my blog. I'm using HHVM + nginx + MySQL. Every few hours (~12h), there's no more RAM left and nginx throws an 502 error.

I check the memory usage for the last 10 minutes with free -m and that's what I get:

             total       used       free     shared    buffers     cached
Mem:           994        714        279         29         20        235
-/+ buffers/cache:        458        535
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        715        278         29         20        235
-/+ buffers/cache:        459        534
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        722        271         29         20        240
-/+ buffers/cache:        461        532
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        729        264         29         20        240
-/+ buffers/cache:        469        524
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           994        725        268         29         20        240
-/+ buffers/cache:        464        529
Swap:            0          0          0

So how can I figure out where the memory leak comes from?

The htop output:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 2107 root       20   0 26284  2528  1408 R  0.7  0.2  0:00.47 htop
 1002 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.99 /usr/sbin/mysqld
 1088 redis      20   0 36996  7256   876 S  0.0  0.7  0:06.46 /usr/bin/redis-server 127.0.0.1:6379
  925 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:17.87 /usr/sbin/mysqld
  949 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.21 /usr/sbin/mysqld
    1 root       20   0 33492  2768  1400 S  0.0  0.3  0:01.30 /sbin/init
  343 root       20   0 19608   648   460 S  0.0  0.1  0:00.12 upstart-udev-bridge --daemon
  351 messagebu  20   0 39228  1292   884 S  0.0  0.1  0:00.07 dbus-daemon --system --fork
  354 root       20   0 51476  1552   952 S  0.0  0.2  0:00.05 /lib/systemd/systemd-udevd --daemon
  399 root       20   0 43452  1756  1396 S  0.0  0.2  0:00.00 /lib/systemd/systemd-logind
  427 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.01 rsyslogd
  428 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.00 rsyslogd
  429 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.02 rsyslogd
  426 syslog     20   0  249M  1388   788 S  0.0  0.1  0:00.03 rsyslogd
  452 root       20   0 15408   648   320 S  0.0  0.1  0:00.05 upstart-file-bridge --daemon
  672 root       20   0 15392   692   332 S  0.0  0.1  0:00.05 upstart-socket-bridge --daemon
  810 root       20   0 15820   928   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty4
  816 root       20   0 15820   928   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty5
  821 root       20   0 15820   932   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty2
  822 root       20   0 15820   924   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty3
  825 root       20   0 15820   932   768 S  0.0  0.1  0:00.00 /sbin/getty -8 38400 tty6
  855 root       20   0 61364  3008  2328 S  0.0  0.3  0:00.03 /usr/sbin/sshd -D
  858 root       20   0  4368   656   512 S  0.0  0.1  0:00.00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
  859 root       20   0 23656  1024   768 S  0.0  0.1  0:00.03 cron
  861 daemon     20   0 19140   160     0 S  0.0  0.0  0:00.00 atd
  894 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
  896 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
  873 whoopsie   20   0  327M  3584  2424 S  0.0  0.4  0:00.00 whoopsie
 1170 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:00.59 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1357 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:23.19 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1437 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:12.09 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1438 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:19.50 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1439 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:14.87 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1440 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:14.93 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1441 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:13.68 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1442 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:11.29 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1443 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:16.50 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
 1444 www-data   20   0 1541M  390M 91108 S  0.0 39.3  0:17.66 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
  916 www-data   20   0 1541M  390M 91108 S  0.0 39.3  2:25.37 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemo
  945 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.18 /usr/sbin/mysqld
  946 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.73 /usr/sbin/mysqld
  947 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.23 /usr/sbin/mysqld
  948 mysql      20   0  737M 72680  7332 S  0.0  7.1  0:00.22 /usr/sbin/mysqld
  950 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.26 /usr/sbin/mysqld
  951 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.23 /usr/sbin/mysqld
  952 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.21 /usr/sbin/mysqld
  955 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.19 /usr/sbin/mysqld
  956 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.17 /usr/sbin/mysqld
 1003 mysql      20   0  737M 66360  7332 S  0.7  6.5  0:01.61 /usr/sbin/mysqld
 1004 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.10 /usr/sbin/mysqld
 1005 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.00 /usr/sbin/mysqld
 1168 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:00.00 /usr/sbin/mysqld
 1173 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.08 /usr/sbin/mysqld
 1202 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.37 /usr/sbin/mysqld
 1445 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:02.37 /usr/sbin/mysqld
 1446 mysql      20   0  737M 66360  7332 S  0.0  6.5  0:03.37 /usr/sbin/mysqld
  957 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.96 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  958 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.97 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  959 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.91 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  960 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.94 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  961 memcache   20   0  357M 42368   876 S  0.0  4.2  0:00.00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  941 memcache   20   0  357M 42368   876 S  0.0  4.2  0:04.53 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
  962 root       20   0 90268  1440   228 S  0.0  0.1  0:00.00 nginx: master process /usr/sbin/nginx
  963 www-data   20   0 91172  2840  1072 S  0.0  0.3  0:01.47 nginx: worker process
 1064 root       20   0 25344  1556  1276 S  0.0  0.2  0:00.04 /usr/lib/postfix/master
 1069 postfix    20   0 27460  1516  1232 S  0.0  0.1  0:00.00 qmgr -l -t unix -u
 1091 redis      20   0 36996  7256   876 S  0.0  0.7  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
 1092 redis      20   0 36996  7256   876 S  0.0  0.7  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
 1151 root       20   0 15820   932   768 S  0.0  0.1  0:00.01 /sbin/getty -8 38400 tty1
 1859 postfix    20   0 27408  1488  1212 S  0.0  0.1  0:00.00 pickup -l -t unix -u -c
 1984 root       20   0  103M  4220  3224 S  0.0  0.4  0:00.11 sshd: root@pts/0
 2054 root       20   0 22764  3876  1676 S  0.0  0.4  0:00.05 -bash

phemmer almost 10 years

Please read linuxatemyram.com. If you want help troubleshooting your nginx issue, you should provide information on that (logs, error message, system memory profile at the time of the issue, etc).
Alen Milakovic almost 10 years

valgrind is the usual (free) tool to use to check for memory leaks.
S edwards almost 10 years

Also you should not disable completely the swap it is sometime useful

Slevin almost 10 years

You're right, half of memory is free at this time. But you can see, that memory usage gets higher from minute to minute. Nginx logs doesn't help. There are just the upstream errors when the website is already down. And I'm using HHVM and not php-fpm, but the error logs are clear. Maybe I have to increase the log level.
Adionditsak almost 10 years

Your memory in M goes 535 -> 534 -> 532 -> 524 -> 529. So jumps up again in the last one? It seems like. This is the number from the free row in buffer/cache line, which is the interesting one. I would like to hear if you found out what was going on. Good idea with the log level, if possible.
Slevin almost 10 years

Thanks! I will run it for an hour and post the results on pastebin.
Slevin almost 10 years

Pastebin is under heavy load, so here are the results: gopeter.de/transfer/psaux_analysis.txt . I used blitz.io for a stress test and reconfigured the nginx.conf according to some guides. At the moment, ~ 250-300 mb of memory are always free.
Adionditsak almost 10 years

Seems stable enough in that hour. Did you experience a mem leak there? Otherwise set it to sleep maybe for 1-2 minutes and let it run for a day. Mysql, Memcached, and other services are very stable. HHVM is the one spanning the most from 43% - 53% circa. Seems ok tho, so you need to have it running when a leak is actually happening.