peak memory measurement of long running process in linux

Question

How do I monitor the peak memory consumed by a process in Linux? This is not a program I can internally modify to measure peak memory usage.

I do not really want detailed measurements, nor do I want them to slow down my program excessively.. so valgrind or anything heavyweight is not what I am looking for... And like other posts earlier [Peak memory usage of a linux/unix process, time -v doesn't seem to report memory for my machine...

I can just run top or ps and extract the memory consumed strings for my process id using a simple script. However, my process runs for about 20-30 minutes so I want to be able to log and get the max. I can tolerate coarse grained samples... every 1-minute or so... Specifically how do I-> 1. fork this simple mem-measure script in zsh? 2. kill it when the process-under-test finishes?

Solution 1

Just use top -n to iterate a specified number of times, and -d to delay between updates. Also you can grab only the output relevant to your process by grepping its pid, like:

top -n 30 -d 60 | grep <process-id>

Read the top manual page for more information

man top

Of course, you can also grab the column you need by using awk.

Solution 2

Actually, what I said before:

"""

try

/usr/bin/time -v yourcommand

that should help. if you use only "time", bash will execute the built-in (that does not have "-v")

"""

does not work (returns 0).

I made the following perl script (that I called smaps):

#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
my $max = 0;
while( open my $f, '<', "/proc/$ARGV[0]/smaps" ) {
  local $/; $_ = <$f>;
  $max = $1 if /Rss:\s*(\d+)/ and $1 > $max;
  open my $g, '>', '/tmp/max';
  say $g $max
}

And then I call it (for instance, to watch qgit's memory usage):

bash -c './smaps $$ & exec qgit'

Use single quotes so the "daughter" shell interprets $$ (that will be the same PID of qgit after the exec). this answer, I tested :-D

HTH

Solution 3

Valgrind with massif should not be too heavy, but, I'd recommend using /proc. You can easily write your own monitor script. Here is mine, for your convenience:

#!/bin/bash

ppid=$$
maxmem=0

$@ &
pid=`pgrep -P ${ppid} -n -f $1` # $! may work here but not later
while [[ ${pid} -ne "" ]]; do
    #mem=`ps v | grep "^[ ]*${pid}" | awk '{print $8}'`
        #the previous does not work with MPI
        mem=`cat /proc/${pid}/status | grep VmRSS | awk '{print $2}'`
    if [[ ${mem} -gt ${maxmem} ]]; then
        maxmem=${mem}
    fi
    sleep 1
    savedpid=${pid}
    pid=`pgrep -P ${ppid} -n -f $1`
done
wait ${savedpid} # don't wait, job is finished
exitstatus=$?   # catch the exit status of wait, the same of $@
echo -e "Memory usage for $@ is: ${maxmem} KB. Exit status: ${exitstatus}\n"

Solution 4

This depends on which kind of memory you want to monitor.

Monitoring the following M.a.p.d by sorting the number of all the process's one (not all of the thread's) will let you monitor the malloc physical memory each process uses.

You can write a c program to make it even faster but I thought awk was the minimum choice for this purpose.

• M.a anonymous mapped memory
  • .p private
    • .d dirty == malloc/mmapped heap and stack allocated and written memory
    • .c clean == malloc/mmapped heap and stack memory once allocated, written, then freed, but not reclaimed yet
  • .s shared
    • .d dirty == there should be none
    • .c clean == there should be none
• M.n named mapped memory
  • .p private
    • .d dirty == file mmapped written memory private
    • .c clean == mapped program/library text private mapped
  • .s shared
    • .d dirty == file mmapped written memory shared
    • .c clean == mapped library text shared mapped

I would prefer to get the numbers as follows to get the real numbers in least overhead.
You have to sum these up in order to divide what ps shows as RSS and get more accurate numbers not to confuse.

M.a.p.d:

 awk '/^[0-9a-f]/{if ($6=="") {anon=1}else{anon=0}} /Private_Dirty/{if(anon) {asum+=$2}else{nasum+=$2}} END{printf "sum=%d\n",asum}' /proc/<pid>/smaps

M.a.p.c:

 awk '/^[0-9a-f]/{if ($6=="") {anon=1}else{anon=0}} /Private_Clean/{if(anon) {asum+=$2}else{nasum+=$2}} END{printf "sum=%d\n",asum}' /proc/<pid>/smaps

M.n.p.d:... and so on

Solution 5

Accurate memory metrics can be given by pagemap kernel interface - utilized in libpagemap library https://fedorahosted.org/libpagemap/. Library provides also userspace utils so you can start monitor memory immediately.

View more solutions

Comments

[badkya]

Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 255315 Voluntary context switches: 1211 Involuntary context switches: 1232 Swaps: 0 File system inputs: 0 File system outputs: 68472 Socket messages sent: 0 Socket messages received: 0 Page size (bytes): 4096

[gcb]

this does not account for memory mapped files, which really does not consume memory like resident memory.