htop reporting much higher memory usage than free or top

Question

The following three outputs were taken essentially simultaneously:

top:

top - 02:54:36 up 2 days, 13:50,  3 users,  load average: 0.05, 0.05, 0.09
Tasks: 181 total,   1 running, 179 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.5 us,  0.8 sy,  0.0 ni, 96.6 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16158632 total, 11234480 used,  4924152 free,      844 buffers
KiB Swap: 16777212 total,        0 used, 16777212 free, 10640832 cached

free -h:

             total       used       free     shared    buffers     cached
Mem:           15G        10G       4.7G         0B       844K        10G
-/+ buffers/cache:       578M        14G
Swap:          15G         0B        15G

htop:

free and top seem to agree. In top there is 11234480 KiB used, subtracting 10640832 KiB cached gives 579.7 MiB, which is pretty close to what free reports under used +/- buffers/cache.

However htop is reporting 1836 (MiB) used, which is neither here nor there as far as I can see. Where does this difference come from? htop is clearly not including the cached data, but it is still reporting more than three times the memory usage of free or top.

I am aware that there are many similar questions, but I haven't come across one that explains this discrepancy (the confusion usually seems to be only the with/without cache counting).

Edit: I should mention that I am running openSUSE, and I see the same kind of discrepancy in both version 12.2 and 12.3 RC1.

Edit2: The included version of htop is 1.0.1. I have also compiled version 1.0.2 from source and see the same discrepancy then as well.

Solution

A complete re-write of my previous post. Got a bit curious and checked out further.

In short: the reason for the difference is that openSUSE uses a patched version of top and free that adds some extra values to `cached'.

---

A) Standard version

top, free, htop, ...:

Usage is calculated by reading data from /proc/meminfo:

E.g.:

#free:
Row   Column  | Corresponding /proc/meminfo entry
-----|--------|----------------------------------
Mem:

      total   : MemTotal
      used    : MemTotal - MemFree
      free    : MemFree
      shared  : MemShared
      buffers : Buffers
      cached  : Cached
-----|--------|----------------------------------
-/+ buffers/cache:

      used    : (MemTotal - MemFree) - (Buffers + Cached)
      free    :  MemFree             + (Buffers + Cached)

#htop:
    Used U*   : ((MemTotal - MemFree) - (Buffers + Cached)) / 1024

^*_{I'm using the name Used U for memory used by User Mode. Aka Used minus (Cached + Buffers).}

So in reality same calculation is used.

htop display the following in the memory meter:

[Used U % of total | Buffers % of total | Cached % of total ] UsedU MB

(MB is actually MiB.)

---

B) Patched version

The base for free and top on Debian, Fedora, openSuse is is procps-ng. However, each flavour add their own patches that might, or migh not become part of the main project.

Under openSUSE we find various additions to the top/free (procps) package. The ones to take notice of here is some additional values used to represent the cache value. (I did not include these in my previous post as my system uses a "clean" procps.)

B.1) Additions

In /proc/meminfo we have Slab which is in-kernel data structure cache. As a sub category we find SReclaimable which is part of Slab that might be reclaimed for other use both by Kernel and User Mode.

Further we have SwapCached which is memory that once was swapped out, is swapped in but also is in the swap-file. Thus if one need to swap it out again, this is already done.

Lastly there is NFS_Unstable which are pages sent to the server but not yet committed to stable storage.

The following values are added to cache in the openSUSE patched version:

SReclaimable
SwapCached
NFS_Unstable

(In addition there are some checks that total has to be greater then free, used has to be greater then buffers + cache etc.)

B.2) Result

Looking at free, as a result the following values are the same: total, used, free and buffers.

The following are changed: cached and "+/- buffers".

used    = MemTotal - MemFree

old:
    cached         : Cached
    +-/buffers-used: used - (Buffers + Cached)
    +/-buffers-free: free + (Buffers + Cached)

patched:
    cached         : Cached + SReclaimable + SwapCached + NFS_Unstable
    +/-buffers-used: used - (Buffers + Cached + SReclaimable +
                     SwapCached + NFS_Unstable)
    +/-buffers-free: free + (Buffers + Cached + SReclaimable +
                     SwapCached + NFS_Unstable)

The same additions are done to top.

htop is unchanged and thus only aligning with older / or un-patched versions of top / free.

Comments

[Admin]

just guessing here but perhaps it adds up memory shared between processes?

[Quantumboredom]

The second snippet does not work (the second pipe with grep -A12 ... does not return anything). But right now memo1 (i.e., htop) shows the memory as "[...] 2087/15779MB" while free -m shows 825 under +/- buffers/cache. I'll update my question with OS and so on.

[Runium]

Yes, the second snipet is a bad hack (trying to parse ncurses output is funny at best). It works here, but I'm not surprised it doesn't work elsewhere.