How to breakdown 'Shmem' RAM usage in Linux
For detailing shmem
memory usage (and more), you have got the ipcs
command.
From man ipcs
NAME ipcs - show information on IPC facilities
SYNOPSIS ipcs [options]
DESCRIPTION ipcs shows information on the inter-process communication facilities for which the calling process has read access. By default it shows information about all three resources: shared memory segments, message queues, and semaphore arrays.
As an example, I am running it here in an Apache server:
$ipcs
------ Message Queues --------
key msqid owner perms used-bytes messages
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0xf4123d98 20643870 root 600 3294 12
0xfb123d9b 20676639 root 600 1242 12
0x0a123d9d 20709408 root 600 3294 12
0x91123da0 20742177 root 600 1242 12
0x75123da2 20774946 root 600 3294 12
0xbc123da5 20807715 root 600 1242 12
0x3d123da7 20840484 root 600 3294 12
0x84123daa 20873253 root 600 1242 12
0x47123dac 20906022 root 600 3294 12
0x0e123daf 20938791 root 600 1242 12
0xa6123db1 20971560 root 600 3294 12
0x2d123db4 21004329 root 600 1242 12
0x77123db6 21037098 root 600 3294 12
0x3e123db9 21069867 root 600 1242 12
0x80123dbb 21102636 root 600 3294 12
0x87123dbe 21135405 root 600 1242 12
0xa4123dc0 21168174 root 600 3294 12
0xab123dc3 21200943 root 600 1242 12
0xfd123dc5 21233712 root 600 3294 12
0x44123dc8 21266481 root 600 1242 12
0x96123dca 21299250 root 600 3294 12
0x1d123dcd 21332019 root 600 1242 12
0xa4123dcf 21364788 root 600 3294 12
0xab123dd2 21397557 root 600 1242 12
0x7a123dd4 21430326 root 600 3294 12
0x01123dd7 21463095 root 600 1242 12
0xd9123dd9 21495864 root 600 3294 12
0x20123ddc 21528633 root 600 1242 12
0x86123dde 21561402 root 600 3294 12
0x0d123de1 21594171 root 600 1242 12
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 425984 www-data 600 1
0x00000000 1441793 www-data 600 1
0x00000000 1474562 www-data 600 1
Related videos on Youtube
Chris Hunt
Updated on September 18, 2022Comments
-
Chris Hunt almost 2 years
I am debugging an issue with a game that is consuming large amounts of memory, essentially all of 24GB available RAM and 8GB of swap is gone a few minutes after start. The memory is not associated with the process itself (according to RSS in
top
), but seems to be categorized underShmem
in/proc/meminfo
. This is based on recording the values as this process is executing, which you can see in the graph here (raw data here).Right now I am trying to break down
Shmem
into something more specific. I have looked at the post here which mentions the following are contained underShmem
:- tmpfs memory
- SysV shared memory
- POSIX shared memory
- shared anonymous mappings
- pages used for GEM
I quickly put together this script:
import glob import os import re import psutil shmem_users = [] # tmpfs memory disks = psutil.disk_partitions(all=True) paths = [d.mountpoint for d in disks if d.fstype == 'tmpfs'] tmpfs_usage = sum(psutil.disk_usage(p).used for p in paths) shmem_users.append({ 'name': 'tmpfs', 'size': tmpfs_usage }) # /dev/shm dev_shm_size = 0 for p in glob.glob('/dev/shm/*'): try: sz = os.stat(p).st_size dev_shm_size += sz except: # e.g. permission errors pass shmem_users.append({ 'name': '/dev/shm', 'size': dev_shm_size }) # TODO: SysV shared memory # TODO: shared anonymous mappings # TODO: GEM allocated pages with open('/proc/meminfo') as f: m = re.search(r'Shmem: *(?P<kbs>\d+)', f.read()) if not m: raise RuntimeError('Could not find Shmem value.') total_shmem = int(m.group('kbs')) * 1024 print('Total shmem: {}'.format(total_shmem)) total_shmem_accounted_for = sum(u['size'] for u in shmem_users) print('Total shmem found: {}'.format(total_shmem_accounted_for)) print('Shmem unknown: {}'.format(total_shmem - total_shmem_accounted_for))
I have the process currently
SIGSTOP
ped part-way through eating my RAM, and it returns:Total shmem: 8450904064 Total shmem found: 681805504 Shmem unknown: 7769098560
So still 7 GBs unaccounted for, hardly closer to understanding what is using up memory.
I have been looking at kernel code for a few hours but have made very little headway in figuring out 2, 4, and 5. I can provide notes if needed. My question: Are these the only other values that could be categorized as "shared memory", and how can I get their specific values?
If it matters my system is:
- Ubuntu 16.04.2 LTS (with KDE 5.5.5)
- 24 GB RAM (+8 GB of swap configured)
- Intel Core i3-6100
- 500GB SSD
- Sapphire Radeon RX470 4GB (with AMDGPU-PRO driver, v17.10)
-
user2948306 over 5 yearsI did some more /proc spelunking in my own question here. It gets a little bit further, but I still can't account for over 50% of mine (running GNOME on Wayland). My script does not find GEM buffers if they are not currently mmap()ed, so that might be my problem. unix.stackexchange.com/questions/482795/…
-
Chris Hunt about 7 yearsThank you. This covers Shmem source #2 (SysV shm). Maybe it's too much to ask for all of this in one question...