Shared library in containers

Question

For two processes A and B, the both use the library libc.so, libc.so is loaded into memory only once. This is a normal situation when A and B both run on the same host and the same rootfs.

When it comes to container, if A and B are running in different containers, are A and B sharing same memory area?

for example

imageA

--libc.so

--programA

imageB

--libc.so

--programB

we use chroot to run A and B in different rootfs. The two libc.so are same. Will libc.so be loaded into memory twice?

Solution 1

Actually, processes A & B that use a shared library libc.so can share the same memory. Somewhat un-intuitively it depends on which docker storage driver you're using. If you use a storage driver that can expose the shared library files as originating from the same device/inode when they reside in the same docker layer then they will share the same virtual memory cache pages. When using the aufs, overlay or overlay2 storage drivers then your shared libraries will share memory but when using any of the other storage drivers they will not.

I'm not sure why this detail isn't made more explicitly obvious in the Docker documentation. Or maybe it is but I've just missed it. It seems like a key differentiater if you're trying to run dense containers.

Solution 2

You can figure out if the .so's in different containers are sharing the same physical memory by comparing the physical addresses of processes (/proc/pid/pagemap) from two different containers as seen on the host.

# ps -ef | grep java | grep -v grep | awk '{ print $2 }'
3906
4018
# sudo pmap -X 3906 | grep -e "Address" -e "libpthread"
     Address Perm   Offset Device     Inode    Size   Rss   Pss Referenced Anonymous LazyFree ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
7f97d9249000 r-xp 00000000  fd:00 135202206     104   104    52        104         0        0              0              0               0    0       0      0 libpthread-2.27.so
# sudo pmap -X 4018 | grep -e "Address" -e "libpthread"
     Address Perm   Offset Device     Inode    Size   Rss   Pss Referenced Anonymous LazyFree ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
7fce739e1000 r-xp 00000000  fd:00 135202206     104   104    52        104         0        0              0              0               0    0       0      0 libpthread-2.27.so
# virt_to_phys_user 3906 0x7f97d9249000
0x59940000
# virt_to_phys_user 4018 0x7fce739e1000
0x59940000

Here 3906 and 4018 are process id's on the host of two instances of a java application running in two different containers. I used virt_to_phys_user which is a simple 'c' program to dump the physical memory given a pid and a virtual memory from this link. Notice that the physical address is the same for both the processes above. Also note that both instances have the same inode addr and Pss indicates that these pages are being shared.

However as the earlier answer mentioned, this behaviour is dependent on the storage driver used. I see that this works on docker-ce on Ubuntu 18.04 and podman on RHEL8 (overlay2 and overlay fs respectively), but it didn't work on RHEL 7.5 with devicemapper.