Top showing virtual memory usage of hundreds of gigabytes

Question

On a new virtual machine, top shows that a process of mine consumes 100 GB of virtual memory. I've read that it's possible because of overcommit, however, it's a bit too much. I've never seen anything like this, especially in a VM having just 6 GB real memory, 2 GB swap and a single 40 GB HD partition.

The funny process with 101 GB VIRT is Eclipse IDE (started a few minutes ago) and there's another one, WebKitWebProcess with 98 GB VIRT. All other processes have sane VIRT values of about 1-2 GB. All three tools seem to agree on that: top, htop and ps.

• Is there any good reason for the processes to request that much virtual memory?
• Isn't such behavior detrimental to the system?
• What are the current limits on the total virtual memory?

Update

Details:
Linux 4.15.0-38-generic x86_64 GNU/Linux, Linux Mint 19 Tara
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
Version: 2018-09 (4.9.0) Build id: 20180917-1800

The content of /proc//maps starts with

00400000-00401000 r-xp 00000000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00600000-00601000 r--p 00000000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00601000-00602000 rw-p 00001000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
014fc000-02ebe000 rw-p 00000000 00:00 0                                  [heap]
c0000000-d0000000 rw-p 00000000 00:00 0 
d0000000-100000000 ---p 00000000 00:00 0 
100000000-1012c6000 rw-p 00000000 00:00 0 
1012c6000-140000000 ---p 00000000 00:00 0 
7ef800000000-7ef800004000 rw-p 00000000 00:00 0 
7ef800004000-7ef8000ec000 rw-p 00000000 00:00 0 
7ef8000ec000-7ef800100000 rw-p 00000000 00:00 0 
7ef800100000-7f0800000000 rw-p 00000000 00:00 0 
7f0800000000-7f1000000000 ---p 00000000 00:00 0 

The last two shown lines line seem to be the culprit as 0x7f0800000000 - 0x7ef800100000 ≈ 68.7e9. From what I've found, it's an unmapped private region with no permissions. That's all I can say....

Solution

Most likely it is mmap'ing every file it can index. I've seen similar with Kafka mapping thousands of files but usually the virtual footprint is much higher (~10x) times). You can run:

cat /proc/{PID}/maps

(where {PID} is your Eclipse process ID) to see if this is the case. If it is indeed the case, you might have to raise vm.max_map_count in sysctl if your virtual footprint is too large.

Comments

[Nasir Riley]

VIRT is the amount of memory that the process is able to access at the time which can also include memory being used by other processes, etc. That process isn't actually using anywhere near that amount of memory

[maaartinus]

@NasirRiley Sure, it isn't using that much memory as that much memory definitely isn't there. But it must have asked the OS for it and I really wonder why. Moreover, I often checked the memory consumption of Eclipse in the past and I've never seen any such numbers.

[Haxiel]

This QA might help: serverfault.com/q/138427/349846

[maaartinus]

But even mapping the whole disk would stay below 50 GB (as I wrote, there's 6 GB real memory, 2 GB swap and a single 40 GB HD partition).

[tk421]

Look at the contents of /proc/{PID}/maps to see if it's taking to account the actual file size or it's just picking an arbitrary size.

[maaartinus]

I've added the contents to my question... but I can't say I'm enlightened.

[tk421]

RE: 50GB disk, you can mmap a larger size than the file like if I want to mmap an empty file so your disk size is immaterial. And in your case, you don't have to worry about hitting the sysctl limit.