Windows 10, 'System' process taking massive amounts of RAM

Question

Since I upgraded to Windows 10, my system has been consuming RAM excessively

I've been reading a bit and determined it's likely a driver leaking memory. So I got myself the Windows Driver Kit and tracked memory usage with poolmon:

However, I don't really know how to proceed from here. Is the item tagged "smNp" the culprit in this issue? How do I go from there to actually identifying the driver?

I tried some stuff like "C:\Windows\System32\drivers>findstr /s smnp ." but it returned no results. I also took a look at the pooltag.txt file and this is the description I found for it:

So yeah, any help would be appreciated. Thanks in advance.

Solution 1

By going into services.msc (via Win+R) and disabling Superfetch completely solves this. I am not sure if Superfetch is just broken as of now or it's "by design".

In addition, apparently getting rid of the paging file will have the same effect but the above solution is a safer bet.

Solution 2

I've looked at xperf traces of several users and here the function ntoskrnl.exe!SmKmStoreHelperWorker of the Kernel starts to allocate memory.

^{(Click image to enlarge)}

I discovered this on sysinternals.

I've asked Microsoft about it and the answer is that this is by design. It is related to System Memory compression.

In the announcement of Windows 10 Build 10525, Microsoft explained it a bit:

In Windows 10, we have added a new concept in the Memory Manager called a compression store, which is an in-memory collection of compressed pages. This means that when Memory Manager feels memory pressure, it will compress unused pages instead of writing them to disk. This reduces the amount of memory used per process, allowing Windows 10 to maintain more applications in physical memory at a time. This also helps provide better responsiveness across Windows 10. The compression store lives in the System process’s working set. Since the system process holds the store in memory, its working set grows larger exactly when memory is being made available for other processes. This is visible in Task Manager and the reason the System process appears to be consuming more memory than previous releases.

So instead of writing memory data to the pagefile it compresses them. And this compressed memory is shown in the System process.

Microsoft also posted more details in the inside hub. Winbeta created a article which includes more details.

Apparently, the reason for this happened to do with Microsoft choosing to suspended UWP apps when they were not in the foreground, very similar to some smartphone OS management. Windows 8 users understood (perhaps not) that if apps weren’t on screen, they wouldn’t run until the user switched back to them. The ‘all or nothing’ approach is being updated with Windows 10 introducing a layer between the pagefile and normal paging activity. Now, when faced with memory pressure issues, MM will determine which pages should be moved to the modified list in a process called trimming. The modified list is a secondary list of pagefiles backing up a list of standby pagefiles. A backup list is captured in case memory is reclaimed from the standby list by another process, and the original process comes looking for its page. Instead of all or nothing, Windows 10 MM will compress unused pages rather than writing them to disk. With less writing, the result should be fewer disk operations – thanks to the compression – and now more data can be stored in memory.

According to the Windows team, “In practice, compressed memory takes up about40% of the uncompressed size, and as a result of a typical device running a typical workload, Windows 10 writes pages out to disk only 50% as often as previous versions of the OS.” If all goes according to plan, Windows users could be experiencing reduced waiting times for all devices as well as extended lifespans on systems that have flash-based hard drives.

Decompression is also something Windows 10 is designed to do well. Windows 10 is using the combination of parallelizability and sequential reads to produce pages into memory once called. The new decompression should result in a speedier experience as Windows 10 is simultaneously decompressing data and reading it in parallel using multiple CPUs. Older versions of Windows may have felt sluggish because of the transfer rates between the disk.

Microsoft also released a Video on channel9 which explains the feature.

Memory Compression in Windows 10 RTM
https://channel9.msdn.com/Blogs/Seth-Juarez/Memory-Compression-in-Windows-10-RTM

In this video Mehmet Iyigun spent some time discussing why the System process in Windows 10 is taking a bit more memory and why it's a good thing. A process taking more memory sounds like a bad thing - that is until I understood more about memory management, paging, and hard / soft page faults. Turns out that that the OS is doing some clever optimizations that allow your processes to trim some of the memory but not necessarily page it out to disk. Not only is the memory preserved in RAM, but it is also compressed - making hard page faults a more rare occurrence. The results should make for a snappier experience.

In the latest TH2 Builds, Microsoft updated the description in the task manager and now also shows that the SYSTEM process hosts the compressed memory:

to avoid confusions about the "high" usage.

In the Window 10 Anniversary Update which was released in August 2016, Microsoft extracted the Compression into now shown in a pseudo process called Memory Compression to no longer confuse users why SYSTEM has such a large memory usage:

But it looks like Taskmgr doesn't show this process, only ProcessExplorer/ProcessHacker are able to show it. The Taskmgr only shows the amount of compressed memory in the overview:

If you hover over the used memory graph in Taskmgr you see a tooltip that shows the amout of data that are compressed.

In this demo 388MB are compressed to 122MB so 267MB are saved with the compression.

Comments

[magicandre1981]

ok, I added the information that was able to find. Check them

[Mani Gandham]

This is a feature of the system to actually keep more things in RAM by using compression instead of paging to disk. @magicandre1981 has the right info here which should be accepted as the right answer.

[Jamie Hanrahan]

The 14 megabytes that poolmon shows is associated with the smNp tag is a purely trivial amount. You're worried about 1.3 GB in the System process private working set - why focus on 14 MB of nonpaged pool (which isn't in any process's working set, at all)?

[Zaibis]

Is this kidn of a bug or is this usual for windows 10? I was thinking about to upgrade but i have only 3GB available and droping 1GB of it jsut for win 10 is not aceptable for me. so will this happen on every windows10?

[Lightness Races in Orbit]

@Zaibis: I think the idea is that you're not "dropping 1GB". 1GB has been reserved by Windows and will be used for allocations. You haven't lost the memory.

[Zaibis]

@LightnessRacesinOrbit: Ah thanks for the information.

[Lightness Races in Orbit]

@Zaibis: For the record, I'm only guessing! We should wait for the blog post, I guess.

[Yarek T]

"Unused memory is wasted memory" has long been the mantra of OS X. Looks like Windows got on board too.

[user2284570]

@magicandre1981 What do you mean bySystem Memory compression

[nkorth]

@deceze I remember there was actually a similar uproar about Vista appearing to use huge amounts of memory, when it was (just like this time) only used for caching and not actually "in use".

[Luaan]

@deceze Windows got "on board" with Vista already, so it's nothing new. It didn't make that much sense back in the XP days when desktops didn't really have any RAM left over anyway. Server Windows had this long before, at least since NT 4.0 (although the Vista generation was still a major improvement of the mechanism).

[Josiah Keller]

I'm not sure I buy this, because just last night I had this same thing happen and the only reason I noticed was because I got a "Low memory" dialog. If the system were intentionally reserving the memory, why would it have to warn me about it?

[magicandre1981]

I can't answer this in details. I only got the information that it is by design, has something to do with a memory compression and this will be explained by a blogpost. When I get new information, I'll update my post.

[Nayncore]

This actually worked. Memory usage for System is back to ~0.1MB even after hours of use. Thanks!

[Ido Naveh]

@den Is it OK to stop this process? Isn't this harm to the computer?

[Bigbio2002]

While typically such high memory usage would be a red flag for a memory leak, this memory is being used legitimately as a FEATURE of Windows 10 (see magicandre's answer below).

[user2284570]

@magicandre1981 Looks like Windows is following Linux Zram years after :)

[magicandre1981]

@user2284570 I have no idea. I don't use Linux that much.

[user2284570]

@magicandre1981 Zram is the equivalent on Linux which does exactly the same thing. It was put in mainline 2 years ago.

[phuclv]

it's not a bug and shouldn't be disabled

[phuclv]

bgr.com/2015/08/20/windows-10-system-process-ram

[phuclv]

lifehacker.com/…

[David Schwartz]

@JosiahKeller The "low memory" dialog is about low virtual memory. It has almost nothing to do with use of, or availability of, physical memory (RAM). They're accounted separately by all modern desktop OSes.

[Francisco Tapia]

as Windows User, i do not find any logic reason to disable Superfetch feature, i really want my computer run faster, preloading my commonly used app in memory.

[Oleg I.]

If the feature causes constant low memory warnings it is not good feature. Now my Surface Pro with 4GB RAM is completely unusable for development and even on laptop with 8GB RAM I constantly get those warnings. Thumb up for solution!

[John Rasch]

Though unrelated, I would like to add that I've been experiencing many query timeout errors running SQL Server 2014 (both Enterprise and Express editions) on a Windows 10 machine with 4GB of RAM. I noticed that the System process was using upwards of 600MB of RAM because of the reasons detailed in this answer, and combined with multiple browser tabs, Outlook, and other programs, SQL Server was not able to obtain a memory grant, which was the ultimate cause of the timeout errors. The solution for me was to increase to 8GB of RAM.

[Mani Gandham]

@OlegI. If you're running out of RAM, that just means you don't have enough. The feature just compressions stuff that's already in RAM rather than writing it to disk which speeds up access. It's not taking up any more RAM than the apps you're running already are and disk reads are only going to make your system seem slower. Seems you might just be running too much stuff.

[ton.yeung]

@ManiGandham i just got a win10 laptop. been getting low mem warnings for the last couple nights. I have 32gb of ram and I use about 5gb of ram when I'm not developing. When I leave the laptop on overnight, I'm left with 19gb free, 8gb of which is in the system process, and still have "low" memory warnings. Somehow I doubt getting 64gb of ram would help much.

[Mani Gandham]

@OlegI. You're saying you're getting this warning now on 3 different machines? I think there's something else you're doing or running that's causing those warnings, if they're actually coming from Windows itself.

[Jamie Hanrahan]

The "low memory warning" is about private committed virtual address space, not RAM. You can have plenty of free RAM and still get those warnings, because the warning is about potential future usage, not immediate. The cure is to increase your pagefile or to add more RAM.

[arkon]

This is a horrible idea. Prefetch/superfetch are used for a reason. If you think you can manage memory better than your OS, you're sadly mistaken.

[arkon]

@ton.yeung You should really go read the documentation about how Windows uses RAM. Free RAM is wasted RAM. If Windows sees free RAM, it's going to do its best to allocate it ("Standby" RAM) to make your system and multitasking more responsive. Whatever issues you're having isn't the result of a faulty Windows feature. It's either the result of malware, broken driver(s), broken 3rd party application(s), or you've screwed up your pagefile settings.

[ton.yeung]

I doubt that. First thing i did was wipe the machine, install drivers, and install chrome.

[ton.yeung]

@JamieHanrahan add more ram than 32gb? What you guys are saying may be correct. It doesn't change the fact that the behavior windows is exhibiting is stupid. I have 32 gb of ram already, and I'm getting advice to add more? I don't know about you, but to me that's absurd.

[Jamie Hanrahan]

@ton.yeung I understand your frustration. The "low memory" warnings have nothing to do with how much free RAM you have. They appear when a process tries to allocate private committed virtual address space ("commit" for short) that would exceed the "commit limit". The commit limit is the sum of your RAM size (again, not just free RAM!) plus your current pagefile size. So the primary problem is not that something is using lots of RAM, but that it's using lots of commit. Please open a new Q with shots of task manager's "Performance" tab, plus the "details" tab sorted by "Commit size".

[Jhourlad Estrella]

I confirm that solution works. After stopping Superfetch, disk activity immediately drops from 100% to 15%. I recommend though to only stop the process when need and not disabling it.

[unbindall]

When was this update released? My SYSTEM is still merged with Memory Compression.

[magicandre1981]

@scriptHero this update is not released yet. You have to install the latest Insider Preview Build to see it. This Redstone 1 update will be released later this year.

[Jamie Hanrahan]

Disabling prefetch (or "cleaning out your prefetch folder") is indeed a very silly idea. Prefetch does not cause the system to use any more RAM. It speeds up the loading of exe's and dll's once you start them, and only then. Re SuperFetch, Microsoft does disable SuperFetch by default on systems with an SSD for the Windows drive; they feel the benefit is not worth it. Personally I keep it enabled because I have other drives, non-SSDs, that I want SF to still operate on.

[speeder]

Just confirming that this applies to all windows versions that can do caching like that. Any windows version that attempt to fill the entire RAM, can have EXTREMELY poor performance by constantly adding random junk to the ram, for example I saw windows chaching on the ram a multi-gb file from a game, it was the music of the game, when I went to work after playing the game, the HDD would work hard to load that file to the RAM every single time it could fit, resulting in constant HDD reading, and programs waiting to use the RAM as windows deleted the thing from the RAM again.

[speeder]

To see the contents of your RAM (including all the files windows decided to put in it as cache), use the program "RAMMap" from Sysinternals: technet.microsoft.com/en-us/sysinternals/rammap.aspx

[Twilight Sparkle]

Just want to say that I tried this and my disk and ram usage went down exponentially. Performance improved significantly, and all I was doing was surfing the internet on chrome. I think there may be something wrong with "superfetch" that could cause it to impair performance.