Windows, why your 8 GB of RAM feel like a few MB?

Question

I'm on Windows 7 x64 with 4-core Intel i7 and 8 GB of RAM, but lately it feels like my computer's "RAM" is located solely on the hard drive.

Here is what the task manager shows:

The total amount of memory used by the processes in the list is just about 1 GB. And what is happening on my computer for a few days now is that one program (Cataloger.exe) is continually processing large quantities of files (near 5-10 MB each), repeatedly opening and reading them for the purposes of cataloging. But it doesn't grow too much in memory and stays about that size, 100 MB. However, the amount of data it processes in, say, 30 minutes can be measured in gigabytes.

So my guess was that Windows file caching has something to do with it. And after some research on the topic, I came across this program, called RamMap, that displays detailed info on a computer's RAM. Here is the screenshot:

So to me it looks like Windows keeps in RAM huge amounts of data that is no longer needed, redirecting any RAM allocation requests to the pagefile on the hard drive. Even when I close Cataloger.exe, the RamMap reports the size of the mapped file as about the same for a long time on. And it's not just this particular program. Earlier I noticed that similar slowdown occurred after some massive file operations with other programs. So it's really not an exceptional behavior of Cataloger.exe and the problem manifested itself many times in the past.

Whatever it is, it slows down the computer by like 50 times. Opening a new tab in Chrome takes 20-30 seconds, opening a new program can take up to a minute. Due to the slowdown, some programs even crash.

So what do you think, is the problem hiding in file caching or somewhere else? How do I solve it?

Solution

This is just business as usual, and everything is working correctly. Windows isn't holding cached files in memory over other data - Cached files means that at one point, Windows had to load them into memory or read them off the disk, and something higher-priority has not come along since that time. Just as soon as you resume using the computer and programs request memory, Windows will happily remove the cached files to make room.

If you look at RAMMap, you'll notice that most of those cached files are allocated under "Standby" - that means that it's being held in memory because Windows needed it in the past, might need it again, and will quite happily discard it if something else actually needs the space.

Basically what you're seeing here is your program requests a large data file to be loaded, so Windows loads it into memory. Managing memory is a very complex process, and what Windows is doing under the covers here is making a judgement call: It looks at the current processes, and sees that Chrome is taking up, say 5GB of memory (lots of tabs!), but that most of that memory hasn't been touched in the last hour. At this point, it has a choice: It can leave Chrome in memory, and not cache the files. This means that the Cataloger process could take hours instead of minutes to process a file, especially if it is jumping around in the file a lot; or, it can page out the chrome tabs and load the Cataloger file into memory, and finish that process quickly.

Now, you'll feel the pain when Windows has to page chrome back into memory, but Windows won't do that until you request it (i.e., bring chrome back into the foreground and select a tab that was paged out), and what ultimately needs to be evaluated by you is if the pain of that is greater or less than the pain of your Cataloger task having to wait on it's memory. You can try running the program as a service and tell Windows to optimize for responsiveness, or try lowering the process priority, but I'm pretty sure what you're going to want is for the Cataloger process to finish as fast as possible - While it's maxing out your disk I/O, your entire system is going to be incredibly sluggish. Everything that requires disk I/O will be put into a queue (even webbrowsing - it has a cache it uses too!). Programs will open slow, new tabs will open fast, but actually using them for something will be slow.

---

If the program is just sequentially reading file after file and then not touching them again, I'll agree that Windows doesn't need to be caching the files. The problem here is Windows doesn't know that, because nobody is telling it that they don't need to be cached.

If you have access to the source code for the Cataloger program (or can request modifications be made to it), it can be configured to open files with FILE_FLAG_NO_BUFFERING, which will cause Windows to skip the disk cache for files opened by the program in this manner.

If that's not an option, then unless the Cataloger process has to be run on your system, I'd look around to see if there's a spare computer that isn't being used, and see if it can be set up as a dedicated machine for cataloging these files. If there's not one already, simply increasing your RAM might be another option; memory prices are quite cheap these days, and developers know this - most things are optimized for speed and responsiveness at the cost of memory. 8GB isn't what it used to be.

Another option is to move all of the files that it uses onto a separate harddrive, and specifically turn disk-caching off for that drive.

Comments

[Oliver Salzburg]

5.2G of those 5.9G are marked as Standby, they are not actually in use. Over half of your system memory is up for whoever needs it, in fact. What operations exactly make you feel like your computer is not performing as well as it could?

[uxout]

Just to be very clear, if you leave the system on for a day or two without having ever run this "Cataloger.exe", do you have the same problem? I think you may be putting the cart ahead of the horse here.

[Desmond Hume]

@OliverSalzburg Programs get non-responsive, a .txt file requested to be opened in a simple text editor opens after 5 minutes of waiting, Chrome freezes almost completely, if Chrome does't freeze, it is incredibly slow and I can physically hear how HDD starts working hard (hence my guess about pagefile) whenever I try open a new tab, which takes very long to complete. As a general rule, when my computer uses HDD intensively, performance drops quite noticeably. When HDD is not in extensive use, my PC is just a sweet pie.

[Desmond Hume]

@Shinrai I haven't done such. My PC is off for the night. But when I turn it on and start some file operations, like moving/copying many files or unarchiving huge amounts of files, it may get unusually slow in just 10 min.

[uxout]

@DesmondHume - So it could be a problem with the hard drive. Or the hard drive controller. Or the file system could be corrupt. There are a number of other possibilities.

[Oliver Salzburg]

@DesmondHume: Well, this could also indicate that "some process" is simply excessively making use of the HDD, which reduces overall system performance. It doesn't have to indicate a low RAM/paging situation (and your screenshot also doesn't indicate one). Please see superuser.com/questions/404617/… and, if needed, superuser.com/questions/26862/…

[tigrou]

Something I notice on a old computer at work : even if windows have plenty of free ram, at some point it will start to write some stuff from ram to harddrive. it happens especially when you keep the computer a very long time inactive (but not enough to put it in sleep/hibernation mode) for eg: a locked station. When you go back to the computer and use it, it will start to move data back from hdd to ram. On most computer you don't notice it, but that pc at work had a terribly bad harddrive. After that i learned about pagefile and disabled it (even if not recommended) and problem was gone.

[Desmond Hume]

@OliverSalzburg The File Summary and File Details tabs are filled up with the very files being processed by Cataloger..

[Desmond Hume]

I still don't see any reasons for such a dramatic slowdown. The Cataloger.exe processes the files one by one, staying about just 100 MB in memory: it opens a file, reads it, closes, goes to another file. The files are not too big, only 5-10 MB. Why does my computer have to almost freeze because of that? How do I solve the problem anyways?

[user1984103]

@DesmondHume It's more that you've found one of the rare instances where Windows makes an incorrect assumption - Ideally, it wouldn't cache those files, but in 99.9% of situations which are similar, you would want to cache those files (i.e., a game loading textures from disk, where they are going to be reused level after level). The trick is figuring out how best to tell Windows that it doesn't need to cache those.

[Desmond Hume]

I don't have the source code for Cataloger.exe. And like I mentioned in my post, Cataloger.exe is not special in any way, most of large-scale file operation has the same negative effect, program regardless. And advises like "buy more RAM" or "get another computer" are really hard to appreciate, sorry.

[user1984103]

@DesmondHume I can understand that - The underlying issue is you, the user, can't arbitrarily tell Windows not to cache specific files, and disabling the disk cache completely will destroy your performance all the time, in all programs. I think you can tell Windows not to cache files on a specific drive, but even there I'm not 100% sure as I've never had to test that myself.

[Desmond Hume]

Can I set a limit on the file cache size using this program or is this irrelevant to my case? technet.microsoft.com/en-us/sysinternals/bb897561.aspx

[user1984103]

@DesmondHume I think that program is referring to the "working set" of an application, which is different. Based on my reading, even then it does not provide guarantees-- it hints at the memory manager that it should keep data in memory (i.e., for chrome), but it's still ultimately up to the memory manager.

[Oliver Salzburg]

Please note that the documentation for CreateFile explicitly states that FILE_FLAG_NO_BUFFERING has no effect on memory mapped files (even though someone in the comments claims it does).

[Jamie Hanrahan]

I'm just not seeing a reason for a slowdown here. Pages on the standby list are immediately available for any use, just as if they were on the free list. It's just that if their current contents are useful and referred to, they can be used. That's why we refer to the Standby list as a page cache.

[user1984103]

@JamieHanrahan The problem is that while they are standby when the screenshot was taken, they were active at one point, flushing out other standby data; The slowdown is from the system loading standby data back into RAM from the pagefile that was flushed to make room. If I map 5GB of data into memory, and then release it, I'll wind up with 5GB of standby memory and my system feeling like molasses as it replaces that 5GB of standby memory with pagefile data that it needs.