What's using all my memory?

Question

SOLUTION: There's a lot of stuff to read in this thread, so i'll just summarize here to help.

The problem is the VMWare ESX balloon driver which "inflates" the amount of memory "in use" that the OS sees in order to force the OS to dump caches and reduce working set size. This creates a large amount of "in use" physical ram that you can't account for by looking at the processes. You can look at the VMWare console performance graph for the VM's and see the balloon driver in use.

The solution to the performance was to raise the amount of reserved memory for the server, which limits the amount of memory the balloon driver can steal from you.

I have a 32-bit Windows 2003 Terminal Server that is running a local copy of SQL Server (full version, not express) and hosts a Visual Studio 2008 development environment for several developers. The machines ia Virtual Machine running on vSphere 4, with 4GB of memory allocated to it.

When I have 2 or 3 users logged in, it's using more than 4GB of committed memory, but I can't seem to find where most of this memory is being used. When I add up all the memory in Task Manager for all users, it only accounts for about 2GB at most. What's using the rest?

I've tried cycling the SQL server, that only takes the memory that I see allocated to SQL servr out. The same is true of visual studio, if I shut it down, it only takes the amount I see allocated in Task Manager out of the committed memory pool. I have also set the memory options for SQL Server to only use 512MB max.

I don't believe this is a system or file cache issue, since Windows doesn't include that memory in the committed statistics AFAIK.

It's driving me nuts. There should be more than enough memory for all these things, based on the amounts they seem to be using. How do I locate where this extra memory is being used?

Update

Regarding which memory statistics i'm looking at, none of them add up.

For example, right now with 1 user logged on and only running Visual Studio and the WebDev server, and SQL Server and IIS in the background.. I add up Private bytes and I get about 1049740K of memory, but task manager says commit charge is 1146M, leaving about 150Mb unaccounted for.

When 3 users are logged in, it's only about 2GB of private bytes (a lot of memory is shared because the users are just running Visual Studio).

VM Size doesn't add up either, as that is typically not that much different than private bytes. Although, oddly enough, the number is very different in Process Explorer for virtual memory than it is in Task Manager. PE shows virtual memory of SQL Server at 1,791,444K for SQL Server and 1,251,432K for Visual Studio, yet commit charge is only 1146M total for the machine.

There is no column for WS Size in Task Manager, but in PE it lists WS as both Private and Sharable, and those numbers are significantly less than the Commit charge, even added up.

UPDATE:

I know this question has a lot of detail, so maybe the question is not clear.

How do I find which processes are using all the committed memory in my system? The memory given by TaskManager and Process Explorer do not add up properly, and only show about 2GB of committed memory even though 4GB is committed.

Solution

sql server and vmware tools fight for memory space in a vm. This is probably compounded by running terminal services and vs.

If you haven't already, go into the vm's settings and create a full 4Gb memory reservation under the resources tab. This'll prevent the baloon driver attempting to reclaim memory within the vm.

You may also want to configure the options within sql server to ensure it only allocates part of the system's ram. Maybe 2gb max.

Comments

[Admin]

I would also say that 4GB memory is very low for a shared dev machine, but thats just me.

[Sonicsmooth]

Yes, I'm serious. We do not need that much memory per developer because A) the user is not using Internet explorer or doing anything other than running visual studio. That's it. Everything else they will do locally on their machine, not in the remote desktop. I cannot upgrade to a newer OS for several reasons. It's only using Visual Studio, and SQL Server. I've limited the SQL to use only 512MB, Visual Studio uses about 350MB per user on average, easily enough RAM for 3 developers.

[Sonicsmooth]

And memory buffer is not as much as you might suggest. Yes, one screen per developer, at 16bpp and around 1920 x 1200 = about 36MB. However, I would assume this memory would show up in the CSRSS or some other process in TaskManager or PE. The problem here is that there is memory committed but no obvious process using it.

[Sonicsmooth]

Furthermore, you're using Windows 7 as your virtual client. That eats up more than 1GB just logging on. Windows 2003 uses significantly less. Including SQL server, startup memory usage is about 350MB. If we were using Windows 2008, we'd need a LOT more memory (there's only 8GB in the host server, which also runs a number of other services). So no, thanks, but your suggestions are not viable, nor do they address my question which is What is using the memory?

[Sonicsmooth]

Yes, I have already limited SQL Server to 512MB. However, i'm not sure if you're addressing my question. My question is, how do I find what processes are using all the committed memory. Adding up the memory that Task Manager says is in use does not equal the amount of committed memory.

[Aashraya Singal]

Missing memory is potentially accounted for by the baloon driver, as described above. Ensure you have the reservation set up, then re-check the figures and see if they've lined up. Also, when you installed all of your apps, you made sure to set your session to Install mode before installing?

[Sonicsmooth]

The terminology you are using is confusing. It's not "missing" memory. It's there. The system says i have 4GB. It's just that when 2 or 3 users are logged in, it shows about 4GB of committed memory, but the amount of memory allocated to processes according to tools like Task Manager and Process Explorer does not add up to the amount of memory that is allocated. Does this "balloon driver" allocate memory, or take it? It's not a performance issue per se, it's a "what is allocating my memory" issue.

[Sonicsmooth]

Well, as I said in several places, I configured SQL Server to only use 512MB. However, I will do as you say and see what it says.

[joeqwerty]

Sorry, I missed that in your post. I suppose it doesn't hurt to look though.

[Sonicsmooth]

Well, those counters claim it's using 262144KB, while TaskManager says the SQL Server process is using 315,128K, and 323,380K VM Size. PE says it's using 323,380K Private and 1,791,956K Virtual Size, and 292,616K Working Set Private, and 22,512K WS Shareable. None of it adds up.

[Sonicsmooth]

I'm curious, what does setting the session to install mode do, and how does it relate to memory usage?

[Sonicsmooth]

According to information i've read, Windows 2003 will automatically switch to Install mode when installing programs, and even if it did not all it would do is affect how the registry is created. I don't see how this would affect memory usage.

[TomTom]

Ok, so no IE - that means no integraed help (hint: help is always rendered using IE HTML engine)

[Sonicsmooth]

I'm aware of that. In most cases, integrated help doesn't seem to work very well anymore anyways. Too much stuff. It's a crap shoot if pressing F1 will bring up relavent data. It's easier to google search for what you need, and the developers do that on their local computers not in the remote one. This is a minimal environment.

[Sonicsmooth]

I'm marking you the answer to this question because it lead me to investigate the VMWare baloon driver, which gave me much more information on what was going on. Thanks.