w3wp.exe hogs memory

Question

On a Small Business Server 2011 installation a whole number of w3wp.exe processes appear to be using a disproportional lot of memory. The SBS out-of-the-box installations comes with a total of 7 sites and 20 ASP.NET application pools (Sharepoint, Exchange, WSUS and SBS-specific stuff like Remote Web Workplace).

The resulting dozen of w3wp.exe processes tends to consume more than 4 GB of the server's memory over time with the peak application pool being the one belonging to WSUS with around 800 MB in the working set. Manually recycling the application pools through the IIS MMC helps temporarily reduce the memory usage (the w3wp.exe processes shrink back to 10 MB, some of them regrowing quickly), but obviously is not something an admin wants to do all day. I was unable to find any recommendations on automatic recycling of the SBS-preinstalled application pools, so I am somewhat reluctant to "just do it" on production systems.

My research on the net on how to limit this only threw up a number of posts stating that w3wp memory consumption would not hurt but benefit performance as memory would be "freed when needed by other applications". The trouble is that it does not work out:

• for one, an SBS is a multi-role server, one of the roles (the major one) being CIFS network storage which immensely benefits from filesystem caching which again relies on memory being "free" as in "not used by other processes in any way" - ASP.NET application pools which are hardly ever seeing users and eating memory are counterproductive
• another thing is that I still have to see substantial decrease of the w3wp instances memory consumption upon memory shortage - what I see is a minor decrease by significantly less than 100 mb and excessive swapping instead - again hurting performance

I hardly ever administer IIS or ASP.NET apps, so any ideas on how to effectively trim the memory requirements for the application pools are welcome.

Solution 1

This is what I ended up doing:

setting the server application cache for the .NET AppPools to a low value (5 MB) by setting the privateBytesLimit parameter in the web.config at %WINDIR%\Microsoft.NET\Framework\<version>\Config as suggested in this answer:

    <configuration>
      <system.web>
         <caching>
           <cache privateBytesLimit="5242880" privateBytesPollTime="00:01:00" />
         </caching>
      </system.web>
    </configuration>

This helped reduce the memory usage to somewhat more than 1 GB with the default pool recycling settings.

Apparently, using the "server" type of garbage collector (<gcServer = "true">) can lead to significant memory consumption as well, but as it seems, <gcServer> is set to false by default.

Solution 2

Welcome to the wonderful world of SBS. Recommended requirements for RAM = 10GB... and it REQUIRES a minimum of 8gb. (according to Microsoft.) for a good reason. It's not a fine-tuned well-oiled machine... it's very sloppy, bloated, and has everything under the sun bundled together. The more RAM you can throw at that box... the better. Unfortunately, you're limited to 32gb max. Which imho... is silly.

Solution 3

If you suspect that the resulting memory consumption is a problem due to a software defect, you can use the Microsoft DebugDiag 1.2 to create a full memory dump and analyze the dump for common issues. If you think there may be a memory issue, you need to enable leak tracking by selecting the "Monitor for Leaks" option and let it run for a while before creating/analyzing the dump.

DebugDiag 1.2 Download
https://www.microsoft.com/download/en/details.aspx?id=26798

Solution 4

You don't need a separate app pool for every app, just those that are unreliable or you wish to give priority to. Many can share (keeping different .net versions separate). You can then more realistically limit the memory an app pool will use. There should be no need to repeatedly recycle pools more than once a day.

Also, there is only so much memory that can be freed up in this manner. While some of it will be cache, each app needs a certain amount of working memory which is highly dependent on the specific web app. Trying to restrict this too much is going to bring things to a grinding halt.

The problem really is that SBS tries to do too much at once, you need to look at what you actually use and shut off what you don't.

But to be honest for just 11 users, where is the rest of the memory going? Exchange and SQL for light use certainly don't need more than 12Gb!

View more solutions

Comments

[Nate]

What does this ASP.NET app do? I've seen w3wp.exe use lots of memory when an app fails to close connections with Linq-To-SQL or Entity Framework.

[the-wabbit]

There are several. Many are related to Exchange like OWA/OMA and Sync services, one exceptionally large one is WSUS, some are used for Sharepoint or customized SBS-specific (CompanyWeb, Remote Web Workplace) sites

[TheCompWiz]

I guess I should refine my answer a bit. If you're concerned by the amount of RAM it's consuming, You'd save yourself a lot of time/headaches by doing one of the following: A) Don't use SBS... use standard server and setup the roles you NEED individually. or B) Throw more RAM into the system... as RAM is pretty cheap. SBS is designed for very small offices... (10-20 workstations...) and does not scale very well.

[the-wabbit]

Thanks for your answer. The system where I observed the behavior had 16 GB of RAM which is the physical limit. The available memory quickly filled up with Exchange's store.exe and numerous instances of SQL Server and w3wp.exe which is not just "not fine-tuned" but outright braindead. I know how to deal with excessive memory consumption issues of the other two as I happen to manage Exchange and SQL Server systems frequently, but with w3wp I am somewhat at a loss. The network itself has just 11 users.

[TheCompWiz]

Exchange, SQL Server, and IIS all have mechanisms that will try & consume 100% of RAM for "cached" data. If something else requests more RAM... all 3 are supposed to scale-back to allow other services to run without resorting to swapping. (in theory) In practice, however I find that you just have to roll with the punches on some things... You can try & tweak everything manually & set hard-limits... but you'll be ever chasing your tail on that one. I'll see if I can find an article I read a few years ago on limiting application pools in IIS for you...

[TheCompWiz]

technet.microsoft.com/en-us/library/cc732519(WS.10).aspx

[TheCompWiz]

technet.microsoft.com/en-us/library/cc725749(WS.10).aspx

[the-wabbit]

Ah, you never experienced the beauty of an SBS, did you? Exchange's information store grows to 8 GB when not restricted, multiple SQL server instances will happily take up another 3 GB. Most of the app pools are running just 1-2 applications with the same .NET version and within the same security context. Yet, I cannot regroup them because of support issues. Also it is unlikely to solve the memory problems - if I just have 4 1-GB processes instead of 12 smaller ones, not much is won.

[the-wabbit]

Thanks for the references. What I need to know if it is safe to recycle the app pools on a very frequent basis (like once every 15 minutes) in order to keep memory requirements low or if doing so is asking for trouble (this is what I suspect).

[the-wabbit]

BTW: "memory is cheap" is a tough argument. On behalf of our customers we are running several SBS installations virtualized (due to lower licensing costs) along with Terminal Servers. Surely there is a big difference if a machine with 184 GB of RAM (this is where RAM is not so cheap anymore) can host 5 or 20 of these installations. Fortunately, the app pools are not so much of an issue there as we can disable most of the .NET services and do not care (much) about the support status.

[TheCompWiz]

@syneticon-dj: As each SBS has a maximum 32gb limit... We're not talking about high-end machines. The cost for RAM in the majority of new-ish (within 3 years) servers is ~$100-$200 bucks (worst-case). Cost of re-configuring each service to consume a fixed amount... plus time debugging all the potential problems that may arise from trying to limit everything... is far more.

[TheCompWiz]

@syneticon-dj: and if you're putting 20 SBS instances on the same hardware... your customers must hate you... or are too stupid to know they're getting screwed.

[JamesRyan]

While there is no need to be rude, TheCompWiz is right, virtualising SBS seems crazy. You could provide each of the apps it includes as a managed service much cheaper and would be a far more optimal setup.

[the-wabbit]

@JamesRyan while you are technically right, there are licensing and regulatory constraints preventing us from doing that. Anyway, the virtualization scenario (where we do not have problems with w3wp processes due to the named reasons) was just an example for why memory conservation is a good thing. The systems where this issue hurts the most are physical ones in the customer's premises where I often cannot give SBS more than 16 GB without replacing the systems. TheCompWiz: as the SBS instances see minimal load, there are no issues consolidating them on powerful hardware.

[TheCompWiz]

"SBS" and "minimal load" are antonyms... imho. But I see the point you're trying to make. The very point of SBS is to load up everything into one box... even if the majority isn't going to be used by a customer.

[the-wabbit]

Thanks for the link, I am going to give it a try. The MSI installation package apparently has trouble when run on localized (non-US-Englisch) versions of Windows, I have to see if there is a workaround to this.

[Greg Askew]

FYI - a while back we started setting gcserver to false after reading these articles: forums.asp.net/t/1654596.aspx/1 and msdn.microsoft.com/en-us/library/ff647787.aspx

[Michael Steele]

@the-wabbit Is the advice to set gcServer = false still current now that server garbage collection happens concurrently?