App Pool doesn't respect memory limits

Question

I am dealing with a legacy .NET app that has a memory leak. In order to try and mitigate a run away memory situation, I've set the app pool memory limits from anywhere between 500KB to 500000KB (500MB) however the app pool doesn't seem to respect the settings as I can login and view the physical memory for it (5GB and above no matter what values). This app is killing the server and I can't seem to determine how to adjust the app pool. What settings do you recommend in order to ensure this app pool doesn't exceed around 500mb of memory.

Here is an example, the app pool is using 3.5GB of

So, the server just crashed again, and here is why:

The same app pool with low memory limits, a 1000 recycle request which cause a recycle event every two or three minutes but sometimes it just runs away.

I am also open to any tool that can monitor this process (either run every 30 seconds as a task or service) and can kill it when it exceeds some limit.

Solution

I found this post because I'm struggling to answer a similar one where limits aren't being restricted. See IIS WebLimits not being respected.

However, I can take a stab at your problem. Try the c# code below. You could do the same with powershell. You'll need to run it with admin rights.

 static void Main(string[] args)
    {

        string appPoolName = args[0];
        int memLimitMegs = Int32.Parse(args[1]);
        var regex = new System.Text.RegularExpressions.Regex(".*w3wp.exe \\-ap \"(.*?)\".*");

        //find w3wp procs....
        foreach (var p in Process.GetProcessesByName("w3wp"))
        {

            string thisAppPoolName = null;

            try
            {
                //Have to use WMI objects to get the command line params...
                using (var searcher = new ManagementObjectSearcher("SELECT CommandLine FROM Win32_Process WHERE ProcessId = " + p.Id))
                {
                    StringBuilder commandLine = new StringBuilder();
                    foreach (ManagementObject @object in searcher.Get())
                    {
                        commandLine.Append(@object["CommandLine"] + " ");
                    }

                    //extract the app pool name from those.
                    var r = regex.Match(commandLine.ToString());
                    if (r.Success)
                    {
                        thisAppPoolName = r.Groups[1].Value;
                    }

                    if (thisAppPoolName == appPoolName)
                    {
                        //Found the one we're looking for. 
                        if (p.PrivateMemorySize64 > memLimitMegs*1024*1024)
                        {

                            //it exceeds limit, recycle it using appcmd. 

                            Process.Start(Path.Combine(System.Environment.SystemDirectory , "inetsrv", "appcmd.exe"), "recycle apppool /apppool.name:" + appPoolName);

                            Console.WriteLine("Recycled:" + appPoolName);
                        }
                    }
                }
            }
            catch (Win32Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }
    }

Comments

[Nathan C]

Try configuring a time limit rather than memory limit to see if you get better results. See this.

[lucuma]

I actually set it to recycle after 100 requests which seems to work better but all the same it seems like some of the app pool settings don't work as I expect.

[MichelZ]

Do you have eventlogging for recyling enabled? anything in there?

[lucuma]

There is an entry every 2 minutes about the private memory limit and recycle. The problem is that every few days the server's memory will blow up and every time I check, this app pool has (as the graphic shows) Gb's of ram used.

[lucuma]

I'll try this out tomorrow and see what happens.

[lucuma]

I tried the script and it doesn't find the process.

[Nik]

And you're running it with full admin rights? Can you see w3wp in task manager for your app pool?

[lucuma]

Yeah, I figured that out after posting the comment. I modified the script to kill the process instead of recycle the app pool as the recycle will bomb if the memory leak consumes all the memory. In theory it should work when the leak goes haywire (hoping).

[Nik]

If I were you I'd then have two thresholds, one for recycling (lower), and one for killing. You could lose data in flight if you just kill, depending on your app. Glad it worked out.

[lucuma]

The app pool actually kills itself but the issue is that sr least one time a day the leak wins. This is the case I'm trying to silolve so drastic measures are called for. Anyways it is only displaying data so data loss isn't an issue. Thanks for your feedback.