Windows Server - "Not enough storage available error"

Question

We have two IIS 7 servers that run in web garden. There's a copy of every site on each server. Files are synchronized using DFS (only source files). Images & big files are located on different server using share.

Atleast few time a week, we get error "Not enough storage available" when reading any file or processing a server code. IIS Server is unable to map path to a local disk (not just the one that synchronizes with DFS), there also seems to be a problem loading group policy permissions (we get various errors) and we can't open any share on any server on lan. There are also I/O errors when reading files from other custom applications. Disk speed and latency during this fall-out are both ok. If we try to copy a file or test a drive speed, usually works. Sometimes, we can't even run "my computer" or any other software (for example performance monitor). It can only start if we "run as administrator". There's more than enough storage available on every disk.

Most of the sites stop working (but not all). After server restart, everything works ok for a few days and then the problem reoccurs.

Servers are virtualized using VMWARE, there are 50 IIS pools (some are mem limited) and around 200 running sites on each server. Both servers have 16GB of RAM (only about 80% used). CPU is constantly between 30 - 60%.

We tried changing the IRPStackSize registry property to a higher number, but there were no changes. Problems still exists. We tried to disable antivirus (NOD) – no success.

Any hint? Maybe we are close to IIS limits (to many sites/pools)? Maybe 32bit OS and 16GB of RAM? Maybe there is some 'secret' registry settings... We don't think that everyday restart is long-term solution.

Solution 1

The Not enough storage available error has nothing to do with disk storage. It is about internal memory.

This error is quite easy to reproduce. Simply select a bunch of large files (images or DLL's or whatever), right-click and use the open-with option to open them all with Notepad. This should eat up allot of internal memory. When the swap-file nears it's maximum capacity, random services will start logging Not enough storage available to the eventlog. Often they will stay in a faulted state until they get restarted.

I have come across this issue with my own services, and had to add some extra error handling to force the service process to exit when such an error occurred on a thread. That way the service would restart and recover in an automated manner.

IIS usually handles this situation quite well (compared to an average windows service).

In order to find the process that causes this (usually a memory leak) you should use the performance monitor mmc snap-in, and record the memory usage of running processes (be selective when deciding what to record, since these logfiles can become quite large). Another option would be to use xperf, which is actually more suited for debugging low-level drivers and so, but can be helpful in this scenario as well.

If it turns out to be the w3wc service that's eating up all memory, you could use a tool such as Microsoft's all-new Application Insights (or any other IIS monitoring tool) to figure out which application pool, web-application or website is responsible. Then you can restrict memory usage for that particular pool, or schedule it to recycle gracefully.

IIS can usually cycle a pool in such a way that no one will notice (depending on how the app handles sessions, if it is custom built you might lose sessions when all static variables are flushed). Often a workaround like this is good enough, and expensive fixing of the bug can be avoided (disclaimer: that's not "my style" of solving problems, but most stakeholders love it that way).

Solution 2

32bit Operating system with 16Gb ram will effectively leave 12Gb's unused. The upgrade to 64bit OS should defenitly be considered.

Use of the PAE switch (Physical Adress Extension) could offer a temporary fix.

For info with regards to max memory which a OS can use along with the PAE switch info look at:

http://msdn.microsoft.com/en-us/library/windows/hardware/gg487503.aspx

Regards,

Solution 3

Could be a memory leak? Possible with that number of IIS pools, depending on what they're doing.

Easy to monitor with POOLMON.EXE from Microsoft (Grab it from the official 2003 Support Tools)

Powershell script I run hourly for the duration of 'reboot -> crash'.

$date = get-date -Format ddMMyy-hhmm

.\poolmon.exe -b -n Output-$date.log

#send-mailmessage -smtpserver 10.1.1.11 -to [email protected] -from [email protected] -subject "Poolmon output $date" -attachments "Output-$date.log"

#Extract top offender

$TAG = (Get-Content Output-$date.log)[3]

$Code = $((($TAG -replace '\s+', ' ').split(" "))[1]) 
$Byte = $((($TAG -replace '\s+', ' ').split(" "))[6])

ECHO "$Code,$Byte" >>chart.csv

Chart.CSV will show you the top TAG and its bytes allocated value.

The Output-$date.log will show you everything.

If you have got a single offender at the top, that stays at the top and increases until crash you have a likely candidate for the cause.

Also look in Eventvwr under system for events from SRV mentioning 'insufficient resources'

Comments

[mfinni]

Or simply reducing the amount of RAM. Could be running out of Page Table Entries.

[TomTom]

Not "considered", baseline. Everyone running a 32 bit OS in 2013 wihtout REALLY good reason (special hardware with driver issues) should reconsider. His career, possibly.

[easwee]

We will try to setup this - the bad thing is only that it takes time for the error to happen - but it's a start. Thx.

[easwee]

Exactly, the physical resources are all ok: cpu 30-60%, ram cca 80% all the time, disks space ok... Fyi, we are monitoring the servers all the time. Also updates and hotfixes was first what we did. Just yesterday issue happened again. What we tried was to restart "Application Host Helper Service" an IIS per-se and sites started to work. Now we don't know are we cured the cause or the result of the problem? Maybe this is the new clue?

[Louis Somers]

You are monitoring the servers, but are you monitoring the services and application pools individually? The 80% you say, was that by the time you got to look what was going on, or was the monitoring history constantly 80% before the error occurred? If it was constantly 80%, then it is the service that logged the Not enough storage available error, which is misbehaving by attempting to allocate an extreme amount of memory in a short time. You should be able to find the original process where the error originated in the event log. From your description it sounds like it could be explorer.exe ?

[Owl]

I don't care - it solves the problem, copying files should not be blocked by windows because another disk that is neither the source nor the destination is full. Attitude is not to OP, but to Microsoft in general. I am leaving the answer here, mark it down to your heart's content.