IIS 7 Windows Authentication Wrong User

Question

IIS 7, Integrated App Pool, Identity set to NetworkService

We've recently moved an asp.net MVC 3 to our production environment. The application uses windows authentication. The production environment is setup on a web farm on IIS 7, running Win 2008 (not sure if R2). There are two servers configured to run the web application.

We've had on two occasions (about 1 week apart) where users are getting authenticated as different users than themselves. Checking the IIS logs, it seems like after some point, all requests are authenticated as one particular user. In the log there are also entries with X-ARR-LOG which is apparently Application Request Routing Module (not sure if related, but more info could be helpful..). My networking background is not very strong, but our networking resources haven't had any luck finding the issue.

Are there any diagnostics tools we can run to see where the issue may be coming from? Is it likely that something is configured incorrectly on the web farm that could be causing this problem?

One important factor is that in the IIS log, there are requests for a plain html page that doesn't go through the asp.net pipeline, and in the log that entry is requested under the user mentioned above. So it seems like after the problem comes up, all requests seem to be authenticated under the same user account.

Appreciate any help, hints that you can provide

Update:

Elaborating on how we are making the determination that different users are getting authenticated as one user...

In the IIS Log, we are looking at the cs-username column which according to the documentation, is set by IIS. We can see different requests from different IPs, in different locations that have the same cs-username. In the log, all the requests have different cs-username until all of a sudden, all requests are from the same cs-username. We've been able to reset this problem by taking the servers out of the farm and recycling the app pool. Not sure specifically which of those two helped us "reset" the problem. Also mentioned that some of the requests are for a plain html page, that doesn't go through the asp.net pipeline, (believe it is used by a monitoring tool, as it is called sitecheck.html) and those requests abruptly have the same cs-username. Also, the application itself uses the WindowsIdentity to display user specific data, which is what led us to look into the problem a little deeper...

Solution

https://stackoverflow.com/questions/7410473/asp-net-application-iis7-sessionstate-is-crossing-over-multiple-users-in-differe

http://xprog.blogspot.com/2011/03/iis7-sessions-getting-crossed-mixed-up.html

Session problem with IIS7
http://forums.iis.net/t/1154347.aspx

IIS7 Sessions Getting Crossed / Mixed Up / Copied
http://lionsden.co.il/codeden/?p=446

The symptom

Users reported seeing data that did not belong to them when they logged into their accounts. Logs showed that nearly 10% of our users were being assigned copies of other users’ sessions. The entire session object was copied, including the session Variables and sessionID. Once duplicated the individual sessions could be modified without affecting the other (i.e. abandoning one did not kill the other).

To spot instances of the problem I stored the Request.RemoteHost (The IP address of the computer making the request) in a session Variable. At the beginning of each request I checked to make sure the session’s IP matched the current request’s IP.

What is was not

It is possible that the IP would change naturally, most simply if a user reset his router/modem. This was not the case because it was happening way too often. Further, there were confirmed instances of one user’s data being crossed with another. Finally, some of the pairs of IPs weren’t just on different computer, but were in different countries.

It is also possible, and was often suggested, that session Variables can stop being unique if they are used with static/shared variables within the WebApp. This was not the case either, because the IP address I stored in the session was only written to once, from the request and afterwards was only read to compare back to the request. This was also ruled out because the SessionID was also duplicated and that is a read only value.

What is was

It is a feature/bug in IIS7. This latest version of IIS introduced some new caching features.

IIS7 automatically caches static content, such as HTML pages, images, and style sheets.
IIS7 now has the ability to cache dynamic content as well.

Caching dynamic content is great if it is a page such as a dynamically generated image gallery, or a page that is generated dynamically based on the browser’s culture. However, in this thread, http://forums.iis.net/t/1154347.aspx, Anil Ruia, a Senior Software Design Engineer on the IIS Core, explains, “You should not be enabling output caching for any response which depends on session state.”

If the page generating the content depends on the session state, it caches the session object along with the rest of the page. The next user to come through ends up pulling the cached session, instead of getting a new one. When I checked our settings I found it was set to cache all .aspx pages for three minutes, including many pages that access the session.

The solution

Edit Cache Rule

In IIS7, disable the caching for .aspx pages in any directory with an asp.net page that depends on the session state. To do this:

Run the Server Management console.
Navigate to Roles -> Web Server (IIS) -> Internet Information Services.
Select the site you wish to modify.
Select the folder that contains the .aspx pages you need to turn caching off for.
In the Feature View, double-click “Output Caching”.
If there is a rule there already for the .aspx extension double click it. Otherwise right click and select “Add…”
Enter .aspx for the “File name extension”
Check “User-mode caching”
Select “Prevent all caching”
Check “Kernel-mode caching”
Select “Prevent all caching”
Click OK
Close the Server management Console

Comments

[Kev]

Is there any Active Directory involved here?

[B Z]

@Kev Yes, the network uses AD

[uSlackr]

if the same workstations are involved with the bad requests, perhaps the users are not logged into their domain account

[Greg Askew]

You need to elaborate on "users are getting authenticated as different users than themselves.". How are you making this determination? Are users seeing data that belongs to another session?

[B Z]

@GregAskew I can see in the IIS Logs the same cs-username across different requests, different IPs. And yes, the application displays data based on the authenticated user's identity (WindowsIdentity)

[B Z]

@uSlackr they are different workstations, in different locations

[B Z]

Thanks Greg, will check the links more thoroughly but we are not using session state and we are not using output caching except for 1 action in the entire application.

[Greg Askew]

This seems like it may be an IIS bug. You may want to open a case with MS.