Exchange 2010 - EMC and OWA non-functional after unexpected server reboot

Question

Problem:

Can't connect to OWA or the Exchange Management Console/Shell on our Exchange 2010 server (Server 2008R2 VM).

Background:

We have an Exchange 2010 VM (ESXi 5.1) we're "in the process" of doing a mail migration to (from Exchange 2003). We're currently in a co-existence situation, and not actively migrating users, though we have a small number on the Exchange 2010 server at the moment.

We had a recent "incident" that caused our ESXi host to go down, and since that time, OWA has been non-functional - clients see either a connection time out trying to connect to OWA, or a 500 error after logging in (we now run OWA off the Exchange 2010 server, and redirect to the Exchange 2003 server for everyone who's still on the older server), and going in to open up the Exchange Management Console or Shell errors out.

Error Information:

When opening the Exchange Management Shell, we get the below:

Connecting to remote server failed with the following error message : The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.

    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [], PSRemotingTransportExc
   eption
    + FullyQualifiedErrorId : PSSessionOpenFailed
Failed to connect to an Exchange server in the current site.

The EMC throws throws a similar, but different error message, below:

The following error occurred while attempting to connect to the specified Exchange server 'ourexchangeserver.domain.tld':

The attempt to connect to http://ourexchangeserver.domain.tld/PowerShell using "Kerberos" authentication failed: Connecting to remote server failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.

We have two relevant error messages in the event log, that occur with a high frequency:

IIS-W3SVC-WP Event 2214, The HTTP Filter DLL E:\Program Files\Exchange Server\V14\ClientAccess\owa\auth\owaauth.dll failed to load. The data is the error.

and

IIS-W3SVC-WP Event 2268, Could not load all ISAPI filters for site 'DEFAULT WEB SITE'. Therefore site startup aborted.

Details for both show binary data of 0000007E ("In Words") / 7E 00 00 00 ("In Bytes"), which translates to ~....

Attempted thus far:

Resolutions from the Microsoft KB here. It seems that none of the conditions listed under the "cause" section are actually true.

The EMTshooter tool. Same as the support KB, it lies and says the cause is one of a missing WSMan entry that's present, a misconfigured Kerbauth module that's configured exactly as it should be or a modified PowerShell virtual directory path, which points at (and always has pointed at) the ..\Exchange Server\v14\ClientAccess\PowerShell directory.

Any thoughts or suggestions that don't involve setting myself on fire and jumping out a window would be appreciated.

Solution

Installed SP2, which corrected the problem.

Disappointing, but we couldn't very well justify leaving it busted to figure out what happened - getting it fixed had to take priority.

Comments

[Florian Haider]

you may be able to gain EMS functionality if you open a regular powershell instance and run "Import-Module <path to servername.psm1>". For some reason, the Exchange installer puts it in the path of the user who installed Exchange (C:\Users\<username>\AppData\Roaming\Microsoft\Exchange\Remo‌​tePowerShell\<server‌​name>.psm1). Then you should hopefully be able to directly manage Exchange and turn on diagnostic logging, run tests, etc...

[HopelessN00b]

@August Thanks, that seems to have worked. (And seems like it might be worth at least an upvote if you were to make it an answer...) Now, off to read up on how to admin Exchange through PS.

[Florian Haider]

Well it isn't an answer but at least gets you some functionality back to try and figure out the answer. EVERYTHING in Exchange 2010 is managed through PS. The EMC is just a GUI for PS cmdlets. You might try resetting the OWA virtual directories. Looks like this is a combo of Get-OwaVirtualDirectory, writing it to a file, followed by Remove-OwaVirtualDirectory, and New-OwaVirtualDirectory using the file config you exported. I think I encountered this same problem over a year ago. Unfortunately I don't remember what I did to fix it :(

[HopelessN00b]

@August Aww, $(*&^ing hell. Well it imported the module successfully, but is throwing the same connection error when running an Exchange-specific cdmlet: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. [...] Damn you, Exchange!!! Well, it's still an answer worthy of an upvote if you want it.

[Florian Haider]

Crap...I want to say this is due to incorrect SSL and/or auth settings on the OWA virtual directories... I'll keep looking for something to jog my memory.

[Florian Haider]

Default Authentication Settings for Exchange-related Virtual Directories - technet.microsoft.com/en-us/library/…

[HopelessN00b]

@August Alright, will take a look... also looking at running an Exchange update as a possible resolution.

[Florian Haider]

Actually I finally found that the resolution to the similar problem we had WAS an Exchange update. It was an anti-climatic discovery for me, b/c I definitely remember messing with the IIS virtual directory perms, but they didn't fix our issue, the update did.

[HopelessN00b]

Yes, all roles, the time is correct and domain authentication to/from this server is working properly. DCDiag shows nothing wrong.

[HopelessN00b]

Well, tried 3, no joy. 1 and 2 aren't exactly accurate, since we installed Exchange on its own drive, and left out the Microsoft part of the path, but those haven't changed since the install, and seem to be referenced properly everywhere (IIS manager, system variables). Output of command: PS > winrm quickconfig WinRM already is set up to receive requests on this machine. WinRM is not set up to allow remote access to this machine for management.

[HostBits]

interesting that SP2 fixed the problem... Very strange.