Hyper-V Manager "RPC service unavailable"

Question

In an Active Directory domain and flat network environment, running Hyper-V Manager as the same administrative user account:

1. On at least 2 Windows 10 PCs and trying to connect to:
   1a. Any of the 10 Hyper-V Server 2012 R2 servers fails with error RPC server unavailable. Unable to establish communication between '<Hyper-V server FQDN>' and '<client PC hostname>'.
   1b. Any of the 2 Hyper-V Server 2016 servers succeeds.
2. On a Windows Server 2012 R2 Domain Controller server (I know, I know), and trying to connect to:
   1a. Any of the 10 Hyper-V Server 2012 R2 servers succeeds.
   1b. Any of the 2 Hyper-V Server 2016 servers succeeds.

So, the root cause seems to be connectivity between the affected clients and servers.

Elevated command cscript "<path>\hvremote.wsf" /show /target:%affectedServer% /override outputted the following (anonymised):

Microsoft (R) Windows Script Host Version 5.812
Copyright (C) Microsoft Corporation. All rights reserved.


Hyper-V Remote Management Configuration & Checkup Utility
John Howard, Hyper-V Team, Microsoft Corporation.
http://blogs.technet.com/jhoward
Version 1.08 9th Sept 2013

INFO: Computername is %affectedClient%
INFO: Computer is in domain %ADDNSDomainName%
INFO: Current user is %ADNetBIOSDomainName%\ben.hooper
INFO: OS is 10.0.16299 64-bit Microsoft Windows 10 Enterprise
INFO: Assuming /mode:client as the Hyper-V role is not installed
WARN: User override to assume Windows 8.1/Windows Server 2012 R2 behaviour
INFO: Hyper-V Tools are enabled

-------------------------------------------------------------------------------
DACL for COM Security Access Permissions
-------------------------------------------------------------------------------

\Everyone    (S-1-1-0)
     Allow: LocalLaunch RemoteLaunch (7)

NT AUTHORITY\ANONYMOUS LOGON    (S-1-5-7)
     Allow: LocalLaunch (3)

BUILTIN\Distributed COM Users    (S-1-5-32-562)
     Allow: LocalLaunch RemoteLaunch (7)

BUILTIN\Performance Log Users    (S-1-5-32-559)
     Allow: LocalLaunch RemoteLaunch (7)

APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES    (S-1-15-2-1)
     Allow: LocalLaunch (3)

\    (S-1-15-3-1024-2405443489-874036122-4286035555-1823921565-1746547431-2453885448-3625952902-991631256)
     Allow: LocalLaunch (3)

-------------------------------------------------------------------------------
ANONYMOUS LOGON Machine DCOM Access
-------------------------------------------------------------------------------

ANONYMOUS LOGON does not have remote access

  This setting should only be enabled if required as security on this
  machine will be lowered. This computer is in a domain. It is not
  required if the server(s) being managed are in the same or trusted
  domains.

  Use hvremote /mode:client /anondcom:grant to turn on if required

  Both computers are in domain %ADDNSDomainName%


-------------------------------------------------------------------------------
Firewall Settings for Hyper-V Management Clients
-------------------------------------------------------------------------------

Domain Firewall Profile is active
Public Firewall Profile is active


-------------------------------------------------------------------------------
IP Configuration
-------------------------------------------------------------------------------


%IPConfig%


-------------------------------------------------------------------------------
Stored Credentials
-------------------------------------------------------------------------------


%storedCredentials%



-------------------------------------------------------------------------------
Testing connectivity to server:%affectedServer%
-------------------------------------------------------------------------------

1: - Remote computer network configuration
     PASS - Found one or more network adapters

     Network adapter 1 of 1
       - Hyper-V Virtual Ethernet Adapter #2
       - Host Name:%affectedServer%
       - IP Addresses: 172.16.100.111 fe80::3cb0:1dda:7e90:c948
       - IP Subnets: 255.255.255.0 64

2: - Remote computer general information
     PASS - Queries succeeded

     - Name: %affectedServer%
     - Domain: %ADDNSDomainName%
     - OS: 6.3.9600 64-bit Microsoft Hyper-V Server 2012 R2
     - OS Type: Server

3: - Ping and resolve name of remote computer
     PASS - Server found
          - Protocol Address:             172.16.100.111
          - Protocol Address resolved:    172.16.100.111

     The name could not be resolved using WMI. A regular ping will be done
     to see if name resolution is working. This is not a sign of an issue.

4: - Ping %affectedServer% using IPv4

     A timeout is OK, but if you get an error that %affectedServer%
     could not be found, you need to fix DNS or edit
     \windows\system32\drivers\etc\hosts.

     >
     > Pinging %affectedServer%.%ADDNSDomainName% [172.16.100.111] with 32 bytes of data:
     > Reply from 172.16.100.111: bytes=32 time=2ms TTL=128
     >
     > Ping statistics for 172.16.100.111:
     >     Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
     > Approximate round trip times in milli-seconds:
     >     Minimum = 2ms, Maximum = 2ms, Average = 2ms
     >


5: - Ping %affectedServer% using IPv6

     A timeout is OK, but if you get an error that %affectedServer%
     could not be found, you need to fix DNS or edit
     \windows\system32\drivers\etc\hosts.

     > Ping request could not find host %affectedServer%. Please check the name and try again.
     >


6: - Connect to root\cimv2 WMI namespace
     PASS - Connection established

7: - Connect to root\virtualization\v2 WMI namespace
     PASS - Connection established

8: - Simple query to root\cimv2 WMI namespace
     PASS - Simple query succeeded

9: - Simple query to root\virtualization\v2 WMI namespace
     PASS - Simple query succeeded
          - 3 computer system(s) located

10: - Async notification query to root\virtualization\v2 WMI namespace
     FAIL - Notification query failed The RPC server is unavailable.

There may be a DNS issue and the server cannot locate this machine.
You should check this by performing a ping test from the server to
this machine verifying that the IP address the server is trying to
reach matches the IP address of this machine shown in the output above.
Note that it does not matter if the ping succeeds or fails, just that
the IP address is correct.

  Run on %affectedServer%: ping -4 %affectedClient%

Note that if you do not have DNS in your infrastructure, you can edit
the \windows\system32\drivers\etc\hosts file on the server to add an
entry for %affectedClient%

If you do have DNS in your infrastructure, you may want to try flushing
the DNS cache on the server, and re-registering against DNS on the client

 Run on %affectedServer%: ipconfig /flushdns
 Run on %affectedClient%: ipconfig /registerdns

If you are connected over a VPN, see %StackExchange-bannedURL% for
information about another likely cause.

If the server is in an untrusted domain to this client, you need to
enable anonymous logon access to DCOM on this machine:

  Run 'hvremote.wsf /mode:client /anondcom:grant' and retry.

If this machine has IPSec policy enforced on it, and the server is in a
workgroup or untrusted domain from this computer, inbound connections
to the client may be blocked by your administrator. You may be able to
temporarily work around this by running net stop bfe on this machine,
but you may lose access to some network services while that service
is stopped. However, this may be against the policy of your administrator.

If the server is behind a router/firewall from the client, WMI/DCOM
calls may be being blocked. This will likely be the case if the server
is, for example, on a public IP directly to the Internet. In this
situation, some solutions to consider are:
 - VPN to tunnel traffic
 - Publish Hyper-V Manager through a TS/RD Gateway
 - Access the server through RDP and perform 'local' management
 - Run a management machine (physical or virtual) and RDP to that

There have been several instances of third party firewalls and/or
anti-virus software having adverse effects on remote management.
In some cases, disabling that software has not been sufficient to
resolve the issue, and it has been necessary to completely remove
the program.



INFO: Are running the latest version

-------------------------------------------------------------------------------
3 warnings or errors were found in the configuration. Review the
detailed output above to determine whether you need to take further action.
Summary is below.

1: Running on a later OS than tested on. User override given
2: Cannot perform async WMI query. See detailed resolution steps above.
3: Some tests were not run due to prior failures

-------------------------------------------------------------------------------
INFO: HVRemote complete

I have thoroughly researched this error and no one has this exact problem / environment but, in any case, no recommended fixes (verifying that domain trust relationships are healthy, verifying that DNS is working correctly on both ends, verifying local administrative permissions, verifying local allowing ICMPv4 redirects, etc) have worked, hence this post.

Important notes:

1. This used to work until Cyber Essentials Plus-related changes.
2. The Hyper-V 2012 R2 and 2016 servers' computer objects are in the same OU so the same GPOs are applying.

 

Update: 2017/12/22 08:47:

According to https://social.technet.microsoft.com/Forums/en-US/268cd630-7fa3-49c8-a98e-21c458c7b7bb/win10-1709-update-broke-hyperv?forum=win10itprovirt, this is:

• Caused by Windows 10 v1709. This fits in with what I've seen.
• Can be resolved by allowing WMI inbound connections in the Windows Firewall of the Hyper-V server. I haven't yet been able to test this but I'll update this when I have.

 

Update: 2017/12/27 14:15:

I created a GPO to reconfigure Windows Firewall creating rules to allow inbound WMI connections and verified that it applied successfully by:

1. Executing gpresult /v and checking the output.
2. Connecting regedit and checking the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\WindowsFirewall\FirewallRules.

However, the problem remains, even after completely disabling the server's Windows Firewall on all profiles.

Solution

The WMI inbound connections need to be allowed in the Windows Firewall on the PC, not the Hyper-V server.

Comments

[Rumi]

To be precise: The incoming rule "Windows Management Instrumentation (ASync-In)" has to be enabled. This solved for me too.