No Instances found when trying to use AWS Console and EC2 Run Command for Windows

Question

I am trying to use the Amazon EC2 Run Command feature, and basically it says it can't find any instances.

I read all the documentation and followed all the steps that I could find on this relatively new feature, and I cant get it to work for the life of me.

Here is the info:

I go to "EC2 -> Commands -> Command History -> Run A Command" I click radio box for the "AWS-RunShellScript" command document I click the "Select Instances" drop down button, and it says "No instances found in this region" as shown in Figure 1 below.

I click the "Where are my instances?" hyperlink, which brings me to the AWS documentation show here:"Troubleshooting SSM Run Command"

The first suggestion for troubleshooting is to make sure that the prerequisites have been met, and provides another hyperlink shown here: "SSM Run Command Prerequisites"

Prerequisites:

1. Supported OS - Windows 2012 R2 - CHECK
2. Latest Agent Version - 3.17.1032 - CHECK (Shown In Figure 2)
3. Access to SSM Run Command - Assign EC2 Instance Role and IAM User Role - CHECK (Shown In Figures 3 and 4)
4. Internet Access - Outbound Internet Access - CHECK

I followed the AWS documentation for creating an Amazon EC2 Instance role for EC2 Run Command Access, and also to create an IAM User with proper Run Command Access. The documentation is shown here: Delegating Access to SSM Run Command

I used the Amazon CLI from my local machine to run the following command also shown in troubleshooting documentation:

aws ssm describe-instance-information --instance-information-filter-list key=InstanceIds,valueSet=instance-ID

The command returns no information:

{ "InstanceInformationList": [] }

UPDATE 1:

I did find the following errors and warnings in the following places, and have spent the last two hours trying to troubleshoot. I uinstalled and reinstalled the Ec2Config Service. I checked all the Ec2Config files. I Checked the 169.254 addresses, and the meta-data and dynamic data exists, and appears to be correct. Not sure what else to try here.

C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog - Error/Warning

2016-06-22T23:44:12.663Z: Warning: Unable to Publish to WMI. | System.Management.Instrumentation.WmiProviderInstallationException: Exception of type 'System.Management.Instrumentation.WMIInfraException' was thrown. at System.Management.Instrumentation.InstrumentationManager.Publish(Object value) 2016-06-22T23:44:16.263Z: Failed to fetch instance metadata http://169.254.169.254/latest/user-data with exception The remote server returned an error: (404) Not Found. 2016-06-22T23:44:16.263Z: Failed to get metadata/user-data The remote server returned an error: (404) Not Found.

Windows Event Viewer Error

2016-06-22 23:46:59,758 [_Worker-2] ERROR [aws:getDocument] - Failed to update instance information., RequestId=9cb8f2dd-38d3-11e6-bc83-19c0650ffecc,ErrorCode=AccessDeniedException,ErrorType=Unknown,StatusCode=BadRequest,Message=Caller
instance identity does not match the given instanceId

UPDATE 2:

Per @praetorian1 's linked stackoverflow article I updated the Ec2Config Services "config.xml" file to enable the "Ec2HandleUserData" setting, rebooted, and still no change.

Per @Kai 's suggestion I also made sure that the "role" attached to the instance matched the "AmazonEC2RoleforSSM" policy, and specifically permitted "ssm:UpdateInstanceInformation". It did.

I also looked under the AWS EC2 Management Console, at the "System Log" file for the specific instance, which did not contain any errors, and towards the end said that SSM Config: status:Active, iam:Yes. Also noteworthy, it contained: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1

2016/06/23 15:01:29Z: EC2ConfigMonitorState: 0 2016/06/23 15:01:29Z: Windows sysprep configuration complete. 2016/06/23 15:01:32Z: AMI Origin Version: 2016.01.13 2016/06/23 15:01:32Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-SQL_2014_SP1_Web 2016/06/23 15:01:32Z: OS: Microsoft Windows NT 6.3.9600 2016/06/23 15:01:32Z: OsVersion: 6.3 2016/06/23 15:01:32Z: OsProductName: Windows Server 2012 R2 Standard 2016/06/23 15:01:32Z: OsBuildLabEx: 9600.18202.amd64fre.winblue_ltsb.160119-0600 2016/06/23 15:01:32Z: Language: en-US 2016/06/23 15:01:32Z: TimeZone: Eastern Standard Time 2016/06/23 15:01:32Z: Offset: UTC -04:00:00 2016/06/23 15:01:32Z: EC2 Agent: Ec2Config service v3.17.1032 2016/06/23 15:01:32Z: Driver: AWS PV Storage Host Adapter v7.3.2.0 2016/06/23 15:01:32Z: Driver: Intel(R) 82599 Virtual Function v1.0.15.3 2016/06/23 15:01:34Z: Message: Waiting for meta-data accessibility... 2016/06/23 15:01:34Z: Message: Meta-data is now available. 2016/06/23 15:01:37Z: Message: Windows is Ready to use 2016/06/23 15:01:40Z: Amazon EC2 Simple Systems Manager (SSM) is an optional service for custom configuration of instances. 2016/06/23 15:01:40Z: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1 2016/06/23 15:01:40Z: SSM Config: status:Active; iam:Yes

Figure 1:

Figure 2:

Figure 4:

Solution 1

Try checking the application logs in windows on the instance. Might be having issues talking to the SSM service for some reason.

Solution 2

As praetorian1 mentioned, Windows Event Viewer is the place to look for SSM - EC2 Run Command related log messages.

The message you posted indicates that the EC2 instance does not have required permission to call ssm:UpdateInstanceInformation API. Please double check that RunCommandInstance role has AmazonEC2RoleforSSM policy attached. (After attaching the policy, please restart EC2Config service or wait up to 15 min for EC2Config to start reporting instance information.)

For the IAM User in Fig. 4, we do not need policies other than AmazonSSMFullAccess for us to call SSM APIs with awscli.

To dive deeper:

EC2 Role and IAM Policy for the EC2 Instance

SSM on Windows requires an IAM EC2 Role attached to the EC2 instance at launch. Required permission for that Role is provided by an Amazon-managed IAM policy AmazonEC2RoleforSSM.

IAM Policy for IAM User

To make API calls to SSM service with awscli, an IAM user has to be setup with SSM permissions. For testing, Amazon provides a managed IAM policy AmazonSSMFullAccess. With the credentials of this IAM user, we will be able to make API call to describe instance information and also send command to the instance.

DescribeInstanceInformation API

Once the EC2 instance is setup with required permission and EC2Config starts reporting instance information back to SSM, we will be able to use DescribeInstanceInformation API to get a list of EC2 instances that are ready for SSM - EC2 Run Command.

Comments

[subverts_rule]

Good catch, I did find some errors in the various log files specified in the "Managing Windows Instance Configuration - Troubleshooting" AWS documentation. I have updated my original post to reflect the findings. Unfortunately after a couple hours of trying different things I still cant see the instances through the AWS EC2 Run Command Console.

[praetorian1]

Are you using one of the standard Windows AMIs or did you create a custom AMI from one of these. Wondering if it's something like stackoverflow.com/questions/26158411/…

[subverts_rule]

It is a custom AMI created off an instance running under the same account. I created the AMI, because the original instance it was created off of, DID NOT have an IAM role assigned to it. So I created AMI, and launched NEW instance with the IAM role assigned to it. I visited the link you suggested, and I changed the "config.xml" file to enable the "Ec2HandleUserData" setting, rebooted, and no change. I also followed @Kai suggestions below. Original post updated.

[subverts_rule]

Hi @Kai, Thanks for the response. You mention that the error indicates a problem with the role, how did you determine that, because the error seems to talk about mismatching identity data? Regardless, I did check the assigned role, and it does contain proper permission to update and describe instance information, it also contains the proper trust relationship. I have updated my original post with new information. Thanks!

[Kai]

Failed to update instance information. with AccessDeniedException indicates a permission issue to call UpdateInstanceInformation API. The best way to make sure the permissions are set correctly is to attach Amazon-managed policies.

[Kai]

2016/06/23 15:01:40Z: SSM Config: status:Active; iam:Yes shows that the instance is setup with correct permission and is ready to receive SSM - EC2 Run Command. If there's still no instances shown when trying to send a command, I believe this should be forward to AWS support.