AWS ECS Error when running task: No Container Instances were found in your cluster

Question

Im trying to deploy a docker container image to AWS using ECS, but the EC2 instance is not being created. I have scoured the internet looking for an explanation as to why I'm receiving the following error:

"A client error (InvalidParameterException) occurred when calling the RunTask operation: No Container Instances were found in your cluster."

Here are my steps:

1. Pushed a docker image FROM Ubuntu to my Amazon ECS repo.

2. Registered an ECS Task Definition:

aws ecs register-task-definition --cli-input-json file://path/to/my-task.json 

3. Ran the task:

aws ecs run-task --task-definition my-task

Yet, it fails.

Here is my task:

{
  "family": "my-task",
  "containerDefinitions": [
    {
        "environment": [],
        "name": "my-container",
        "image": "my-namespace/my-image",
        "cpu": 10,
        "memory": 500,
        "portMappings": [
            {
                "containerPort": 8080,
                "hostPort": 80
            }
        ],
        "entryPoint": [
            "java",
            "-jar",
            "my-jar.jar"
        ],
        "essential": true
    }
  ]
}

I have also tried using the management console to configure a cluster and services, yet I get the same error. How do I configure the cluster to have ec2 instances, and what kind of container instances do I need to use? I thought this whole process was to create the EC2 instances to begin with!!

Solution 1

I figured this out after a few more hours of investigating. Amazon, if you are listening, you should state this somewhere in your management console when creating a cluster or adding instances to the cluster:

"Before you can add ECS instances to a cluster you must first go to the EC2 Management Console and create ecs-optimized instances with an IAM role that has the AmazonEC2ContainerServiceforEC2Role policy attached"

Here is the rigmarole:

1. Go to your EC2 Dashboard, and click the Launch Instance button.

2. Under Community AMIs, Search for ecs-optimized, and select the one that best fits your project needs. Any will work. Click next.

3. When you get to Configure Instance Details, click on the create new IAM role link and create a new role called ecsInstanceRole.

4. Attach the AmazonEC2ContainerServiceforEC2Role policy to that role.

5. Then, finish configuring your ECS Instance.
NOTE: If you are creating a web server you will want to create a securityGroup to allow access to port 80.

After a few minutes, when the instance is initialized and running you can refresh the ECS Instances tab you are trying to add instances too.

Solution 2

Currently, the Amazon AWS web interface can automatically create instances with the correct AMI and the correct name so it'll register to the correct cluster.

Even though all instances were created by Amazon with the correct settings, my instances wouldn't register. On the Amazon AWS forums I found a clue. It turns out that your clusters need internet access and if your private VPC does not have an internet gateway, the clusters won't be able to connect.

The fix

In the VPC dashboard you should create a new Internet Gateway and connect it to the VPC used by the cluster. Once attached you must update (or create) the route table for the VPC and add as last line

0.0.0.0/0 igw-24b16740  

Where igw-24b16740 is the name of your freshly created internet gateway.

Solution 3

I ran into this issue when using Fargate. I fixed it when I explicitly defined launchType="FARGATE" when calling run_task.

Solution 4

Other suggested checks

1. Selecting the suggested AMI which was specified for the given region solved my problem.

   To find out the AMI - check Launching an Amazon ECS Container Instance.

2. By default all the ec2 instances are added to default cluster . So the name of the cluster also matters.

See point 10 at Launching an Amazon ECS Container Instance.

More information available in this thread.

Solution 5

Just in case someone else is blocked with this problem as I was... I've tried everything here and didn't work for me.

Besides what was said here regards the EC2 Instance Role, as commented here, in my case only worked if I still configured the EC2 Instance with simple information. Using the User Data an initial script like this:

#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_CLUSTER=quarkus-ec2
EOF

Informing the related ECS Cluster Name created at this ecs config file, resolved my problem. Without this config, the ECS Agent Log at the EC2 Instance was showing an error that was not possible to connect to the ECS, doing this I've got the EC2 Instance visible to the ECS Cluster.

After doing this, I could get the EC2 Instance available for my EC2 Cluster:

The AWS documentation said that this part is optional, but in my case, it didn't work without this "optional" configuration.

View more solutions

Comments

[sanath_p]

Selecting the suggested ami which was specified for the given region solved my problem. To find out the ami you should select check this url docs.aws.amazon.com/AmazonECS/latest/developerguide/…

[Justin M. Keyes]

This is mentioned at docs.aws.amazon.com/AWSEC2/latest/UserGuide/… [EC2-VPC] Check the route table for the subnet. You need a route that sends all traffic destined outside the VPC to the Internet gateway for the VPC.

[BlueDolphin]

not seeing AmazonEC2ContainerServiceforEC2Role, is there any change since your last post?

[Mikepote]

Thank you for the clue! I had assigned a security group to my instances which blocked all outbound traffic except for traffic to my load balancer. I switched this to allow all outbound traffic but restrict incoming traffic from the load balancer only, and the instances then appeared in my EC2 cluster.

[cyrf]

@BlueDolphin I was able to find that policy when I created a new Role in IAM.

[Rodrigo]

you just saved me, I would vote this twice if I could

[declension]

It's worth noting now (April 2018) if you're using the currently Beta Fargate (aws.amazon.com/fargate) to avoid this whole type of hassle, you might like me just have selected EC2 as the launch type when you got this error...

[Jakub Czaplicki]

And, when creating EC2, don't forget to add #!/bin/bash echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config in Advanced Details -> User data if you have your own, non Default cluster. Other wise the newly created EC2 instance will create Default cluster.

[alexkb]

I wish I could up vote your answer more than once. This fixed my exact problem "No Container Instances were found in your cluster." and the original answer voted more had nothing to do with your actual solution. Thank you again.

[ronit]

hi..i am facing the same problem ..my ec2 instance is not launched on creating a new cluster other then t2.micro...and then leads to same error on creatig any service on that cluster. i check that i have 'AmazonEC2ContainerServiceforEC2Role' attached to ecsinstance Role and i explicitly added this as a policy to my IAM user. BUt still the same issue. ANy Help!!!

[Greg]

As well as associating the gateway with your VPC, you may need to route internet traffic through it for the specific subnet that houses your ECS instances. - for eg aws ec2 associate-route-table --subnet-id $SUBNET_ID --route-table-id $ROUTE_TABLE_ID - can I suggest adding this to the answer?

[fred271828]

When you copy an existing launch configuration and change its name, the ecsInstanceRole "IAM Instance Profile" gets defaulted back to None, and even if you're using an ecs-optimized AMI you still get the same "No container instances were found" message. That is the piece I was missing.

[iomv]

Why has it got to be so convoluted and in need of obscure steps, I wonder? It should be supposed to be easy! thank you Sir you saved my day

[Totty.js]

It's interesting that I randomly started a EC2 and create a cluster and it just worked... I didn't had to do anything. Now I tried again and doing randomly stopped working. I did by your instructions and it's still not finding a container.

[Henrik Pingel]

I think it is worth mentioning that the moment you add a route for 0.0.0..0/0 pointing to a IGW the subnet is no longer a private subnet. From a network security perspective it is better to create a route pointing to a NAT-Gateway.

[shlomiLan]

With the CLI I added --launch-type FARGATE

[Martynas Jusevičius]

@declension I'm using Fargate and I'm getting this error when launching as EC2 type

[Lars]

Thanks Milan, that worked for me even with Python API.

[jon077]

Totally fixed it for me. The answer above didn't make sense, because I didn't need to do that from the wizard.

[Snowcrash]

Lol, AWS ECS - making deploying containers easy...!

[Naveen Reddy Marthala]

This helped me to start a container. But, port mapping didn't happen. more details: stackoverflow.com/questions/64632279/…

[dz902]

This is great, also try to remember "auto-assign public IP", which you may forget even routes to igw is set. If you got it wrong, no worries, stop the ECS container instance in EC2, it will actually terminate itself and restart a new one then register.

[rayepps]

Same same. For clarity, there are only 2 launch types: 'FARGATE' or 'EC2'. AWS docs don't show it well but the ECS default is EC2. EC2 requires provisioned instances to start the containers for a task in. So its a little obscure but if your using fargate but you don't specify the fargate launch type your running into the classic 'no container instance' issue seen here

[Dustin Oprea]

This wasn't the solution in my case and was already configured as such.

[Dustin Oprea]

Didn't work for me. The permission was already assigned (automatically?) and finding/choosing a supposed ecs-optimized AMI didn't change anything.

[Admin]

Thats correct! but when try to create a taks says "RESOURCE:MEMORY (Error)", memory loss for t2.small, moral, can not be used under free tier

[Ubaid Qureshi]

if anyone dont want to use ecs-optimised then you can add ecs agent to any regular AMI docs.aws.amazon.com/AmazonECS/latest/developerguide/…

[Matt The Ninja]

This is the stuff i just love SO for.

[flowfelis]

There is also 'EXTERNAL' launch type. Feel free to check the docs --> boto3.amazonaws.com/v1/documentation/api/latest/reference/…