ECS task not starting - STOPPED (CannotPullContainerError: “Error response from daemon request canceled while waiting for connection”

Question

I'm starting a task in ECS using Fargate and after being in PENDING for a little bit it ends up in STOPPED with the following error:

STOPPED (CannotPullContainerError: "Error response from daem

When I expand out the details I see

STOPPED (CannotPullContainerError: "Error response from daemon: Get https://id.dkr.ecr.ap-southeast-2.amazonaws.com/v2/: net/http: request canceled while waiting for connection"

with the reason

(Client.Timeout exceeded while awaiting headers)

So the task can't access the container for some reason, but I'm not sure what permission is missing and from what resource. I've read around a bit and the only real suggestion I've found is to add AssignPublicIp: ENABLED to the AwsvpcConfiguration but that didn't help.

Solution 1

So it looks the error message has changed at some point: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-pull-container-api-error-ecr/ has steps to work through but mentions the error CannotPullContainerError: API error which might be synonymous with CannotPullContainerError: "Error response from daem?

For me at least, creating an AWS::EC2::VPCEndpoint seems to have got me further.

Solution 2

I managed to fix this error by enabling the public IP for every Fargate instance created with my service on ECS.

Service configuration:

{
  ...
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": [
        "my-subnets",
      ],
      "securityGroups": [
        "my-security-group"
      ],
      "assignPublicIp": "ENABLED" // <-- ENABLED HERE
    }
  },
}

Solution 3

I found a solution that worked for me using Fargate. Their documentations states:

1. If you're running a task using an Amazon Elastic Compute Cloud (Amazon EC2) launch type and your container instance is in a private subnet, or if you're running a task using the AWS Fargate launch type in a private subnet, confirm that your subnet has a route to a NAT gateway in the route table.

That simply means,

1. you have to find VPC that you use.
2. Then in the table where you have your VPC, you will find its main route table.
3. Open the route table, then make sure you have entry that link to internet gateway.

It would look something like igw-006b1917dc348d10d. Once you have, your vpc will have access to the internet, and will be able to fetch your ECR image.

Source: AWS docs

Solution 4

This is an error comes when you are not able to pull the image..it may have many reasons like permission and Internet access inside the VPC.

If your VPC is Public only subnet then you need to add Internet gateway for Internet access. If your VPC is Private only then you need a NAT Gateway so that task can reach to docker image to pull.

Solution 5

Alan Sereb's solution worked for me.

It seems after AWS launched Fargate platform version 1.4.0, access to remote image registry (like Gitlab Registry in my case) is done using ECS Service configured VPC.

So now the Fargate container network interfaces (and therefore the VPC used by ECS) need to have internet access, so setting up Internet Gateway in the VPC Routing Table is mandatory.

View more solutions