ECS Fargate Scheduled Task not running

Question

I'm trying to setup a scheduled task with ECS Fargate but I cannot figure out why it is not running. I can confirm the task works correctly using RunTask but when I try to trigger it on a schedule all I get is a bunch of 'FailedInvocations' with no explanation.

I do know though that the rule is being triggered so this is a good sign. See the screenshot below:

But everytime it is triggered there is just a 'FailedInvocation'. Here's the scheduling rule:

And the default permissions on the ecsEventRole with just ecs:runTask:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

My hunch says that this ecsEventsRole doesn't have enough permissions. Should I try to give it the ones that ecsTaskExecutionRole has?

Thanks

EDIT: This is now supported in us-east-1 region. See comments.

Solution 1

Although it's been over an year, AWS still don't have a proper way to see invocation logs.

As you already know we can invoke tasks by RunTask manually, so does Scheduled Task.

The only difference is that Scheduled Task is triggered by CloudWatch rules.

We can easily see invoke logs in CloudTrail Event history, go there and filter events with Event name: RunTask and select the time range you want to check, find the event and click View Event, you'll see the error code and response.

Solution 2

I ran into a similar issue where regular ECS Scheduled Tasks were not running.

I finally resolved it by adding an additional policy to ecsEventsRole which allows CloudWatch Events to pass IAM roles to ECS Tasks:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListInstanceProfiles",
                "iam:ListRoles",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

Solution 3

Here is a possible workaround: use a lambda function as target for the cloudwatch rule and create the task in the lambda function code.

Here is an example code for the lambda function: https://lobster1234.github.io/2017/12/03/run-tasks-with-aws-fargate-and-lambda/

The links describes how to pack the new boto version with the lambda function but this is not necessary anymore since AWS already updated the lambda boto version to 1.4.8

I've tested and it works.

Solution 4

When defining a Rule in CloudFormation, I had to ensure that the Target RoleArn property referenced ecsEventsRole:

Targets:
- Id: !Sub ${AWS::StackName}-cron-test-1
  Arn: arn:aws:ecs:eu-west-1:<account id>:cluster/fargate-cluster
  RoleArn: !Join
    - ':'
    - - "arn:aws:iam"
      - ""
      - !Ref "AWS::AccountId"
      - "role/ecsEventsRole"

And where ecsEventsRole defined the AmazonEC2ContainerServiceEventsRole policy

View more solutions

Comments

[Marc Young]

Any updates? I'm at exactly the same spot and stuck

[coolboyjules]

Nope. I'm still stuck :( Given up on all options.

[Marc Young]

I talked to AWS support, its not supported yet. No ETA

[coolboyjules]

Brutal. Thanks Marc

[simon]

@MarcYoung exactly what is not supported?

[Marc Young]

Scheduled Fargate tasks are not supported. Only EC2 scheduled ecs tasks

[FBryant87]

Fargate tasks are now supported: aws.amazon.com/about-aws/whats-new/2018/08/…

[coolboyjules]

Works in us-east-1, but unavailable in ca-central-1. Assuming there are other problem regions.

[Chris]

I think this is hitting a few people because the first bit of documentation (docs.aws.amazon.com/AmazonECS/latest/developerguide/…) says 'You must add iam:PassRole permissions for any task role overrides to the CloudWatch IAM role', but it turns out that the task needs it to use a role already applied (explained in a different area of the documentation docs.aws.amazon.com/AmazonECS/latest/developerguide/… - 'If your scheduled tasks require the use of the task execution role or a task role override, then you must add iam:PassRole permissions').

[Marc Young]

Thats what I was trying to avoid but ended up with

[coolboyjules]

I ended up using this. Probably the most reasonable solution presently.

[tanvi]

make sure to edit the security group, subnet and account ID strings before running it - they're just placeholders in the command i used above.

[gileri]

In addition to ecs:RunTask, I only needed iam:PassRole on both the Task Role and Task Execution role resources.

[鄭元傑]

After I combined top 2 answer, added ecs:RunTask and "iam:ListInstanceProfiles", "iam:ListRoles", "iam:PassRole" to the cloudwatch event iam role, finally the task run successfully.

[Stephen Wright]

This is actually the correct answer. Errors in the actual config aside, this problem was exacerbated by the logs being hidden. I had no idea Cloudtrail even existed, never mind that it ingests internal AWS events.