Can I force delete an AWS CloudFormation stack that is In Progress of Rollback

Question

An AWS CloudFormation rollback (e.g., UPDATE_ROLLBACK_IN_PROGRESS) has been in progress forever, like over an hour and a half. I want to delete the stack altogether or force stop any activity. Is this possible?

Thanks!

Solution 1

Another common cause of blocked stack updates/rollbacks is errors in ECS::Service resource updates: it doesn't look like that is currently detected (in some cases?). Cloudformation is waiting for the service event for the service reaching a steady state, so simply updating the service to something that works (e.g. desired tasks to 0) will unblock it. Try to get the state back to what Cloudformation expects before sending more updates to avoid problems, though.

Solution 2

I guess your stack resources are changed or deleted by outside. You can find official guide as below.

Manually sync resources so that they match the original stack's template, and then continue rolling back the update. For example, if you manually deleted a resource that AWS CloudFormation is attempting to roll back to, you must manually create that resource with the same name and properties it had in the original stack.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-update-rollback-failed

or (as @talentedmrjones said)

To fix the stack, contact AWS customer support.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-nested-stacks-are-stuck

In my case, I can stop same situation via re-creating deleted resource.

Solution 3

In my case it is an EC2 security group that cannot be deleted because it is referenced from another EC2 security group.

Solution 4

When dealing with a custom resource it is possible to construct a mocked up version of the return url.

The easiest way to do this is to grab the url which was used during the create. If you can get your hands on it, replace the section after the last %2F with the "Client Request Token" which you can get from your event log for the cloudformation.

If not, then here's the format of the url you'll have to construct.

https://{region}.console.aws.amazon.com/cloudformation/home?region={region}#/stacks?filter=active&tab=events&stackId={stack arn}%2F{stack name}%2F{client request token}

Run that url as a get and it will cause the resource to fail rollback or delete.

Solution 5

Sometimes this will occur if your user role is missing permissions to delete roles. This can be tested by trying to manually delete roles or users that have been created by the CloudFormation stack.

View more solutions

Comments

[Justin]

When clicking on the stack in the AWS Console while it's rolling back, it prevents me from deleting the stack.

[wjordan]

Oops my mistake, sorry! Can you cancel update then delete stack, perhaps?

[Justin]

I was able to cancel the update in progress as that was taking forever. It was at that point, the rollback in progress happened. At this stage, it appears I can no longer do anything besides wait, it seems.

[wjordan]

@Justin Depending on the exact type of resource, there might be something you can do to force the resource to complete. Could you provide more context?

[shogochiai]

Same thing for me. My case is during OOM error for deploying next.js project, I tried to scale down memory size, and UPDATE_ROLLBACK_IN_PROGRESS occured. aws-cli command (sometimes they have stronger rights than AWS GUI) was also validated by An error occurred (ValidationError) when calling the DeleteStack operation: Stack [arn:aws:cloudformation:ap-<region>-1:<id>:stack/<stackname>‌​/<id>] cannot be deleted while in status UPDATE_ROLLBACK_IN_PROGRESS error

[JLM]

Got a similar issue. Deploying with Cloudformation : - 3 hours for unsuccessful deployment - then 3 hours again for unsuccessful rollback My problem came from a combination of placement constraints and desired count which looks ok but my service never stabilized to reach a ready state. Not reaching steady state locks the Cloudformation updates and rollbacks. Thanks to Simon : changing the desired task count to 0 allowed me to stop the Cloudformation process.

[Stretch]

can you give more info on how to get the client request token please? All I can see for the custom resources is "physicalID" Also, how would one get the url that was used during the create?

[user2981810]

For a custom resource you need to look at the line where it says "CREATE_IN_PROGRESS", which is also expandable. Expand that line and it gives you two values: "Physical ID" and "Client Request Token". The "Client Request Token" is what you're looking for.

[andrew lorien]

In my case : two stacks with overlapping resources both trying to roll back but blocking each other. Setting desired tasks to 0 fixed it.

[lapkritinis]

Got same issue. In my case I had to recreated exactly same ECS::Service and it made rollback process unstuck. (I had deleted them)

[Felipe Alvarez]

saved our production deployment of transit gateway. Thank you.

[harryhare]

My answer address the OP's question very well. By force delete some resource in the stack, the stack can be finally deleted. OP said " I want to delete the stack altogether or force stop any activity." This answer provide exactly what they want.

[Slava Fomin II]

What if you don't have a paid support plan?

[Austin Poole]

Dang. That did it! This is such a glaring issue for cloudformation. AWS should create a simple solution to just force it to rerty again if user initiated

[Janusz]

Which lambda function in "Lambda Functions" do you mean? Which log group?