How to restart DAG in Airflow?

Question

One of my DAGs with many steps stopped part way through because of a DB shutdown. I'd like to get the DAG to start where it left off but all I can do is start the individual tasks of the DAG one by one. Is there a way to tell Airflow to start the DAG where it left off based on the success of the tasks already completed?

Here's a sample where one of the first tasks finished and the rest are queued or have no status:

When I click on an individual task (runme_1) I can restart it:

However, if I try to start a task which hasn't had it dependencies met (such as run_after_loop), I see the following error message:

I'd like to be able to click on any task, ask it to be queued and have all dependencies run in the correct order. Is that possible?

I'm running Airflow 1.9.0.

Solution 1

It sounds like you need to start a dag from the task where it failed.

Instead of going into the DAG and clicking on a task and clicking run, go to the Dag run with the failed task, click the failed task and click clear.

Clearing the task will re-run the task, and any other tasks after it will run.

Solution 2

To add to @jhnclvr you may also wish to:

1. Go to Browse > Task Instances, filter for and select the failed task(s), and Delete. This is essentially the same as clearing individual tasks in the DAG graph or tree view.

2. Go to Browse > DAG Runs, filter for and select the failed DAG(s), and Set state to 'running'. This needs to be done if failed tasks have caused the DAG to fail, as you will see indicated by a round red circle at the top of the tree view, as tasks will not automatically re-run if the DAG has failed.

This will automatically trigger a DAG re-run starting with the first unsuccessful task(s).

In order to minimize future manual intervention, you may wish to adjust configurations for the operator that fails - e.g. retries, timeout, retry delay, etc.

Comments

[Gabe]

I don't think #2 works.. at least not on 1.10.15. It's best to clear the task in tree view or task instances.