DAG not visible in Web-UI

Question

I am new to Airflow. I am following a tutorial and written following code.

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
from models.correctness_prediction import CorrectnessPrediction

default_args = {
    'owner': 'abc',
    'depends_on_past': False,
    'start_date': datetime.now(),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

def correctness_prediction(arg):
    CorrectnessPrediction.train()

dag = DAG('daily_processing', default_args=default_args)

task_1 = PythonOperator(
    task_id='print_the_context',
    provide_context=True,
    python_callable=correctness_prediction,
    dag=dag)

On running the script, it doesn't show any errors but when I check for dags in Web-UI it doesn't show under Menu->DAGs

But I can see the scheduled job under Menu->Browse->Jobs

I also cannot see anything in $AIRFLOW_HOME/dags. Is it supposed to be like this only? Can someone explain why?

Solution 1

The ScheduleJob that you see on the jobs page is an entry for the Scheduler. Thats not the dag being scheduled.

Its weird that your $AIRFLOW_HOME/dags is empty. All dags must live within the $AIRFLOW_HOME/dags directory (specifically in the dags directory configured in your airflow.cfg file). Looks like you are not storing the actual dag in the right directory (the dags directory).

Alternatively, sometimes you also need to restart the webserver for the dag to show up (though that doesn't seem to be the issue here).

Solution 2

Run airflow list_dags to check, whether the dag file is located correctly.

For some reason, I didn't see my dag in the browser UI before I executed this. Must be issue with browser cache or something.

If that doesn't work, you should just restart the webserver with airflow webserver -p 8080 -D

Solution 3

I have the same issue. To resolve I need to run scheduler

airflow scheduler

Without this command, I don't see my new DAGs BTW: the UI show me warning related to that problem:

The scheduler does not appear to be running. Last heartbeat was received 9 seconds ago. The DAGs list may not update, and new tasks will not be scheduled.

Solution 4

We need to clarify several things:

1. By no means you need to run the DAG file yourself (unless you're testing it for syntax errors). This is the job of Scheduler/Executor.
2. For DAG file to be visible by Scheduler (and consequently, Webserver), you need to add it to dags_folder (specified in airflow.cfg. By default it's $AIRFLOW_HOME/dags subfolder).

Airflow Scheduler checks dags_folder for new DAG files every 5 minutes by default (governed by dag_dir_list_interval in airflow.cfg). So if you just added a new file, you have two options:

1. Restart Scheduler
2. Wait until current Scheduler process picks up new DAGs.

Solution 5

Check the dags_folder variable in airflow.cfg. If you have a virtual environment then run the command export AIRFLOW_HOME=$(pwd) from the main project directory. Note that running export AIRFLOW_HOME=$(pwd) expects your dags to be in a dags subdirectory in the project directory.

View more solutions

Comments

[Rusty]

Do I need to run the script mentioned in the question in $AIRFLOW_HOME/dags folder ?

[Vineet Goel]

Yes, thats right. All your dag definitions (python files initialize dags - the line dag = DAG(...) in your example above) should be in the global scope within the DAGs dir configured in your airflow.cfg file.

[Eric Bellet]

Do you know how to fix the browser UI problem?

[samu]

@EricBellet for me airflow list_dags helped as quick fix, I don't know the root cause for this

[Eric Bellet]

Yes. Restarting the UI with airflow webserver -p 8080 -D it is other quick fix

[Stephen]

Sometimes even this takes a while to work. I had an experience just now where I followed all of the instructions in this answer, but it still took about 3 minutes for the new DAG to show up in the UI. At some point maybe I'll dig into the configuration settings to see if this is a refresh frequency that can be tweaked.

[Doug F]

Ah for me, that was it -- i didn't have the scheduler running to pick up new dags. thanks!

[ChristopherTull]

I had a DAG that was throwing an error, but rather than the error propagating to the UI, the DAG just wouldn't show up. Running airflow list_dags allowed me to see the error and debug that way. I am using an older version of Airflow.

[Requin]

For Airflow 2, try airflow dags list