How to change the dag bag folder for Airflow web ui?

Question

Airflow scheduler is picking up the dags from the correct folder as per set in the airflow.cfg file. However, Airflow ui webserver is picking the dags from wrong folder. I realised this via looking at the stdout of the terminal in which airflow webserver is running: [2018-10-08 14:26:48,519] [15535] {models.py:167} INFO - Filling up the DagBag from /home/aviral/airflow/dags

Note, my airflow.cfg file has:

# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /home/aviral/dev/airflow_learning/airflow_home/dags

Solution 1

It looks to me like you have multiple Airflow installations. I think you have to check the UI in the Admin -> Configuration menu.

That will tell you which Airflow is active and set as AIRFLOW_HOME and AIRFLOW_CONFIG.

If the AIRFLOW_CONFIG environment variable is not set, it will always default to your home directory. You'll have to set it to something like this: AIRFLOW_CONFIG=$AIRFLOW_HOME/airflow.cfg.

Solution 2

Change the Dag folder by using the following commands.

Open the .bashrc or .zshrc user file of the terminal

vi ~/.bashrc

Add the below commands in the file

 export AIRFLOW_HOME='/var/www/html/airflow'
 export AIRFLOW_CONFIG=$AIRFLOW_HOME/airflow.cfg

Run the .bashrc file by using the below command.

source ~/.bashrc

Now if you run airflow webserver, it will pick the dags from the AIRFLOW_HOME/dags directory.

Comments

[chris.mclennon]

Are you sure the webserver is using the same config file?

[aviral sanjay]

It was my folly, forgot to put the same environment variable.