Bash Operator error: No such file or directory in airflow

python bash airflow directed-acyclic-graphs

11,865

Solution 1

Try this:

bash_operator = BashOperator(
    task_id = 'task',
    bash_command = '${AIRFLOW_HOME}/myfirstdag/dag/lib/script.sh '
    dag = your_dag)

Solution 2

For those running a docker version.

I had this same issue, took me a while to realise the problem, the behaviour can be different with docker. When the DAG is run it moves it tmp file, if you do not have airflow on docker this is on the same machine. with my the docker version it moves it to another container to run, which of course when it is run would not have the script file on.

check the task logs carefully, you show see this happen before the task is run. This may also depend on your airflow-docker setup.

Solution 3

Try the following. It needs to have a full file path to your bash file.

cmd = "/home/notebook/work/myfirstdag/dag/lib/script.sh "

t_1 = BashOperator(
    task_id='start',
    bash_command=cmd
)

11,865

Marvin

Updated on July 14, 2022

Comments

Marvin almost 2 years
I am a newbie to Airflow and struggling with BashOperator. I want to access a shell script using bash operatory in my dag.py.

I checked: How to run bash script file in Airflow and BashOperator doen't run bash file apache airflow

on how to access shell script through bash operator.

This is what I did:
```
 cmd = "./myfirstdag/dag/lib/script.sh "

        t_1 = BashOperator(
            task_id='start',
            bash_command=cmd
        )
```
On running my recipe and checking in airflow I got the below error:
```
[2018-11-01 10:44:05,078] {bash_operator.py:77} INFO - /tmp/airflowtmp7VmPci/startUDmFWW: line 1: ./myfirstdag/dag/lib/script.sh: No such file or directory
[2018-11-01 10:44:05,082] {bash_operator.py:80} INFO - Command exited with return code 127
[2018-11-01 10:44:05,083] {models.py:1361} ERROR - Bash command failed
```
Not sure why this is happening. Any help would be appreciated.

Thanks !

EDIT NOTE: I assume that it's searching in some airflow tmp location rather than the path I provided. But how do I make it search for the right path.
Marvin over 5 years

I tried without the . too. It gives the same error.
Bsquare ℬℬ over 5 years

Try setting cmd="pwd" and gives us output of a find - ls in the corresponding directory?
Marvin over 5 years

I tried that already. It gives a tmp directory which gets deleted after every run. But I am not sure how do I set the path to the one where my script is present.
Bsquare ℬℬ over 5 years

Ok, so put the find -ls as the cmd itself, and we will see where is located the myfirstdag/dag/lib/script.sh
Marvin over 5 years

Seems like it cannot locate that using the find command. It's because it cannot find the directory itself. So I did cd /home ; ls -lrt. All I got is just one folder as airflow/ where as I have two other folders in it named example/ and notebook/ which isn't showing when I am doing it through the bashOperator. All I get is this: Running command: cd / ; cd home/; ls Output: airflow
Marvin over 5 years

have tried. Doesn't work. Please refer to the comments above.
kaxil over 5 years

Can you for the sake of debugging, try removing the .sh extension and the run with cmd=bash /home/notebook/work/myfirstdag/dag/lib/script and let me know it it works or not.
Bsquare ℬℬ over 5 years

You misunderstood me, I've updated my answer consequently for the command you should try.
Marvin over 5 years

This is what I got : Running command: find -ls Output: 22273487 4 drwx------ 2 root root 4096 Nov 5 08:46 . 22273488 4 -rw------- 1 root root 8 Nov 5 08:46 ./startRIKEG9
Bsquare ℬℬ over 5 years

Ok, so it's 100% sure, there is no myfirstdag/dag/lib/script.sh in a subdirectory of this temporary directory. So you can't either use the relative path ./myfirstdag/dag/lib/script.sh , or you priorly need to perform action to copy missing files.
Marvin over 5 years

yep. That's what I am looking for. Either how to copy the file to this location or access the path I gave.
Bsquare ℬℬ over 5 years

Where is located your script.sh file on your computer? In addition, maybe you should add dag=dag definition in your bash operator, See airflow.apache.org/tutorial.html#it-s-a-dag-definition-file
Bsquare ℬℬ over 4 years

@Marvin On Stackoverflow you can give up-vote to people's helpful answers to thank them and select any one of the answer as correct answer too out of all.
Anupam Kumar about 3 years

i am using docker version only. How did you solve the problem?