airflow startup failed due to gunicorn

Question

Traceback (most recent call last):
  File "./python_runtime/bin/airflow", line 15, in <module>
    args.func(args)
  File "/home/admin/airflow/python_runtime/lib/python2.7/site-packages/airflow/bin/cli.py", line 423, in webserver
    'gunicorn', run_args
  File "/home/admin/airflow/python_runtime/lib/python2.7/os.py", line 346, in execvp
    _execvpe(file, args)
  File "/home/admin/airflow/python_runtime/lib/python2.7/os.py", line 385, in _execvpe
    func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

I find out that this error was cause by no gunicorn avaiable,so I print all paths in os.py:

/home/admin/airflow/python_runtime/bin/gunicorn
/home/admin/python2.7.2/bin/gunicorn
/home/admin/tools/local/bin/gunicorn
/home/admin/tools/bin/gunicorn
/home/admin/tools/opsdb/gunicorn
/home/admin/odps_tools/dship/gunicorn
/home/admin/php/bin/gunicorn
/home/admin/cmake/bin/gunicorn
/home/admin/jdk1.6.0_24/bin/gunicorn
/home/admin/tools/bin/gunicorn
/usr/kerberos/bin/gunicorn
/usr/local/sbin/gunicorn
/usr/sbin/gunicorn
/sbin/gunicorn
/usr/local/bin/gunicorn
/bin/gunicorn
/usr/bin/gunicorn
/usr/X11R6/bin/gunicorn

while gunicorn exist in this path

/home/admin/airflow/python_runtime/bin/gunicorn

something wrong with my env?

Solution 1

Airflow seems to expect gunicorn in PATH. For my ubuntu instance this fixed it:

$ export PATH=$PATH:~/.local/bin $ airflow webserver

Solution 2

On MacOS, it wasn't enough with the proposed solution. If I used relative path (with either ~ or $HOME) it didn't work, but using absolute path solved the issue: PATH="${PATH}:/path/to/.venv/bin"
As you can see, I'm using a hidden folder that I created with Python venv, maybe this is connected.

To test it, is not enough to run gunicorn alone, this worked to me with the relative path as well, but which gunicorn returned nothing, and this indicates that airflow webserver will fail to start.

Comments

[gpk27]

Did you find a fix?

[Daniel]

As I said in the answer, add gunicorns path to your $PATH.

[Kyle Bridenstine]

For me guinicorn (as well as the Airflow service) is located here on AWS EC2-Instance (Amazon Linux 1): /usr/local/bin/airflow or /usr/local/bin/gunicorn so for example I say sudo -E /usr/local/bin/airflow webserver when I want to run Airflow (the -E loads the environment variables for sudo).

[Kyle Bridenstine]

FYI adding guinicorn to my path did not fix the issue for me. I can type gunicorn in on my terminal and the service is found so it's definitely on the path.

[Łukasz Koniecki]

Does not work for me as well. I found a proper solution: stackoverflow.com/questions/51352998/…

[kindjacket]

fixed for me by running export PATH=$PATH:/usr/local/bin instead

[Kunal Saini]

This solution seem strange, but worked for me too :-)