Airflow + celery or dask. For what, when?

Question

I read in the official Airflow documentation the following:

What does this mean exactly? What do the authors mean by scaling out? That is, when is it not enough to use Airflow or when would anyone use Airflow in combination with something like Celery? (same for dask)

Solution 1

In Airflow terminology an "Executor" is the component responsible for running your task. The LocalExecutor does this by spawning threads on the computer Airflow runs on and lets the thread execute the task.

Naturally your capacity is then limited by the available resources on the local machine. The CeleryExecutor distributes the load to several machines. The executor itself publishes a request to execute a task to a queue, and one of several worker nodes picks up the request and executes it. You can now scale the cluster of worker nodes to increase overall capacity.

Finally, and not ready yet, there's a KubernetesExecutor in the works (link). This will run tasks on a Kubernetes cluster. This will not only give your tasks complete isolation since they're run in containers, you can also leverage the existing capabilities in Kubernetes to for instance auto scale your cluster so that you always have an optimal amount of resources available.

Solution 2

You may enjoy reading this comparison of dask to celery/airflow task managers http://matthewrocklin.com/blog/work/2016/09/13/dask-and-celery

Since you are not asking a specific question, general reading like that should be informative, and maybe you can clarify what you are after.

-EDIT-

Some people coming to this more recently may wish to look into prefect, which is a sort of rewritten airflow with dask in mind (comes in open-source core with paid enterprise features).

Comments

[Alan]

For LocalExecutor, tasks are executed as subprocess: ...If it happens to be the LocalExecutor, tasks will be executed as subprocesses; in the case of CeleryExecutor and MesosExecutor, tasks are executed remotely...

[gogstad]

An undercommunicated feature of SO is that it's a wiki (at least in some sense). You're absolutely right, please feel free to edit the original answer.