Running `airflow Scheduler` Launches 33 Scheduler Processes

When using LocalExecutor with a MySQL backend, running airflow scheduler on my Centos 6 box creates 33 scheduler processes, e.g. deploy 55362 13.5 1.8 574224 73272 ? Sl

Solution 1:

Yes this is normal. These are scheduler processes. You can control this using below parameter in airflow.cfg

# The amount of parallelism as a setting to the executor. This defines# the max number of task instances that should run simultaneously# on this airflow installationparallelism = 32

These are spawned from scheduler whose pid can be found in airflow-scheduler.pid file

so 32+1=33 processes that you are seeing.

Hope this clears out your doubt.

Cheers!

Solution 2:

As of v1.10.3, this is what I found. My settings are:

parallelism = 32max_threads = 4

There are a total of

• 1 (main) +
• 32 (executors) +
• 1 (dag_processor_manager) +
• 4 (dag processors)

= 38 processes!