Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.
Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities significantly ease the performance of complex surgeries on DAGs. The rich user interface lets you visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.
An example demo has been provided for integrating the Dis.co platform into an Airflow installation.
The example finds and prints prime numbers inside a given range. The sample Airflow DAG file (
iqoqo_python_operator) distributes the range between multiple Dis.co workers.
A running Airflow installation (see
An active, registered Dis.co account (sign up at
Important! In your bash:
export DISCO_EMAIL=<Your DISCO Email credentials>
export DISCO_PASSWORD= <Your DISCO Password credentials>
Note: If you are running on a Mac, and you receive
ValueError: unknown locale: UTF-8 in Python when running the operator, add to your bash:
Add both Python files to your Airflow's installation DAGS folders. (
dis.co from the Web view or from the bash.