Airflow

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities significantly ease the performance of complex surgeries on DAGs. The rich user interface lets you visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Simple Dis.co Airflow integration example

An example demo has been provided for integrating the Dis.co platform into an Airflow installation.

The example finds and prints prime numbers inside a given range. The sample Airflow DAG file (iqoqo_python_operator) distributes the range between multiple Dis.co workers.

Prerequisites

  • A running Airflow installation (see https://airflow.apache.org/installation.html)

  • An active, registered Dis.co account (sign up at https://app.dis.co/signup)

  • Important! In your bash: export DISCO_EMAIL=<Your DISCO Email credentials> export DISCO_PASSWORD= <Your DISCO Password credentials>

Note: If you are running on a Mac, and you receive ValueError: unknown locale: UTF-8 in Python when running the operator, add to your bash:

export LC_ALL=en_US.UTF-8

and

export LANG=en_US.UTF-8

Running the demo on your local machine

  • Add both Python files to your Airflow's installation DAGS folders. (~/airflow/dags/)

  • Trigger dis.co from the Web view or from the bash.