Celery

This topic guides you in integrating Dis.co with Celery. This will parallelize Celery jobs to launch on a Dis.co cloud.

Celery is a simple and flexible distributed system to process vast amounts of messages while providing operations with the tools required to maintain such a system.

Celery is a task queue with a focus on real-time processing. It also supports task scheduling.

Simple Dis.co Celery integration example

A simple example demo using the Celery platform and Dis.co has been provided.

The example runs a prime number finder in a distributed way using the Celery platform and Dis.co. The prime finder module splits a range of numbers into subranges and runs a prime finder function on them in parallel using Celery. By adding a single line of code the module runs the prime finder function on the subranges in parallel using the Dis.co platform on several machines. The demo achieves this by using Celery as a job queue manager. Later, it shows how to run it using the Dis.co platform and distribute the jobs to machines.

The demo shows the ease with which users who already use Celery to run jobs in parallel, can use the Dis.co platform to run their jobs.

Prerequisites

To integrate with Celery, you will need the following:

  • RabbitMQ installed and running

  • Celery installed and running

For assistance with the installation, see this guide.

Limitations

  • The script must be contained in a single file. Including other modules is not currently supported.

  • Including global modules will work if running in venv only or providing a docker container.