e-learning

Setting up Celery Workers for Galaxy

View material

Abstract

Celery is a distributed task queue written in Python that can spawn multiple workers and enables asynchronous task processing on multiple nodes. It supports scheduling, but focuses more on real-time operations.

About This Material

This is a Hands-on Tutorial from the GTN which is usable either for individual self-study, or as a teaching material in a classroom.

Learning Objectives

  • Have an understanding of what Celery is and how it works
  • Install Redis
  • Configure and start Celery workers
  • Install Flower to the Galaxy venv and configure it
  • Use an Ansible playbook for all of the above.
  • Monitor a Celery task using the Flower dashboard

Licence: Creative Commons Attribution 4.0 International

Keywords: Galaxy Server administration, ansible, git-gat

Competency level: • Beginner

Target audience: Galaxy Administrators

Resource type: e-learning

Version: 19

Status: Active

Prerequisites:

  • Ansible
  • Galaxy Installation with Ansible
  • Running Jobs on Remote Resources with Pulsar

Learning objectives:

  • Have an understanding of what Celery is and how it works
  • Install Redis
  • Configure and start Celery workers
  • Install Flower to the Galaxy venv and configure it
  • Use an Ansible playbook for all of the above.
  • Monitor a Celery task using the Flower dashboard

Date modified: 2025-11-24

Date published: 2023-04-16

Authors: Mira Kuntz

Contributors: Marius van den Beek, Armin Dadras, Björn Grüning, Helena Rasche, Lucille Delisle, Martin Čech, Nate Coraor, Paul De Geest, Saskia Hiltemann

Activity log