In Unix-like operating systems (Linux, BSD, Mac OS) these periodic tasks are called as Cron jobs. For Windows, we call them Scheduled Tasks.

In this article, I will briefly discuss cron jobs, how the syntax works and its limitations. Next, I will jump into the celery beat scheduler and it works. Then, I’ll discuss how we can use Django (specifically Django Admin) to control the scheduled tasks easily. Finally, we will see how to use the celery flower plugin to visualize submitted tasks and execution results.

This blog assumes the reader has some working knowledge of Celery, the distributed task queue. If you want to quickly implement a task scheduler for your application, this article can be a quick start. If you are unaware of Celery, it is recommended that you read my previous blog: Distributed Computing with Python: Unleashing the Power of Celery for Scalable Applications.

Cron Jobs🔧

How to

Cron jobs are tasks scheduled to run at specified intervals. Cron is available in Unix-based operating systems like Linux and Mac OS.

Cron jobs are executed by a background process called cron daemon. It checks the cron table (or crontab) for scheduled commands and executes them according to the timing information associated. The syntax of a crontab is five time-and-date fields followed by the command to execute. For example -

* * * * * /command/to/execute

Each time fields from left to right are-

1. Minute (0-59)

2. Hour (0-23)

3. Day of the month (1-31)

4. Month (1-12)

5. Day of week (0-7, 0 and 7 both represent Sunday)

Let us see some examples of how crontab entries can be interpreted -

# At 22:00 on every day-of-week from Monday through Friday
0 22 * * 1-5 /command/to/execute

# At 00:05 in August
5 0 * 8 * /command/to/execute

# At 04:05 on Sunday
5 4 * * sun /command/to/execute

# At every 5th minute
*/5 * * * * /command/to/execute

To edit a crontab you need to run -

# To edit the current user crontab
crontab -e
# To edit the root user crontab
sudo crontab -e

Use cases

1. Backups: Database and file system backups are essential to any application to protect it from data loss. Backup jobs are scheduled during off-peak hours with cron jobs.

2. System Updates: With crontabs, we can check for system updates in intervals ensuring security and performance of the systems.

3. Health Checks: Task monitoring and health checks can be done periodically to check on server health/performance using cron jobs.

4. Log Rotation: Compression, backup and deletion of logs are essential to save server storage space. These operations can be automated using cron jobs.

5. Email Automation: We can send automated emails or alerts using cron jobs.

6. and many more…

Limitations

1. Minimum Interval: The minimum interval supported by cron is 1 minute (defined by */1 * * * * or * * * * *). Cron can not run any jobs that require more granular intervals.

2. No logging: Cron does not output any logs, so troubleshooting cron jobs can be difficult unless you set up logging with your task.

3. No error handling: Cron does not retry your failed jobs, it also does not report or retry them. If you want your errors handled, you need to go DIY.

4. Bad resource utilization: Cron can hog system resources if the job is too heavy or has some bugs, and slow down the server hindering normal operations.

5. Task Overlap: If a job takes a long time to execute, it can overlap with the same job for the next run. This can have unexpected results in program logic and also can be a resource hog.

6. No support for complex or non-periodic schedules: Cron has no built-in support for jobs like Run every first Monday of a month at 5 AM or Run every other week at Tuesday midnight. To implement these complex schedules, you need to find a workaround or add checks inside the job to do so.

7. Limited Environment: Cron runs in a minimal shell environment that does not have all the environment variables set like the user’s interactive shell. Commands that run perfectly in the user’s shell might fail due to this in a cron job.

Celery Beat🥬

In my last article, I have discussed celery in depth. To schedule a task in a celery we usually call <task>.delay() or <task>.apply_async(). However, these methods have to be run by your program logic and do not fulfil the criteria for running periodic tasks.

Celery Beat is a scheduler that enqueues tasks based on a schedule that are executed by available celery worker nodes.

Run Beat

For our basic setup, create a Python environment (my choice of environment manager is miniconda) and install the following dependencies -

requirements.txt

celery==5.4.0
redis==5.2.0

pip install -r requirements.txt

Next, run the following docker-compose file to launch the celery backend (rabbitmq) and broker (redis):

services:
  rabbitmq:
    image: rabbitmq:latest
    ports:
      - "5672:5672"
  redis:
    image: redis:latest
    ports:
      - "6379:6379"

docker compose up -d

Next, define the Celery app in app.py -

from celery import Celery

app = Celery("beat-worker", 
             backend="redis://localhost:6379/0",
             broker="amqp://guest@localhost:5672//")

Finally, run the Celery Beat Scheduler -

$ celery -A app beat -l INFO
__    -    ... __   -        _
LocalTime -> 2024-11-05 22:26:38
Configuration ->
    . broker -> amqp://guest:**@localhost:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 minutes (300s)
[2024-11-05 22:26:38,868: INFO/MainProcess] beat: Starting...

After running beat, you might notice that it does not do anything. That’s because we have not created a task and created a schedule for the beat scheduler. Let’s define a task and a schedule.

Create a new file called tasks.py in the same folder, this is where we will define our tasks.

from app import app
import subprocess

@app.task()
def log_uptime(logfile_name: str) -> None:
    subprocess.run(f"uptime >> {logfile_name}", shell=True)

We just declared our first task, which will log any system’s uptime, the number of logged-in users, and the average load of the system in a file. When run periodically, it will append the output to the given filename.

Next, we also need to include the task in the Celery configuration and define a schedule so that it can run periodically. To do this, modify app.py like this -

from celery import Celery
import random

app = Celery("beat-worker",
             backend="redis://localhost:6379/0",
             broker="amqp://guest@localhost:5672//",
             include="tasks") # include the tasks file where our tasks are
app.conf.beat_schedule = {
    'log-uptime': {
        'task': 'tasks.log_uptime', # define our task method
        'schedule': 10.0, # define our interval, 10 second
        'args': ('uptime.log',) # pass arguments to our task, if any
    }
}

Next, restart the celery beat and spin up a worker node, in separate shells -

$ celery -A app beat -l INFO

$ celery -A worker beat -l INFO

The beat schedules the task every 10 seconds and the worker node executes it. After several runs, the uptime.log looks like this:

12:26  up 9 days, 19:50, 2 users, load averages: 2.27 2.20 2.14
12:26  up 9 days, 19:50, 2 users, load averages: 2.15 2.18 2.13
12:27  up 9 days, 19:50, 2 users, load averages: 2.20 2.19 2.14
12:27  up 9 days, 19:51, 2 users, load averages: 2.03 2.15 2.13
21:32  up 11 days,  4:55, 2 users, load averages: 3.44 3.24 3.77
21:32  up 11 days,  4:56, 2 users, load averages: 3.23 3.20 3.74
21:32  up 11 days,  4:56, 2 users, load averages: 2.82 3.11 3.71
21:32  up 11 days,  4:56, 2 users, load averages: 2.62 3.06 3.68

Now that we have established the basics, let us discuss how to configure different celery schedules.

Celery Schedules

By default, Celery offers three different types of schedules -

1. Interval

2. Crontab

3. Solar

Interval

This is the simplest type of schedule available, a demo of this schedule is already shown in the above example. It takes seconds as interval input which is a float. So unlike Cron Jobs, setting up sub-minute resolution tasks is possible with this type of schedule.

Crontab

Celery beat supports crontab-style schedule expressions too. If we want to run our uptime logger every Monday at 3 AM, the crontab expression will be 0 3 * * mon. To use this as a Celery schedule -

from celery.schedules import crontab

app.conf.beat_schedule = {
    'log-uptime': {
        'task': 'tasks.log_uptime', # define our task method
        'schedule': crontab(minute='0', hour='3', day_of_month='*', month_of_year='*', day_of_week='mon'),
        'args': ('uptime.log',) # pass arguments to our task, if any
    }
}

While vanilla Celery does not have any way to parse the whole cron string, it should be very easy to just split the Cron text and feed the individual expressions into the crontab class.

cron_str = '0 3 * * mon'

def parse_crontab(cron_str: str):
    minute, hour, day_of_month, month_of_year, day_of_week = cron_str.split()
    return crontab(
        minute=minute,
        hour=hour,
        day_of_month=day_of_month,
        month_of_year=month_of_year,
        day_of_week=day_of_week
    )

Solar

Solar schedules are a unique way of scheduling tasks based on location coordinates, and key solar events like dawn, sunrise, dusk, sunset etc. These schedules are useful in applications like -

1. Smart Home and Smart Street Lights

2. Agricultural Automation

3. Renewable Energy Monitoring

4. Wildlife Tracking

5. Environmental Data Collection

My home city is Kolkata, and its co-ordinate is 22.5744° N, 88.3629° E. If I want to schedule a task that triggers at sunrise in Kolkata, there is no crontab or interval way to do this. We can use the solar schedule here like this-

from celery.schedules import solar

app.conf.beat_schedule = {
    'log-uptime': {
        'task': 'tasks.log_uptime', # define our task method
        'schedule': solar('sunrise', 22.5744, 88.3629),
        'args': ('uptime.log',) # pass arguments to our task, if any
    }
}

You can find a list of available solar events here: https://docs.celeryq.dev/en/latest/userguide/periodic-tasks.html#solar-schedules

To use solar schedules, we need to include the ephem library in our requirements.txt:

celery==5.4.0
redis==5.2.0
ephem==4.1.6

Database-backed Tasks with Django💽

Django is an open-source Python web framework that follows the Model-View-Template(MVT) architecture. It is one of the leading Python web frameworks offering security, modularity, and reusability in one package. Built-in Object Relation Mapper (ORM), authentication and an admin interface make it easy to start writing your logic while Django handles the boilerplate.

In this section, we will set up a Django project, configure Celery with it and use the django-celery-beat library to create Celery schedules with a GUI. Also, we will discuss some schedule types that are offered by the library.

Setup a Django Project

As discussing Django in depth is not the goal of this article, I will show a very basic setup of Django that serves our purpose. If you are experienced in Django, you can skip to the setup of django-celery-beat.

To start, add Django to your requirements.txt and install it -

celery==5.4.0
redis==5.2.0
ephem==4.1.6
Django==5.1.3

Create a new Django project named beat_it -

django-admin startproject beat_it

Django creates all of the necessary files in this structure -

beat_it
├── beat_it
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
└── manage.py

By default Django uses sqlite3 as database backend, which will be stored in a local file. for the first time, we need to initialize the database by performing a migration. Go inside the project directory and run -

cd beat_it
python manage.py migrate

Django Admin automates the creation of admin sites for the defined models which removes the need to develop any admin panel. All models registered to Django Admin support CRUD operations through GUI. As we are done setting up the database, we need to create a superuser to access the Django admin dashboard. Fill up the username, email and password as instructed to create a superuser.

python manage.py createsuperuser

Next, we will run the development server and access the admin panel -

python manage.py runserver

Open http://127.0.0.1:8000 in your browser, and you will see the welcome page:

To access the admin panel, open http://127.0.0.1:8000/admin and log in with your superuser credentials. You will see a page like this -

Next, create an app demoapp inside our project where we will define our tasks -

python manage.py startapp demoapp

Congratulations, you have successfully set up your Django project. Next, we will set up the django-celery-beat library.

Set up django-celery-beat

django-celery-beat is a library that uses Django ORM to save schedules and tasks in the database, adds some handy schedule types, and allows us to manage all of these things from Django Admin, making it an almost no-code solution.

To get started, include it in requirements.txt and install -

django-celery-beat==2.7.0

We will first configure Celery, but this time we will do it by defining the backend and broker in Django settings.py and later importing those settings while initializing Celery.

In beat_it/beat_it/settings.py, inside INSTALLED_APPS array add our created app demoapp, and django_celery_beat and add CELERY_BROKER_URL, CELERY_RESULT_BACKEND, and CELERY_BEAT_SCHEDULER lines at the end of the file -

INSTALLED_APPS = [
    ...,
    'demoapp',
    'django_celery_beat',
]
...
CELERY_BROKER_URL = 'amqp://guest@localhost:5672//'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers.DatabaseScheduler'

Next, create a new celery.py in beat_it/beat_it path of our project. The file is explained below -

import os
from celery import Celery

# Set the location of django settings in environment
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "beat_it.settings")

app = Celery('beat_it') # Initialize celery
# import celery config from settings.py, use namespace prefix 'CELERY'
# This is how BROKER_URL, RESULT_BACKEND, BEAT_SCHEDULER variables 
# are imported
app.config_from_object("django.conf:settings", namespace="CELERY")
# Automatically discover tasks defined in a 'tasks.py' present
# in the modules from the INSTALLED_APPS array
app.autodiscover_tasks()

django-celery-beat defines some models that are stored in the database and are possible to edit in Django Admin. We need to run migrations once more to sync the database and create the tables for those models.

python manage.py migrate django_celery_beat

The Django Admin dashboard will look like this after the migrations -

Setting up Schedules

django-celery-beat offers some useful wrappers for existing celery schedules and more. It has the following schedules supported -

1. Clocked Schedule: Useful for defining a schedule for tasks that run once at a fixed date and time. We can create one with code -

    from django_celery_beat.models import ClockedSchedule
    from datetime import datetime, timedelta
   
    # A schedule that will run once after 10 minutes from now
    time_after_10_min = datetime.now() + timedelta(minutes=10)
    ClockedSchedule.objects.create(clocked_time=time_after_10_min)
   

   This same operation can also be done with Django Admin panel. To create, visit http://127.0.0.1:8000/admin/django_celery_beat/clockedschedule/add/ and fill up the date and time.

   

2. Crontab Schedule: This schedule is very similar to vanilla Celery crontabs discussed before, with the added feature of Timezone awareness. To create a crontab schedule for expression 0 3 * * mon for Indian Standard Time (IST) zone -

    from django_celery_beat.models import CrontabSchedule
    import zoneinfo # for setting the timezone
   
    CrontabSchedule.objects.create(
        minute='0',
        hour='3',
        day_of_week='mon',
        day_of_month='*',
        month_of_year='*',
        timezone=zoneinfo.ZoneInfo('Asia/Kolkata')
    )
   

   Similarly in Django Admin UI -

   

3. Interval Schedule: This one is a very useful wrapper around the basic interval scheduler that Celery implements. While the basic Celery scheduler offers an interval setting only in seconds, this one has resolution options of days, hours, minutes, seconds, and microseconds.

   

4. Solar Schedule: Solar Schedule is also a wrapper around the Celery implementation. You can set the Latitude, Longitude and Solar Event type like sunrise, sunset, dawn, dusk etc.

   

Setting up Tasks

Now we can define some tasks in beat_it/demoapp/tasks.py. We can define our same uptime logger script here -

from celery import shared_task
import subprocess

@shared_task
def log_uptime(logfile_name: str) -> None:
    subprocess.run(f"uptime >> {logfile_name}", shell=True)

As we have set up task auto-discovery, this task will be already visible to the Celery worker when we run it.

To add this task to django-celery-beat go to Django Admin and add a Periodic Task: http://127.0.0.1:8000/admin/django_celery_beat/periodictask/add/

Add a Name, under Task (custom) write the full module path of the task, e.g. demoapp.tasks.log_uptime, keep Enabled checked and select a schedule of any type that you have created before, I chose an interval of every 5 seconds. If the task has any argument, pass it on to the Arguments or Keyword Arguments section. Leave other settings as default and click save.

Our task is now set up and ready to run!

Running Tasks

Running a Celery worker cluster does not change much with django-celery-beat. The same Celery worker concurrency models that I discussed in my previous blog apply here as well! To keep things simple, we will go with the default prefork model and run the Celery worker -

celery -A beat_it worker -l INFO

As our worker node is ready to accept tasks, we now need to launch the Celery Beat scheduler -

celery -A beat_it beat -l INFO

Beat will start enqueuing the tasks as defined in the database according to the schedule.

Cluster Monitoring with Celery Flower 🌸

Flower is an open-source web interface to monitor and manage Celery clusters. It can show real-time information on celery workers, tasks, and brokers.

Installation

Flower is available from pypi.org, run -

pip install flower==2.0.1

Run Flower

From our beat_it project directory, run -

celery -A beat_it flower -l INFO

Celery Flower will be running at http://0.0.0.0:5555.

Monitoring

Flower displays the number of active, processed, failed, succeeded, and retried events for workers on the home page. You can find detailed information about tasks, their results, errors, runtime, and much more on the Flower dashboard. I'll leave the exploration of it up to you!

It’s a Wrap!🎁

In conclusion, Celery Beat with Django offers a robust solution for managing periodic tasks in applications. While traditional cron jobs have their limitations, such as minimum intervals and lack of error handling, Celery Beat tries to address these issues with precise job schedules, logging via Celery backend, unique schedules etc.

By integrating Celery with Django, developers can leverage the Django Admin interface to easily manage and schedule tasks, reducing the boilerplate code and introducing dynamic adjustments. Additionally, using tools like Celery Flower for monitoring provides valuable insights into task execution, error tracking, and system performance.

This combination of technologies empowers developers to implement complex scheduling requirements and maintain optimal resource utilization, making it an ideal choice for modern applications.

Please find the GitHub repo with all the code here: https://github.com/f0rkb0mbZ/celery-beat-demo.git

It takes a lot of time and effort to write lengthy blogs with demos like this. If you've made it this far and appreciate my effort, please give it a 💜. You can also share this blog with anyone who might find it useful.

References 📝

1. Celery Beat Docs

2. Django-celery-beat Docs

3. Celery Flower Docs

In this blog, you will learn how to build scalable distributed Python applications capable of handling millions of tasks a minute. This is the second part of my two-part blog series on Multitasking with Python. You can read the previous part here.

First, I will explain task queues and how they work. Then, we'll explore message brokers to build a solid foundation. Once we've covered the basics, we'll dive into Celery in detail and finally create a project to transcode videos in bulk using Celery. Get ready; this is going to be a long blog!

Task Queues🏭

Before we discuss Celery's architecture, we need to establish what task queues are and how they help achieve computing in a distributed manner. In the last part, we discussed techniques for getting more work from a single multi-core, multi-thread system. However, task queues can span multiple systems, even data centres, and thus offer excellent horizontal scalability, which is only limited by hardware, cost, and more physical constraints. Some of the key concepts of task queues are listed below:

1. Task: Tasks are individual units of work (usually a function or method) placed in the queue to be processed. They are usually compute-heavy or I/O-heavy and take some time to complete.

2. Producers and Consumers: Producers are parts of an application that creates tasks and add them to the task queue. Consumers of workers are applications that pick up tasks from the task queue and execute them. Producers and consumers run independently of each other, and this level of decoupling adds to the scalable nature of task queues. There can be multiple producers and consumers in a Task Queue. Producers and consumers can be interchangeably named as publishers and subscribers, and this programming model is known as the pub/sub model.

3. Broker: Producers and consumers are connected via task queues. The physical implementation of task queues is provided by message brokers. Brokers act as a communication highway between producers and consumers. Usually, the queues implemented by brokers are First-In-First-Out by nature, but we can assign priority to tasks to control the execution order.

Some common use cases of task queues include background job processing, data pipelines, long-running asynchronous processes etc. Some examples of task queues include Celery (obviously), Sidekiq (written in Ruby, used in the backend of GitLab), RQ or Redis Queue (Redis-based job queue in Python), etc. Here is a list of the most commonly used task queues: https://taskqueues.com/#libraries.

Message Brokers and RabbitMQ

A 6-lane highway 🛣️ for pub/sub model

If Task is the unit of work for a Task Queue, the unit of communication for a message broker is Message. The message can be in JSON, XML, binary, or any other serialised format and can include events, complete functions, commands etc. A message broker is a piece of software that enables communication between producers and consumers, and is the cornerstone of this decoupled architecture where both producers and consumers can work and scale independently.

Before we discuss Celery, we should discuss RabbitMQ, the most used, fast and resource-efficient message broker for Celery.

RabbitMQ: One broker to queue them all🐰

RabbitMQ is an open-source message broker written in Erlang💡 that allows communication between different applications by routing messages. It uses widely adopted AMQP (Advanced Message Queueing Protocol) to communicate between systems. RabbitMQ also supports MQTT (Message Queuing Telemetry Transport, heavily used in IoT applications), and STOMP (Simple/Streaming Text Orientated Messaging Protocol) protocols to interact with other systems.

RabbitMQ comes with many customisations that are out of the box and introduce some features that can cater to various requirements. Some of the key features of RabbitMQ are-

1. Queues: This is a buffer where messages reside until a consumer consumes them. Queues are heavily configurable in RabbitMQ, like you can choose where to store the messages, in memory or on disk, should the messages persist a system restart etc. Multiple consumers can consume from the same queue.

2. Exchanges: An exchange enforces rules on how to route messages to queues and in turn consumers based on routing rules. The exchange types are discussed later.

3. Bindings: Binding is a connection between exchange and queues, the routing rules are defined here. The binding holds the binding key.

4. Routing Keys: Routing keys are set by the producer to route messages to the desired queue(s).

5. Acknowledgements: After processing the message, a consumer (worker) can send an acknowledgement (Ack) signal to RabbitMQ. In case of failed processing, the worker sends back a negative acknowledgement (Nack). RabbitMQ can be configured to re-queue or discard the message subject to requirement.

Based on the routing rules, exchanges can be classified into four types -

1. Direct Exchange: Direct exchange compares the routing key of the message and the binding key of the message and only allows messages to pass to queues if there is an exact match between routing key and binding key.

   

2. Fanout Exchange: Fanout exchange pass messages to all bound queues, regardless of the routing key or binding key.

   

3. Topic Exchange: Topic exchange is almost the same as Direct Exchange. But we match routing key and binding key using patterns here. If a routing key is xyz.* and binding key is xyz.tango or xyz.foxtrot then the topic exchange will match and forward the message.

   

   But if the binding key is xyz.tango.zulu it will not match xyz.*, because * matches a single word, while # matches multiple words. Let’s arrange a table to understand this better -

4. Header Exchange: Header exchange matches message headers and an exact match of header is needed to route the messages. Unlike a topic, there can be multiple headers of a message. To handle this, you can specify a special argument called x-match in the binding. Supported x-match values are -

   1. any: If any one of the headers matches, route the message.

   2. all: If all of the headers match, then only route the message.

Celery - a Distributed Task Queue🥬

As we already had an introduction to task queues, we can start with different components of Celery and see how it works! But, as we have learned enough theory already, we can start with our project and do some hands-on! We can start with installing the dependencies.

Celery dependencies

As we already established, task queues need message brokers to communicate between producers/publishers and consumers/subscribers. Celery officially supports multiple brokers like -

1. RabbitMQ

2. Redis

3. Amazon SQS

We will be using RabbitMQ for our example. To install RabbitMQ, head over to rabbitmq.com/docs/download and install it for your system of choice. I strongly suggest Ubuntu, Docker or any Linux distribution to use. I have tested my example project in Windows Subsystem for Linux(WSL) 2 - Ubuntu 24.04 distribution and MacOS Sequoia running in Macbook Air M1 (16GB).

Next, we have to install Celery, as it is a Python library we can easily install it using pip -

pip install celery

The most basic program in Celery

Now that we have installed our dependencies, we create a Python file called celery_worker.py.

from celery import Celery

app = Celery('hello', broker='amqp://guest@localhost//')

@app.task
def hello():
    print('Hello, World!')

Now that we have our starting file, I will explain it step by step -

1. We are importing the main Celery class first.

2. Next, we instantiate Celery into a variable called app. The first argument is the name of the current module.

3. For the broker keyword argument, we pass the broker URL. For RabbitMQ, the template for the broker URL looks like this - amqp://myuser:mypassword@localhost:5672/myvhost

   Where amqp:// is the main protocol supported by RabbitMQ, myuser is the username and mypassword is the username and password if credentials are set, localhost:5672 is the domain/IP and port where the RabbitMQ server is running, and finally myvhost is the virtual host of RabbitMQ which helps segregate multiple applications using the same RabbitMQ server.

   So, for our first celery program, we are connecting to a RabbitMQ broker running in localhost with the guest user, that does not require any password, and using the default vhost /.

4. Next, we create a new method called hello that prints Hello, World!. We are also annotating the method with app.task using the celery instance we created.

Summarising what we did, we made a very basic celery config and declared a celery task. So we have two of the main components of a task queue configured. Now we will start with the consumer (or worker) first. To launch a celery worker, run this command from the same directory of celery_worker.py and observe the output -

$ celery -A celery_worker worker
 -------------- celery@forkbomb-pc v5.4.0 (opalescent)
--- ***** ----- 
-- ******* ---- Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.39 2024-10-04 22:52:46
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         hello:0x7fd1cd3b1bd0
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

Though a Python module, celery is also available as an executable. In this command, -A denotes the module name (the filename without extension) and worker keyword asks celery to run a worker. Once the worker is started, it will wait for any task forever. The task is also created, but we now need a producer to enqueue the task.

Let us create another file called producer.py. We will import the Task from celery_worker.py here and enqueue the task.

from celery_worker import hello
# Enqueue the task
hello.delay()

<task_method>.delay() enqueue the task and after running producer.py, immediately you will see that the worker outputs the following -

[2024-10-04 23:02:42,342: WARNING/ForkPoolWorker-8] Hello, World!

If you can see the above output, then congratulations! You have successfully run your first Celery code.

But before we continue further doing advanced stuff, we need to discuss two key components of Celery: 1. Backend (or Result Backend), 2. Worker concurrency models.

Celery Result Backends

In our last example, we have seen the task only prints a line of text to stdout. But what if we want to enqueue a task that produces a result and returns it. Celery has provisions for defining a backend (or storage backend) that can store, retrieve, and monitor task results. The backend can also store task states like pending, success and failure along with task results. Celery backends can be broadly categorized like below -

1. Database Backends: Celery can store results in databases like PostgreSQL, MySQL, SQLite etc.

2. Cache Backends: We can store results in in-memory caches like Redis or Memcached. These backends are usually the fastest.

3. ORM Backends: ORMs or Object Relational Mappers abstract SQL queries to OOP-object or model operations. Celery supports Django ORM and SQLAlchemy as backends.

4. Message Queue Backends: RabbitMQ, though designed as a broker, can act as a result backend too. Celery writes the task result to a temporary queue and the producer can read the task result from the queue.

Considering the trade-offs, the fastest backends are cache (Redis or Memcached) but they offer limited persistence. But, if the task results are important, any DB or ORM backend will offer much better data persistence but will not be very performant under heavy load.

Celery Worker Concurrency Models⚙️

Concurrency in Celery Workers helps execute tasks in parallel. In my last blog, I discussed multiple ways to achieve parallel execution of tasks in Python. Celery uses similar concurrency models like -

1. Prefork: The main worker process of Celery launches multiple forked processes that can execute tasks. This worker model is the default, and as it is based on processes, this is recommended for CPU-bound tasks and common use cases. We do not need to specify the prefork worker using the -P flag while launching the worker as it is the default, but we can specify the number of forked processes one worker can have using the —concurrency flag, the default concurrency value for prefork worker is the number of CPU cores.

    celery -A celery_worker worker --concurrency=4
   

2. Greenlets: Greenlets are lightweight threads that offer very high concurrency for running I/O-bound tasks. To run a celery worker with greenlets, we need either eventlet (older, not recommended) or gevent (newer, maintained) library installed.

    celery -A celery_worker worker --concurrency=1000 -P eventlet
    celery -A celery_worker worker --concurrency=1000 -P gevent
   

   Note as greenlets are very lightweight, we can easily run thousands of green threads for each worker.

3. Threads: Celery uses Python threads to achieve concurrency and needs the Python concurrent.futures module to be present. Due to the GIL limitation of Python, this worker model is not recommended for CPU-bound workloads.

    celery -A celery_worker worker --concurrency=10 -P threads
   

4. Solo: The solo worker executes one task at a time sequentially in the main worker process.

    celery -A celery_worker worker -P solo
   

5. Custom: Adds support for custom worker pools for third-party implementations.

Getting Task Results with Redis Backend

Now that we have established the required backend and worker knowledge, we can start implementing a backend. We will be using Redis - an in-memory database as a backend. To install Redis, follow the official guide from here: https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/

We also need to install the redis client library so that Python can connect to Redis -

pip install redis

Next, we can change our celery_worker.py to include the Redis backend running in localhost:6379 and db 0 -

app = Celery('hello', backend='redis://localhost:6379/0', broker='amqp://guest@localhost//')

Let’s modify the task method to take a number as input and return its square -

@app.task
def square(x):
    return x * x

In the producer, we can now enqueue the task -

from celery_worker import square

result = square.delay(12)

print(result.get())
# Outputs 144

We have now executed two programs and have seen how to schedule tasks and get task results using a backend. The <task_method>.delay() call replicates the method signature and we can pass both method argument and keyword arguments to it.

There is another more advanced method called apply_async() that supports task chaining, callbacks, errbacks (code to execute if the task fails), execution time and expiry. Visit the official Celery doc to learn more about apply_async().

Video🎥 Transcoding with FastAPI, Celery and FFmpeg

From the YouTube example in the Introduction, the video transcoding happens in the background while you add a title, some description, tags etc. and can take many hours depending on your original video quality.

Video transcoding (or downscaling here) is a compute-intensive task that is heavily CPU-bound (or GPU-bound if you use GPU as an accelerator). Distributed task queues can be a perfect fit to solve this problem statement.

Ingredients for our project

We will be using the following tools for our project -

1. Celery

2. RabbitMQ as broker

3. Redis as a result backend

4. FastAPI for our web framework

5. FFmpeg - an open-source tool to record, stream, and edit audio, video and other multimedia files. Install it in your system of choice from the official page.

Our requirements.txt looks like this -

celery~=5.4.0 # Celery library
fastapi[standard]~=0.115.0 # FastAPI and its standard dependencies
ffmpeg-python~=0.2.0 # Python library for ffmpeg
redis~=5.1.0 # Python library for connecting to redis

Worker setup

Please clone my repository before proceeding. As this blog is getting longer, it will not be possible to explain every line of source code. I will explain the key syntaxes and the source code will be well commented.

celery_worker.py

from celery import Celery

app = Celery('celery-video-transcoder',
             backend='redis://localhost:6379/0', 
             broker='amqp://guest@localhost//',
             include=['helpers.video'])

We are connecting to RabbitMQ as a broker and Redis as a backend with their default settings. However, our tasks are not defined in the same file in the project. To make Celery find the tasks, we need to pass an include argument with the module where the tasks are defined. Our task resize_video_with_progress is defined in helpers/video.py.

API Routes

In the api.py, we have three main routes -

1. /: We host the home page here which takes a video file as input.

2. /upload_file/: Used for handling upload, generating the video thumbnail, and enqueueing a video downscaling task for each quality setting from 2160p to 144p.

3. /task_status/{task_id}/: Used for polling video conversion progress to show the progress in the front end.

About FFmpeg

FFmpeg doesn't come with a GUI or library by default. To use it with Python, we are using the ffmpeg-python library. FFmpeg creates filter graphs for any audio or video operation. Complex filter graphs, with many operations like trimming, merging, mixing, cropping, and adding overlays, can be difficult to manage in the command line. However, the Python library offers a simple syntax to handle these operations.

For example, to downscale a video with a scale filter named rickroll.mp4 to 480p has a syntax like this -

import ffmpeg

ffmpeg.input('rickroll.mp4').filter('scale', -1, 480).output('rickroll_480p.mp4').run()

Here the scale filter takes two arguments, the first is width and the second is height. To scale any video to 480p we set the height to 480 and set the with to -1✳️ to let ffmpeg auto-calculate it.

Also using the same scale filter and only taking out a single frame from a random timestamp of the video we can extract a thumbnail.

import random

file_path = 'rickroll.mp4'
thumbnail_path = 'rickroll_thumbnail.jpg'

ffmpeg.input(file_path, ss=random.randint(0, int(float(media_length)))).filter('scale', 1280, -1).output(
                thumbnail_path, vframes=1).overwrite_output().run()

I have also used ffmpeg.probe(<file_path>) to extract codec information from the input file which contains video dimension and length.

I added two types of filters, one for normal CPU encoding (with the scale filter) and another one that uses CUDA and nvenc encoder to accelerate the process using NVIDIA GPUs. There is a cuda flag that enables GPU encoding in our task method. Video encoding with nvenc is out of the scope of this blog.

Kowalski, status report!🐧

Celery tracks the status of a task with six predefined states: PENDING, STARTED, SUCCESS, FAILURE, RETRY, REVOKED. To implement status polling from the UI and track the video transcoding progress, I have implemented a custom task state called PROGRESS.

We can call the update_state method from inside the task, that periodically sets a state and marks the progress. I am setting the state PROGRESS and setting a dictionary with a key progress with the percentage of the task completed.

self.update_state(state='PROGRESS', meta={'progress': percent_complete})

To read the task status, I am sending the enqueued task IDs to the front end and polling the /task_status/{task_id} endpoint to get the task status.

from celery.result import AsyncResult

task = AsyncResult(task_id)
if task.state == 'PROGRESS':
    task.info.get('progress', 0)

The demo!

To run the demo on your machine you can clone the repository here: https://github.com/f0rkb0mbZ/celery-video-transcoder. I have provided a docker-compose.yml file to make the setup of Redis and Rabbitmq easier. Please pardon my poor UI and javascript skills as I am not a frontend developer. In the demo, I have used the internet’s favourite video in 1080p to downscale it to 720p, 480p, 360p, 240p and 144p, also it generates a thumbnail as a bonus!

Ending note✍🏼

In conclusion, distributed computing with Python, particularly using Celery, offers a robust solution for building scalable applications capable of handling millions of tasks efficiently. We have learned that -

1. By leveraging task queues and message brokers like RabbitMQ, developers can achieve significant horizontal scalability and decoupling of application components.

2. Celery's integration with various backends and concurrency models provides flexibility in managing task results and optimizing performance for different workloads.

3. The practical example of video transcoding demonstrates Celery's capability to handle compute-intensive tasks, showcasing its potential in real-world applications.

As you explore Celery further, you'll find it a valuable tool in your distributed computing toolkit, enabling you to build resilient and scalable systems.

It took a lot of time and effort for me to gather all this information and write this blog. If you've made it this far and appreciate my effort, please give it a 💚. Share this blog with anyone who might find it useful. If you have any questions or need help understanding any concept, feel free to ask in the comments section! Thank you for reading!

References📋

1. The Official Celery Documentation

2. The Official FastAPI Documentation

3. Blogs of Bjoern Stiel in celery.school

It was a network programming challenge, similar to building a high-performance, low-latency server. To achieve this with Python, I had to experiment with various multitasking technologies such as concurrency, parallelism, threads, processes, asynchronous programming, and greenlets. This is the first part of a two-part blog and this one covers the most common techniques that I learned for achieving multitasking in Python. I will discuss Celery, a distributed task queue in detail in the next part of this blog.

Basic Multitasking Foundations

We often use words like 'multitasking,' 'concurrency,' 'parallelism,' 'thread,' and 'process' interchangeably. While they all aim to achieve similar goals, their internal workings can be quite different. Let's review them one by one -

1. Multitasking: Performing multiple tasks at the same time, or the execution by a computer of more than one program or task simultaneously.

   

2. CPU Bound and I/O Bound Tasks: Tasks that involve computation by CPU are called CPU-bound tasks. Examples include arithmetic operations, image/video processing etc.

   Tasks that involve input/output operations, where waiting for data is needed, are called I/O-bound tasks. Examples include reading/writing to disk, network requests, database queries etc.

3. Concurrency: It can be defined as a computer's ability to execute different tasks at the same time✳️ by switching between multiple tasks without necessarily completing them. This behaviour is called context switching and we call the tasks as threads. Threads can be implemented in Python using the threading library and ThreadPoolExecutor from concurrent.futures. Threads are created within the same process, sharing code, data, and resources. They are lightweight.

    import threading
   
    def task():
        # Your task here
   
    thread = threading.Thread(target=task)
    thread.start()
   

   ✳

   Modern CPUs can run at frequencies 2-5 GHz. They can switch between threads so fast, that a single CPU core can handle a lot of simultaneous tasks that feel like multitasking.

   

4. Parallelism: Parallel execution of tasks is only possible when there are more than one CPU core is present. One CPU core can handle a task while the other core handles another. In Python, we can implement parallelism with the help of multiprocessing library and ProcessPoolExecutor from concurrent.futures.

    import multiprocessing
   
    def task():
        # Your task here
   
    process = multiprocessing.Process(target=task)
    process.start()
   

   💡

   The first desktop-class dual-core processor was the Intel Pentium D which was quickly followed by AMD with their Athlon 64 X2 in the year 2005!

   

5. Concurrent and Parallel: When a CPU schedules multiple tasks over multiple cores, it benefits from both concurrency and parallelism. While some programming languages like Java and C++ can multiplex threads to run on different CPU cores and achieve concurrent parallel multitasking, this is not possible in Python due to Global Interpreter Lock (GIL). More on this later.

   

6. AsyncIO: Introduced in Python 3.4, asyncio is a built-in library for asynchronous programming that runs an event loop that is capable of running multiple asynchronous tasks and coroutines at the same time. You write your functions as coroutines that start with async def and can be paused and resumed. When your function encounters any I/O operation it yields back control to the event loop and the event loop continues executing other tasks. When the I/O operation completes, it triggers a callback and the event loop picks up the task again.

   The I/O operation is executed by the operating system. The event loop uses low-level I/O polling mechanisms like epoll, select, kqueue etc. based on the OS, to manage task completion notifications.

    import asyncio
   
    async def task():
        # Your async task here
   
    asyncio.run(task())
   

   Though asyncio is extremely good for I/O bound tasks (like network programming), as the main event loop is single-threaded, this mechanism is not suited for CPU-bound or compute-heavy tasks.

   

Problem with Multithreading in Python

The biggest challenge with multitasking in Python is Global Interpreter Lock (GIL). As we saw earlier, threads can run in a truly parallel manner in multi-core systems. In languages like C++ and Java, threads with CPU-bound tasks can run on different CPU cores simultaneously.

GIL is a mutex that prevents native threads from executing Python bytecode simultaneously in CPython (the most common Python language implementation). GIL ensures that even in multi-core systems, only one thread runs in the interpreter.

Due to the GIL, threads in Python are not effective for CPU-bound tasks and can sometimes degrade performance because of context-switching overhead. The easiest way to bypass the GIL is to use multiprocessing, but this comes with overheads in startup time and memory. For I/O-bound tasks (like network requests and disk I/O), threads can still be effective because the GIL is released during I/O operations.

Other Solutions

There are more ways that you can implement multitasking in Python. Some of them are Greenlets (with third-party libraries like eventlet and gevent), Subinterpreters (a new way of multitasking with Python 3.12+), Celery (scalable, distributed task queue) etc.

Before Asyncio

We already talked about how asyncio with its event loop model improves the execution of I/O-bound tasks in Python. Some leading Python web frameworks like FastAPI and AioHTTP are made using asyncio and can handle thousands of connections in a single thread.

But, before Python 3.4 there was no asyncio (and no async def and await syntax). To build web servers, popular models like pre-fork and threading were used -

1. Pre-fork: The web server starts with a main process, and forks multiple child processes. Each child process then waits for an incoming connection. Apache web server with mpm_prefork module is an example of this.

2. Threading: The main server process starts and spawns multiple worker threads. These worker threads handle the incoming connections. Once the connection is responded to, the worker thread becomes available again. Apache web server with mpm_worker module is an example of this.

Greenlets, Eventlet and Gevent

Eventlet is a Python library that allows you to write concurrent code using a blocking style of programming (no async/await). It was created when there was no implementation of asyncio in Python - about 18 years ago.

Eventlet is not as updated and maintained nowadays as Python’s asyncio grew and its use is discouraged. Gevent is a newer library that implements green threads for massively concurrent workloads like web servers. Gevent can run thousands of green threads under a single OS-based thread.

The working mechanism of Eventlet or Gevent is something like this:

1. Green Threads: It uses something called green threads or greenlets which are lightweight threads managed by the library itself (not by the operating system).

2. Cooperative multitasking: Green threads voluntarily yield control to other green threads which is called as cooperative multitasking.

3. Event loop: Just like asyncio, Eventlet or Gevent has an event loop that manages the execution of the green threads.

4. Monkey(🐒) Patching: Eventlet/Gevent can make changes in the Python standard library to make blocking operations non-blocking. This is called monkey patching. For example, network operations and file I/O will be able to yield control to other green threads while waiting for network response or file read operations. There are some strict rules for monkey patching to work, like -

   1. Monkey patching should be done at the beginning of the code but after the imports so that Eventlet / Gevent can patch the blocking methods of those imports. Any imports after patching can cause issues because they might contain unpatched methods.

       # Any other imports
       import eventlet
       eventlet.monkey_patch()
      
       # Any other imports
       from gevent import monkey
       monkey.patch_all()
      

   2. Always perform monkey patching on the main thread so that all greenlets can use the patched functions.

   3. Monkey Patching might not work for any third-party library other than the standard library. Check if the third-party library is supported for patching by Eventlet/Gevent first.

There are small examples given below for both Eventlet and Gevent. You might notice no monkey patching done in the examples because the code does not use any blocking function from the standard library.

# eventlet example
import eventlet

# Define a task
def task(name):
    print(f"{name} started")
    eventlet.sleep(1)
    print(f"{name} finished")

pool = eventlet.GreenPool()
pool.spawn(task, "Task 1")
pool.spawn(task, "Task 2")
pool.waitall()

# gevent example
import gevent

# Define a task
def task(name):
    print(f"{name} started")
    gevent.sleep(1)
    print(f"{name} finished")

gevent.joinall([
    gevent.spawn(task, "Task 1"),
    gevent.spawn(task, "Task 2")
])

Subinterpreters

By now, we know that the Python interpreter is single-threaded. However, there is another way to run code in parallel within the main interpreter, which is by using isolated execution units. This is called sub-interpreters and has been a part of Python for a long time. Global Interpreter Lock was never implemented at the Interpreter level until recently when Python 3.12 was released. From Python 3.12 each (sub)-interpreter has its own GIL and can run under the main Python process parallel.

From this point, I should warn the readers that the high-level subinterpreters API available in Python 3.12 is experimental and should not be used in production workloads.

To run a subinterpreter with Python 3.12, we can import a hidden module -

import _xxsubinterpreters as interpreters

# Create a new sub interpreter and store its id
interp1 = interpreters.create()

# Run python code in the interpreter
interpreters.run_string(interp1, '''print('Hello, World!')''')

In the later versions of Python, we will be able to access this API with import interpreters. Compared to multiprocessing, subinterpreters have less overhead but are heavier than threads/coroutines/greenlets. Though experimental, subinterpreters can make way for true parallelism in Python in future releases.

Celery

Celery is an asynchronous distributed task queue that can truly run Python code in scale (or in parallel, or concurrent parallel). By nature, task queues, process tasks in a distributed way (from threads to different machines). Like other task queues, Celery communicates with messages between clients (who schedule a task) and workers (who execute a task) with the help of a message broker like Redis, RabbitMQ, etc. With multiple brokers and workers, Celery can support High Availability and Horizontal Scaling.

The next part of this blog will discuss Celery in detail.

Wrapping up!

In this article, we explored various multitasking techniques in Python, including concurrency, parallelism, threading, multiprocessing, and asynchronous programming with asyncio. We also discussed the challenges posed by the Global Interpreter Lock (GIL) and alternative solutions like greenlets, subinterpreters, and libraries such as Eventlet and Gevent. Understanding these concepts is crucial for building efficient applications, especially for network programming and I/O-bound tasks. To keep the blog concise, I focused on the brief theoretical aspects, as each multitasking paradigm could fill an entire book. You can follow the given references to learn more about each of them.

In the next part, we will dive deeper into Celery, a powerful distributed task queue, to achieve scalable and parallel task execution in Python. Stay tuned!

Also, please drop a 💙 if you made it this far and appreciate my effort and share this blog with anyone who might find it helpful. Thank you for reading!

References

1. Concurrency Vs Parallelism! by ByteByteGo

2. Async IO in Python: A Complete Walkthrough by RealPython

3. gevent For the Working Python Developer by the Gevent Community

4. Python Gevent in practice: common pitfalls to keep in mind

5. Running Python Parallel Applications with Sub Interpreters by Anthony Shaw

6. Python 3.12 Subinterpreters: A New Era of Concurrency by Thinh Dang

7. excalidraw.com is where I made the diagrams for this blog

8. Easter Egg: Michaelsoft Binbows (knowyourmeme, youtube)

Reverse Proxy

A reverse proxy is a server that sits between a client and one or more web servers while intercepting connections from client to server. It can manipulate request data, load balance requests, protect from cyber attacks and many more!

Nginx Example

Let's take an example of a Nginx config file. The example shows a reverse proxy that listens on the default HTTP port 80, forwards requests to an upstream http://localhost:8080 and modifies some request headers along the way.

server {
    listen 80;

    server_name example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

If you already know Nginx, this is the most basic reverse proxy configuration you can use. We define a few things here:

1. server Block: This defines a virtual server. Multiple server blocks can route connections based on IP, port and server_name.

2. listen: Port to listen to.

3. server_name: Nginx matches the Host header of the connection to this and routes the request.

4. location block: A location block is used to reach different resources in a server block.

5. proxy_pass: This delegate requests or connections to one or more upstream servers.

6. proxy_set_header: This directive is used to edit/add/remove headers to the upstream requests.

Generating SSL certificate

There are many Certificate Authority (CA) that issue SSL certificates. CAs are trusted issuers of digital certificates. If you ask how we determine which CAs are trusted, for most of us browsing the internet, It is the browser or OS vendor like Google, Microsoft, Mozilla, Apple etc. However, there is a governing forum called CA/Browser Forum that decides additional rules and regulations for CAs.

For our example, we can use a tool called Certbot, a tool from the Electronic Frontier Foundation that generates SSL certificates from a non-profit certificate authority called Let's Encrypt.

You can go to certbot.eff.org and install certbot for your system by following their guide. Once certbot is installed, just run -

$ sudo certbot —-nginx

This will automatically verify your domain (you have to set your domain name to server_name), request a certificate from Let's Encrypt CA and modify your Nginx configuration to enable HTTPS for your server block.

Caddy Server

Just like Nginx, Caddy is a cross-platform, open-source web server. While Nginx is very fast and a big name in the industry, Caddy has some awesome features like -

1. Automatic HTTPS: Caddy, by default obtains/renews SSL/TLS certificates from Let's Encrypt / ZeroSSL. Even for localhost or, internal addresses, Caddy can provision short-lived, auto-renewing certificates using a locally trusted certificate authority.

2. Simple Configuration Syntax: Caddy's configuration file - Caddyfile is an extremely simple and human-readable format compared to the Nginx configuration syntax.

3. Built-in static file server: Serve compressed files or compress on the go to save bandwidth. Serve static sites from a database, local file system or remote cloud storage. Also, serving a directory (that does not have an index file) looks like this:

   

   To start a local file server, run caddy file-server in a directory that you want to serve.

4. REST API for configuration: Caddy has a REST API that can be used to make dynamic configuration changes.

5. Extensibility: Caddy has a huge library of pre built modules that can be used to supercharge it's capabilities.

6. and many more...

Caddy Example

This blog is not a getting-started or how-to guide for Caddy, so I will stick to the very basic Nginx example and try to recreate that in Caddy, step by step.

1. Install Caddy: Install caddy for your platform by following the official guide.

2. Write Caddyfile: The Caddy configuration file is written in a file called Caddyfile, no extension. If we translate the same Nginx config above to Caddy in the simplest form, it will look like this -

    :80
   
    reverse_proxy http://localhost:8080
   

   💡

   Caddy also has support for JSON configuration. For complex setups JSON is preferred.

3. Run Caddy: From the same directory of the Caddyfile use caddy run to start caddy.

4. Browse the page: Open http://127.0.0.1 in the browser. Caddy will now be proxying the traffic to http://localhost:8080.

Automatic HTTPS

To enable automatic HTTPS, we need to tweak the Caddyfile a little, by including a domain name -

example.com

reverse_proxy http://localhost:8080

If you run the above Caddyfile, caddy will try to generate a certificate for the domain example.com, and it will fail, as you do not own example.com.

If you change the domain name from example.com to localhost and then run Caddy, caddy will generate a new self-signed certificate for localhost and add it to the local trust store. So in the browser, it looks like this -

If you run this on a Cloud VPS that has a domain name pointed to it, with the domain in Caddyfile, caddy will provision a certificate from Let's Encrypt.

Request Header Manipulation

To do request header manipulation just like the Nginx config, there is a directive in caddy called header_up that adds/edits upstream request headers. The complete configuration after header manipulation for localhost looks like this -

localhost

reverse_proxy http://localhost:8080 {
    header_up X-Forwarded-For {http.request.header.X-Forwarded-For}
    header_up X-Real-IP {http.request.remote.host}
    header_up Host {http.request.host}
    header_up X-Forwarded-Proto {http.request.scheme}
}

And for a domain that you own -

snehangshu.dev

reverse_proxy http://localhost:8080 {
    header_up X-Forwarded-For {http.request.header.X-Forwarded-For}
    header_up X-Real-IP {http.request.remote.host}
    header_up Host {http.request.host}
    header_up X-Forwarded-Proto {http.request.scheme}
}

Production Deployment

It is recommended to run caddy in production with their official systemd Unit file(s).

In the official unit file, the default location of the Caddyfile is /etc/caddy/Caddyfile. So you need to insert your change to that file or edit the official unit file with your Caddyfile location to enable your configuration. Once the configuration is done, start/restart Caddy using systemctl -

$ sudo systemctl restart caddy.service

Conclusion

Caddy offers a streamlined and efficient way to set up a reverse proxy with HTTPS, making it an excellent alternative to Nginx for many use cases. Its automatic HTTPS feature, simple configuration syntax, and built-in static file server significantly reduce the complexity and effort required to manage web servers.

While Nginx remains a robust and industry-standard tool, Caddy's modern features and ease of use make it a compelling choice for developers looking for a hassle-free setup. Ultimately, the choice between Nginx and Caddy depends on your specific needs and preferences, but Caddy certainly stands out for its user-friendly approach and powerful capabilities.

Thank you for reading, and if you found this post helpful, please give it a 👍, ask any questions in the comments, and share it with others who might benefit.

The Cloud is just someone else's computer

- Anonymous

We are all familiar with the above sentence. But it does not tell the whole story about modern-day cloud infrastructure and its complexities that we rely on every day. When cloud computing gained traction, it only offered raw compute (e.g. Servers with CPU, RAM and Storage). We call this kind of cloud offering Infrastructure-as-a-Service (IaaS).

Built on top of IaaS, Platform-as-a-Service (PaaS) like managed databases, container hosting platforms emerged. Some popular examples are the GCP App Engine, AWS Elastic Beanstalk, Azure App Service etc.

Finally, we get to the Software-as-a-Service (SaaS) paradigm which is the most used format of Cloud Computing we use daily. Some examples are given below:

As we can see, we are surrounded by Cloud-based SaaS software and it is challenging to imagine modern life without them. In this blog, we will see how we can replace the impact of some of these services or improve how we use software with a low-cost home cloud solution. But first, we need to talk about network-attached storage or NAS devices.

What is a NAS?

Network Attached Storage or NAS is a device on your network that acts like a big shared storage device. NAS devices are very similar to desktop PCs, they contain a CPU, RAM, Storage and very good networking hardware, but they do not have a display attached - they are accessed via Web UI, SSH, NFS or SMB. Commercial dedicated NAS devices have 4+ hard drive bays and network support ranging from 2.5 GBps to 10 GBps. These devices usually have hardware RAID chips that offer data sharding, redundancy and failure recovery. The main idea of a NAS is mostly to offload your storage needs from personal devices to a dedicated and always online device. You can store documents, music, photos, 4K movies, local file backups, etc. in a NAS.

How can you assemble a NAS device?

As I mentioned before, NAS devices are desktop computers with some special power. If you assemble a similar desktop it can act as a NAS with some special OS.

Some of these open-source OSes are -

1. TrueNAS

2. OpenMediaVault

3. Rockstor

Apart from file storage, these OSes offer some PaaS and IaaS services. They can host container platforms and VMs with various levels of difficulty which depends on that particular OS.

Is assembling or purchasing a NAS worth it?

While the OSes that I mention abstract away a lot of underlying complexities, purchasing and maintaining a NAS device require low to high domain expertise depending on the application. If not configured correctly, it can leave your data vulnerable to cyber attacks (if your NAS is connected to the internet), or data loss due to drive failure can be very costly for you or your business.

Also, NAS devices do not make much sense in those countries (or regions) where cloud storage and internet costs are low. I live in India 🇮🇳, and here Internet is one of the cheapest in the world. So people prefer paying for Google One, iCloud or DropBox subscriptions rather than buying a NAS.

In my opinion, if you live in a region where the internet is expensive or slow, or you have a specific business need, then a NAS device makes sense for you. If you are a developer or a tinkerer, build your own NAS, otherwise, invest in a readymade one from any reputed brand like Synology or Asustor.

What if NAS devices offer some SaaS?

As discussed earlier, the Internet and our daily lives run on various SaaS products. What if there was a simple NAS device that we could build, which is way cheaper than any commercial NAS and does not need much expertise to set up, also can run some SaaS services along with basic file storage? Sounds too good to be true right?

I also did not believe it, until I used Umbrel. Some offering of Umbrel includes Nextcloud (self-hosted Google Drive alternative), Plex (Cross-platform media server), Home Assistant (IoT hub for complete home automation), Pi-hole (Network-wide ad blocker), Photoprism (Self-hosted photo and video library), LlamaGPT (Self-hosted GPT powered by Llama 2) and Portainer (Run thousands of apps as a docker container or make your app with a Dockerfile). Also, did I mention that the entire Umbrel platform is open-source, the OS and the apps, everything?

With Umbrel, NAS or Home Cloud suddenly made sense to me! While I made some introduction to Umbrel, let me quickly discuss how I set Umbrel up with a Raspberry Pi and an SSD.

My Humble Umbrel Setup!

Since I used parts from past projects, my Umbrel setup cost is different from the current price. I have included a breakdown of the key hardware and their estimated April 2024 price in Indian Rupees (₹) to help you build your own. If you're located outside of India, you can find similar components on your local Amazon or electronics store.

Some insights about the hardware

1. USB 3.0 Speeds: Raspberry PI 4 Model B does not support the full 5 Gbps (625 MBps) speed on its USB 3.0 ports because the USB controller chip is connected to the processor via a single PCIe 2.0 x1 lane which supports speeds of 4 Gbps (500 MBps). Any good branded NVMe PCIe 3.0 SSD(Samsung, WD, Crucial etc.) supports speeds up to 3000+ MBps. So if you are trying to build this, do not spend on a fast NVMe SSD if you only intend to use that SSD in a Raspberry Pi project.

   SATA SSDs support speeds up to 6 Gbps. You can grab a 2.5-inch SATA SSD / m.2 SATA SSD and one 2.5-inch SATA SSD enclosure / m.2 SATA SSD enclosure which is cheaper. TL;DR SATA SSDs and their enclosures cost less than their NVMe cousins.

2. USB Cables: If your SSD enclosure comes with a USB C to USB A cable that supports 5 Gbps or 10 Gbps speeds, you can skip this point. If your SSD enclosure comes with only a USB C to USB C cable with speeds of 5 Gbps or 10 Gbps, you will need one USB C to USB A adapter that supports at least 5 Gbps speed like this. If you do not like the adapter solution, I suggest you find a good USB C to USB A 5 Gbps cable.

3. Memory variant of RPi: Raspberry Pi 4 Model B is available in 4 GB and 8 GB RAM variants. While UmbrelOS is supported for both, it is always better to get 8 GB of RAM for faster performance.

4. Power Supply: While the official Raspberry Pi power supply is recommended, you can use any USB-PD compatible mobile charger rated 2.5 A and above.

Installing UmbrelOS

There is a comprehensive installation guide available on the site of UmbrelOS. In short, the steps are following -

1. Download the UmbrelOS image for Raspberry Pi 4.

2. Download balenaEtcher to flash the image to the microSD card.

3. Connect the microSD card to Pi after flashing.

4. Connect the SSD to any USB 3.0 port (blue port) of your Pi. If you have any data on the SSD please take a backup because it will be erased by UmbrelOS during the setup process.

5. Connect one end of the Ethernet cable to your home router and another end to the Raspberry Pi. The router must have DHCP enabled for the LAN ports.

6. Connect the power supply and switch it on.

7. UmbrelOS will complete the setup within 5 minutes and will be available at http://umbrel.local/ or http://umbrel on Windows.

8. Once you enter your name and set your password, you will experience something like this -

Umbrel Ecosystem

The App Store

The app store in Umbrel OS has plenty of apps that cover a lot of ground when it comes to self-hosting applications. The categories range from productivity, media, networking, IoT, developer tools and AI. Here's a sneak peek of the app store -

You can browse the app store as a web page here: https://apps.umbrel.com/

My favourite apps

Of all of these available apps, some apps I liked and found very useful. I am listing them below -

1. Pi-hole: Pi-hole is a network-wide ad blocker and does not require any browser extensions. It can achieve this by blocking DNS queries for the domains where the ads are served. Just install Pi-hole on your Umbrel and configure your router to use the IP of Umbrel as the primary DNS server. Ads will now get blocked by Pi-hole and you get a squeaky clean web browsing experience.

2. Gitea: It is a self-hosted version control service like GitHub, BitBucket or GitLab. Written in Golang, it is very minimal and can easily run on a Raspberry Pi.

3. qBittorrent: One of the best open-source torrent clients, totally ad-free. Offload your torrent downloads to Umbrel and transfer them to your device when it is done. This app also comes with an alternative UI of qBitorrent called VueTorrent which is modern, responsive and sleek.

4. Cloudflare Tunnel: Connect to your Umbrel device from anywhere using Cloudflare Tunnel. This app runs a very lightweight daemon called cloudflared to maintain a tunnel, you can easily access your Umbrel apps from the Internet.

5. Home Assistant: Self-hosted home automation system compatible with thousands of IoT devices and services like Amazon Alexa, Google Assistant, Samsung SmartThings or Apple HomeKit. Privacy first and local control makes it a safer home automation choice than cloud services.

6. Back that Mac Up: An extremely easy-to-use SMB or Samba server is made available to your local network by this app. Then it can be mounted and used as a time machine backup location and data can be restored when needed.

7. File Browser: A no-nonsense file browser for all your downloads and creations on Umbrel. You can preview basic files, and images, and download them to your machine with this app.

8. Immich: Self-hosted Google Photos alternative to store, view, edit, and organize your photos and videos. Comes with a companion mobile app for Android and iOS.

9. Snapdrop: An alternative to airdrop for your local network. Open Snapdrop on both devices that you want to transfer in between and transfer files just like Airdrop.

10. NextCloud: The most popular self-hosted file storage service. It comes with a beautiful web interface, file-sharing control and many more features. Nextcloud has a cross-platform (Android and iOS) companion app for easy file sync and backup.

11. Plex and Jellyfin: These are the best self-hosted media servers found on the Internet. Plex is a user-friendly media server with a wider range of features, client app support, and optional paid extras while Jellyfin is a free, open-source option that prioritizes privacy and gives you more control over your media.

12. MeTube: Download videos from YouTube and watch them later. As simple as that.

13. Chatpad AI: Alternative UI for ChatGPT with a lot of fine-grained control.

14. Portainer: This very powerful container hosting platform is intended for power users who are looking to extend Umbrel's capability. Run docker container, compose stacks and manage, monitor, and modify containers, networks and volumes with this. Using Portainer you can run any web app that can be containerized, on your Umbrel.

15. LlamaGPT (Special Mention): Self-hosted LLM on a Raspberry Pi with a ChatGPT-like UI. It takes 5 GB of RAM to run, so it is not suitable to run on the 4 GB RAM variant of the Pi. While great as a proof-of-concept, the generation is very slow (1 word / second) on the Pi 4 8 GB variant.

16. Crypto / Blockchain / Web3 Apps (Special Mention): I have avoided listing blockchain-driven apps because I have very limited knowledge of these technologies and have not tested any apps that are available on the Umbrel App Store. If you have used any of these apps, please leave a comment!

All Umbrel apps are containers

Umbrel is by design, container driven. It means every Umbrel app is a container. As the entire Umbrel software ecosystem is open-source, we can look into one of the most basic Umbrel apps - File Browser.

As we can see, it is not rocket science to run Umbrel apps. If you want to build and run your app on Umbrel, you can read the official documentation for Umbrel App Framework.

Umbrel Home

Umbrel sells its own hardware called 'Umbrel Home' with Umbrel OS preinstalled which has some pretty impressive specifications -

1. Intel N100 Quad-core CPU @ 3.4 Ghz

2. 16 GB DDR5 RAM

3. 2 TB SSD

4. Gigabit Ethernet

5. 3x USB 3.0 ports

You can get this one if you are interested in a clean and easy-to-use device than my hacky Raspberry Pi setup!

No Hardware Install

If you do not have any hardware, you can run an older version(💡) of Umbrel on a virtual machine to try it out. Install and run a Debian 12 (Bookworm) or Ubuntu 22.04 (Jammy Jellyfish) in a virtual machine on VMWare Workstation, VirtualBox or any cloud platform (kind of beats the purpose of running Umbrel, but it's okay to test it out!).

After the OS is installed, run this command in the terminal and follow the instructions:

curl -L https://umbrel.sh | bash

Limitations

1. While very promising, as a new platform, it is not entirely free of bugs, as I have encountered a few while using it.

2. It does not yet have support for any data redundancy technology like RAID. If you are storing any important data on UmbrelOS, please be careful and keep a backup!

Wrapping it up

I never had so much fun exploring an OS before (saying as a former Linux distrohopper)! Umbrel brings along a very new and unique way of setting up a home cloud, and with it, we can somewhat beat internet censorship, and tracking, block ads, and offload some data and tasks from our computing devices! I would encourage all of you who own a Raspberry Pi to try Umbrel, tinker with it and share the experience.

If you are still here, thank you for reading! If you like this work of mine, give a like and please let me know your feedback below in the comment section! Also, share it with someone who might learn something from my work!

Shutting down....🤖

Why do app deployments fail?

Before understanding why apps fail, we must understand what kind of apps we are discussing. By far the most popular kind of apps are web apps that run our beloved internet. Usually, the web apps need to be served by a web server. Here are some reasons why web servers might fail:

1. Resource overload, like server dying due to too much traffic, out-of-memory etc.

2. Cyberattacks, like DDoS (distributed denial-of-service) or other vulnerabilities.

3. Bugs, which can happen to the best of software.

4. Server component failures, power outages or any natural causes that can impact a data centre where the server is hosted.

Dead man what?

Dead Man's Switch, existed long before computers were born. Quoting Wikipedia, here goes the definition:

A dead man's switch is a switch that is designed to be activated or deactivated if the human operator becomes incapacitated, such as through death, loss of consciousness, or being bodily removed from control. Originally applied to switches on a vehicle or machine, it has since come to be used to describe other intangible uses, as in computer software.

Focusing on the last line of the definition, a software use of Dead Man's Switch should work like this in our server failure scenario:

Now, how to detect if the server has failed or not? There are two ways to do that -

1. Heartbeat: Our server sends a signal (say an HTTP request) to a monitoring service in periodic intervals.

2. Polling: A monitoring service sends a request to our server to check if it's alive in periodic intervals.

In this blog, we are going ahead with Heartbeat. So, when there is no heartbeat, we consider the server dead and flip the switch to do something about the incident.

Our demo app

So here's the plan, we will make a very simple Flask app and implement a library called apscheduler to periodically ping our monitoring service. We will use another library called requests to send a GET request to the monitoring service. Our requirements.txt for this project looks like:

Flask==3.0.2
APScheduler==3.10.4
requests==2.31.0

Our goal here is not to build a flask app, so I just made an app that gets a random cat image (😸) from an API and renders it.

import requests
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.interval import IntervalTrigger
from flask import Flask, render_template_string

app = Flask(__name__)
scheduler = BackgroundScheduler()


def send_heartbeat():
    print('The cat is alive 😸, heartbeat sent!')
    requests.get('<heartbeat-monitoring-service-url>')


scheduler.add_job(
    func=send_heartbeat,
    trigger=IntervalTrigger(minutes=1)
)

scheduler.start()


@app.route('/')
def hello_world():  # put application's code here
    res = requests.get('https://api.thecatapi.com/v1/images/search')
    url = res.json()[0]['url']
    width = res.json()[0]['width']
    height = res.json()[0]['height']
    return render_template_string(f'<img src="{url}" width="{width}" height="{height}"></img>')


if __name__ == '__main__':
    app.run(debug=True)

In the above code, BackgroundScheduler class is used to run a separate thread in our flask app, which pings our monitoring service. To set the schedule of the heartbeat request IntervalScheduler is used. I have used 1-minute intervals for practical reasons. You can change the interval ranging from seconds to weeks as per your requirement.

Let's run our app:

$ python app.py

Finally, our app looks like this when run locally -

Our monitoring service: healthchecks.io

healthchecks.io is a great open-source tool for monitoring cron/background jobs and has a lot of integrations built in to notify us via email, webhook, SMS, phone calls, discord, slack, telegram, WhatsApp, teams and many more ways. They have a fully managed service that charges around 5$ / month. But as it is open-source, we can easily self-host it for almost free (like free-tier VMs in Google Cloud). I will be showing how to send alerts if our app is down via emails, calls, and SMS.

Before we get started with configuring healthchecks.io, we need to configure another service that will help us route our emails, calls, SMSes and WhatsApp messages.

Sendgrid and Twilio

We will be using SendGrid for sending our emails and Twilio for calling, sending SMS and WhatsApp messages.

Sendgrid

Sendgrid is a very popular email delivery platform that helps businesses send both marketing and transactional emails with features like email templates, tracking and analytics. We would not need all of these features, just mail-sending will do!

Go to https://sendgrid.com/en-us/solutions/email-api and sign up, after finishing all the steps, you'll be looking at a page like this:

Click Create sender identity button, you will be presented with a form like this:

As this is only a demo, I have filled in only the necessary details. You can use your own gmail address as from email address and reply-to address. After you click create, a verification mail will be sent to your gmail inbox, follow the instructions to verify your email.

Once you are done, go back to https://app.sendgrid.com/ and click Email API -> Integration Guide. Choose the option SMTP Relay.

Now, give a name to your API key and create one. Once the API key is created, save all of the configuration options somewhere safe. We will use these later.

Twilio

Twilio is a cloud communications platform that equips developers with the tools to build communication features directly into their applications. These features can include things like - SMS, calls, chat and video conferencing applications.

Twilio provides the building blocks for developers to create a variety of communication experiences within their software. They accomplish this through a set of Application Programming Interfaces (APIs) that can be integrated into the developer's code. We will be using these powerful APIs of Twilio from our healthchecks.io deployment. But first, we need to sign up for a Twilio account.

Go to https://www.twilio.com/try-twilio and sign up with your email, you can also sign up faster using your Google account. After this step, Twilio will ask you to verify your phone number (this verification is mandatory). You can either verify via an SMS or a voice call.

After the verification is complete, you will get a recovery code, please save it somewhere safe!

Next, create your account, give it a name and click verify to start your trial.

Once the account creation and verification are completed, you will be redirected to the Twilio console and there you will be asked to get a Twilio phone number. This step is mandatory for sending SMS and dial numbers. Click get a phone number and Twilio will assign a random US phone number to you from their pool of numbers.

Once you get a number, scroll down the console homepage and you will see the Account Info section with Account SID, Auth Token and My Twilio phone number.

This is the information that we need to make calls and send SMS from healthchecks.io.

One last caveat, as this Twilio account is a trial account with no payment methods, we get credits of $ 15.50, but we can only make calls and send messages to the numbers that we verify. To verify a number, navigate to Phone Numbers -> Manage -> Verified Caller IDs and verify the number(s) that you want to receive a call/SMS on. Your own number (the number that you verified your Twilio account with) will already be added to the verified caller IDs.

Deploying healthchecks.io

Healthchecks.io is available as an open-source project on GitHub: https://github.com/healthchecks/healthchecks as mentioned earlier. It is built using a very popular Python web framework - Django. If you are not familiar with Django, it's fine, because setting up healthchecks does not need any Django expertise. The basic setup procedure is described in the readme.

To do this deployment we will be using two things:

1. A docker image of healthchecks (already available officially at DockerHub)

2. A PostgreSQL database instance for the database.

I will be using a service called render.com to host the healthchecks service which is a cloud platform that supports rapid development. It offers features to streamline the process of building, deploying, and scaling applications. It comes with a generous free tier to run under-development applications easily.

To get started, visit render.com and sign up for an account. Then follow these steps -

Create a PostgreSQL database

1. Click on New + and then click PostgreSQL.

2. Give a unique name to your instance, select a region that is closest to you (like I chose Singapore as I live in India), choose the instance type as Free (unless you want to pay and are doing a production deployment), and leave all the other options as default.

3. Click on Create Database.

4. After the database is created, on the detail page there will be a connections section. We will need some of these pieces of information to connect to the db from our healthchecks container.

Deploy the docker image of healthchecks

1. On the render.com dashboard, click on New + and then click Web Service .

2. From the next page, choose the option Deploy an existing image from a registry.

3. Enter the Image URL as healthchecks/healthchecks (mentioned in the DockerHub page) and click next.

4. Now give a unique name, choose the closest region and Instance type as Free.

5. There is a section called Environment Variables where we need to set a lot of configuration parameters for our healthchecks instance. The variables come as key-value pairs and are explained in the following table.

6. After filling in all the environment variables, click Create Web Service. The service will be deployed within a few minutes.

7. After the deployment is done, on the service page you will find the service URL that looks like this: https://<service_name>-<random_string>.onrender.com

8. Copy the URL, and then go to the Environment tab of the left sidebar and paste it as the value of SITE_ROOT. Click Save Changes to save and restart the deployment.

9. Once the deployment is done, our healthchecks site will be up at the URL.

Sign up and set up healthchecks

1. Click Sign Up on the top right corner of our healthchecks site, enter your email address and click Email me a link. You will receive a login link in your email.

2. Once you click the login link, you will be directed to the home page that looks like this below:

3. Now that we are ready with a working deployment, let's proceed with the call and SMS notifications.

How checks work in healthchecks.io

1. You register a check in the dashboard, and you get a ping URL.

2. Every new check is marked as inactive at the start.

3. Every check has a period and grace time setting.

4. You can set the period via simple interval, cron string or onCalender expressions.

5. You need to configure your app (that you want to monitor) to send an HTTP HEAD/GET/POST/PUT request to the ping URL according to the interval. Once a check gets a ping, its status is marked as up.

6. If healthchecks does not receive the ping at the specified interval, the check is marked delayed and the grace time counter starts.

7. If there is no ping even within the grace time, healthchecks marks the check as down .

8. As soon as the check is marked down, healthchecks send notifications to all registered integrations associated with the check.

Call notifications

Call notifications use Twilio's API, which is built in with healthchecks. The idea is simple: call the specified number and play a predefined message when a check goes down. The steps are -

1. Go to Integrations menu and find Phone Call and click it's Add Integration button.

2. 

   Add a Label and your phone number where you want to receive the alert call and save the integration.

   💡

   Note: As the Twilio account is a trial account, you must add a verified caller ID (phone number) here. Otherwise, this integration would not work.

3. Send a test notification From the integrations list to check if the integration works. You will receive a call within a minute if everything is working.

The audio sample for this test is given below:

SMS Notifications

SMS Notifications are very similar to phone calls as they also use Twilio APIs. To enable SMS notifications follow the steps -

1. In the Integrations menu find SMS and click it's Add Integration button.

2. 

   Add a label, your phone number, and check when you want a notification. Remember to verify your number if you are using a Twilio trial account.

3. Send a test notification and verify the integration. A sample notification looks like this:

   

Email Notifications

The most basic integration, i.e. email should be already added as a sample where you get notifications in the email that you use to sign up. You can more email integrations if you want to send email notifications to more people.

Conclusion

Phew! That was a very long blog. If you are still here, thank you for reading!

Important takeaways for deployment

I have made a very quick and easy deployment of healthchecks which is never recommended for production use. The free tier of render.com is very limited where web services spin down when inactive and Postgres instances are deleted after 90 days. You should consider deploying healthchecks in a properly designed fault-tolerant VM or container to use it in production.

Aren't you forgetting something?

Remember our cat image generator app? You can now plug in the ping URL to this function below and get a health check every minute!

def send_heartbeat():
    print('The cat is alive 😸, heartbeat sent!')
    requests.get('<heartbeat-monitoring-service-url>')

Healthcheck will send a notification via call and SMS when your app goes down.

Please let me know in the comments how you are implementing your own use cases of heathchecks (my favourite use case is monitoring cron jobs that fail silently). If you like this post, please drop a like to this blog, and share it with anyone who might get some benefit out of it.

Signing off 😴.