Background Tasks

Django Intermediate/Advanced — Background Tasks
Django Intermediate/Advanced
Course 2 · Chapter 5 · Background Tasks

⏳ Background Tasks

The TypeScript course's Performance chapter warned about blocking Node's single event loop with slow synchronous work; Django faces the same problem from a different angle — a slow email send or report generation blocks the entire request/response cycle for the one user waiting on it. Celery is Python's standard answer: hand slow work to a separate worker process entirely, and return to the user immediately.

The Architecture: App, Broker, Worker

Celery introduces three separate pieces working together — a genuinely different shape from Node's in-process worker_threads:

Your Django App

Calls my_task.delay(...) — this doesn't run the task, it just places a message describing the task onto a queue and returns immediately.

The Broker (Redis/RabbitMQ)

A message queue sitting between the app and the workers — it holds pending task messages until a worker is free to pick one up.

Worker Process(es)

One or more separate, long-running processes (often on a different machine entirely) that pull tasks off the broker and actually execute them.

Separate Process, Not Just a Thread

Unlike Node's worker_threads (same machine, same process tree), a Celery worker can run anywhere — scale it independently of your web servers entirely.

⚙️ Setting Up Celery

pip install celery redis
# mysite/celery.py import os from celery import Celery os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings") app = Celery("mysite") app.config_from_object("django.conf:settings", namespace="CELERY") app.autodiscover_tasks()
# catalog/tasks.py from celery import shared_task from django.core.mail import send_mail @shared_task def send_welcome_email(user_id): from django.contrib.auth.models import User user = User.objects.get(pk=user_id) # re-fetch — see this chapter's gotcha send_mail("Welcome!", f"Thanks for signing up, {user.username}!", "noreply@example.com", [user.email])

Calling a Task: .delay()

Blocking vs Backgrounded

Synchronous: Blocks the Response
def register(request): user = User.objects.create_user(...) send_mail("Welcome!", ...) # the request waits for the SMTP round trip return redirect("dashboard")
Backgrounded: Returns Immediately
def register(request): user = User.objects.create_user(...) send_welcome_email.delay(user.pk) # queues the task, returns instantly return redirect("dashboard")

.delay(...) is shorthand for the more configurable .apply_async(args=[...]) — reach for apply_async directly when you need to set things like a countdown delay or a specific queue.

Upgrading Chapter 4's Signal to Use a Task

The post_save welcome-email idea from the last chapter had one problem: a signal receiver runs synchronously, inline with the request that triggered it. Combining it with a task fixes that directly:

# catalog/signals.py from django.db.models.signals import post_save from django.dispatch import receiver from django.contrib.auth.models import User from .tasks import send_welcome_email @receiver(post_save, sender=User) def queue_welcome_email(sender, instance, created, **kwargs): if created: send_welcome_email.delay(instance.pk) # decoupled AND non-blocking

Task Retries

A task can fail — an email provider timing out shouldn't mean the email is lost forever:

@shared_task(bind=True, autoretry_for=(ConnectionError,), retry_backoff=True, max_retries=3) def send_welcome_email(self, user_id): from django.contrib.auth.models import User user = User.objects.get(pk=user_id) send_mail("Welcome!", f"Thanks, {user.username}!", "noreply@example.com", [user.email])

autoretry_for + retry_backoff automatically retries with increasing delays on the listed exception types — the same "don't just fail silently" instinct behind the retry logic in the TypeScript course's Deployment chapter, applied here to a single task instead of a whole deployment.

Periodic Tasks: Celery Beat

Beyond one-off background work, Celery Beat schedules a task to run repeatedly — a nightly report, an hourly cache warm-up — configured in settings.py rather than a separate cron entry on the server.

💻 Coding Challenges

Challenge 1: Write a Background Task

Write a generate_book_report(book_id) task that re-fetches a Book by its pk and prints a summary line, then call it with .delay() from a view instead of running it inline.

Goal: Practice the pass-an-ID-not-an-object pattern for task arguments.

→ Solution

Challenge 2: Add Retry Logic

Extend the task from Challenge 1 to automatically retry up to 3 times with backoff if it raises a ConnectionError, using @shared_task's autoretry_for and retry_backoff options.

Goal: Practice configuring automatic retries instead of letting a transient failure silently drop the task.

→ Solution

Challenge 3: Fix a Task That Passes a Model Instance

Given a broken task defined as def process_order(order): ... called as process_order.delay(order) (passing the actual Order instance), rewrite both the task and the call site to pass order.pk instead, and explain what would actually go wrong with the original version.

Goal: Practice recognizing and fixing the model-instance-as-task-argument mistake.

→ Solution

⚠️ Gotcha: Passing a Model Instance Instead of Its ID

Celery task arguments are serialized (typically to JSON) to travel across the broker to a worker — my_task.delay(book_instance) either fails outright or silently pickles a stale snapshot of that object as it existed at the moment .delay() was called. By the time a worker actually runs the task (seconds, or under load, minutes later), the real database row may have changed or been deleted entirely. Always pass a primary key (my_task.delay(book.pk)) and re-fetch the object fresh inside the task. Separately: .delay() calls queue silently even with zero workers running — if tasks never seem to execute, check that celery -A mysite worker is actually running before assuming the code is broken.

🎯 What's Next

Background tasks handle slow work; the next chapter speeds up repeated work instead: Caching — Django's cache framework, and per-view and template-fragment caching for data that doesn't need to be recomputed on every single request.