Django Asynchronous Processing with Celery & Redis Task Queues

Blocking an HTTP thread while waiting for a remote SMTP email server or generating a multi-page PDF invoice is a critical architectural anti-pattern. Celery is an enterprise-grade distributed task queue that runs long-running operations asynchronously in the background.

Figure 40.1: The 4-Tier Decoupled Celery Architecture — Web App, Redis Broker, Worker Pool & Periodic Beat.

Step 1: Installing and Configuring Celery

# Install Celery with Redis support
pip install celery redis

Create mywebsite/celery.py alongside your settings.py:

# mywebsite/celery.py
import os
from celery import Celery

# Set default settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mywebsite.settings')

app = Celery('mywebsite')

# Read config from Django settings using namespace 'CELERY'
app.config_from_object('django.conf:settings', namespace='CELERY')

# Auto-discover tasks in all installed apps (tasks.py)
app.autodiscover_tasks()

Ensure Celery loads when Django starts in mywebsite/__init__.py:

# mywebsite/__init__.py
from .celery import app as celery_app

__all__ = ('celery_app',)

# mywebsite/settings.py
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

Step 2: Defining an Asynchronous Task with Retries

# blog/tasks.py
import logging
from celery import shared_task
from django.core.mail import send_mail

logger = logging.getLogger(__name__)

@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def send_welcome_email_task(self, user_email, user_name):
    """Sends registration email in the background with automatic retries."""
    try:
        send_mail(
            subject="Welcome to Veewom Master Class!",
            message=f"Hi {user_name},

Thank you for enrolling in our Django course.",
            from_email="noreply@veewom.com",
            recipient_list=[user_email],
            fail_silently=False,
        )
        logger.info(f"Welcome email successfully sent to {user_email}")
    except Exception as exc:
        logger.error(f"Failed sending email to {user_email}. Retrying... Error: {exc}")
        # Retry task after exponential backoff
        raise self.retry(exc=exc)

Step 3: Triggering Tasks Non-Blockingly in Views

# In accounts/views.py
from blog.tasks import send_welcome_email_task

def register_view(request):
    # ... form validation ...
    user = form.save()
    
    # Offload email to Celery! Returns instantly in under 1 millisecond:
    send_welcome_email_task.delay(user.email, user.full_name)
    
    return redirect('dashboard')

Running Workers in Development

# Terminal 1: Run Django Server
python manage.py runserver

# Terminal 2: Run Celery Worker
celery -A mywebsite worker --loglevel=info -P solo