How to Build Progress Bars for the Web with Django and Celery

Progress bars are an essential part of modern web applications. They visually indicate the completion of a process that takes significant time, such as uploading files, generating reports, or performing complex calculations. A well-designed progress bar improves the user experience by communicating that the application is working on the request, as opposed to being unresponsive.

In this post, we‘ll walk through building progress bars in a Django web application. We‘ll use Celery, a popular Python library for background task processing, to handle the time-consuming work asynchronously. The frontend will poll the backend for updates and reflect the progress in a styled HTML bar.

Here‘s a high-level view of the components involved:

• Frontend (HTML, CSS, JavaScript) – displays the progress bar and polls for updates
• Backend (Django, Celery) – processes tasks asynchronously and provides progress data
• Communication (HTTP, Celery result backend) – frontend and backend exchange progress info

Let‘s dive in and start building!

Creating the Frontend Progress Bar

We can achieve the visual appearance of a progress bar with just a few HTML elements and some CSS styling. The basic markup looks like:

<div id="progress-wrapper">
  <div id="progress-bar"></div>
  <div id="progress-message"></div>
</div>

The outer div serves as a container. Inside it, the progress-bar will expand from 0 to 100% width to indicate progress. The progress-message displays a text representation of the progress like "37 out of 100 items processed."

Some simple CSS positions the elements and makes the progress bar look nice:

#progress-wrapper {
  border: 1px solid #ccc;
  width: 500px;
  height: 25px;
  background-color: #f0f0f0;
  margin-top: 10px;
}

#progress-bar {
  width: 0;
  height: 100%;
  background-color: #4CAF50;
}

#progress-message {
  margin-top: 5px;
}

To update the progress bar from JavaScript, we‘ll write a function that takes a progress object and updates the bar‘s width and message:

function updateProgress(progress) {
  const bar = document.getElementById("progress-bar");
  const message = document.getElementById("progress-message");

  bar.style.width = progress.percent + "%";
  message.textContent = progress.current + ‘ out of ‘ + progress.total + ‘ processed.‘;
}

With the frontend pieces in place, let‘s turn our attention to the backend.

Processing Tasks Asynchronously

Imagine the user kicks off a long-running operation by submitting a form in the browser. The straightforward approach would be to do the work directly in the Django view function that handles the form:

def my_view(request):
    do_time_consuming_work()
    return HttpResponse(‘Work done!‘)

However, this has some serious drawbacks. Most notably, the user would be stuck waiting for the request to complete without any indication of progress. Web servers can only handle a limited number of simultaneous requests, so tying them up with long operations leads to poor performance.

A better approach is to delegate the work to a background task queue. Celery is a powerful library for this purpose. When the form is submitted, the view creates a Celery task representing the work and returns immediately. Celery distributes the tasks to worker processes which run independently from the web server.

Here‘s what the view looks like with Celery:

from my_app.tasks import do_work

def my_view(request):
    do_work.delay()
    return HttpResponse(‘Work started!‘)

The delay() call is all it takes to invoke the task asynchronously.

Of course, we need to define the actual Celery task:

from celery import task

@task
def do_work():
    # do the actual work here...
    pass 

The @task decorator tells Celery that this function represents an asynchronous task.

To run Celery, we need to start some worker processes and make sure they can communicate with the web application. This communication happens via a message broker like RabbitMQ or Redis. Check out the Celery docs for detailed instructions on setting up a working environment.

Tracking Progress

Now that the heavy lifting is happening in the background, how can we know what the progress is?

We‘ll update our task to accept a progress_observer that it notifies after each unit of work:

@task
def do_work(progress_observer):
    items = [...] # some collection of work items
    total = len(items)

    for i, item in enumerate(items):
        # do work on item...
        # notify observer of progress
        progress_observer.update(i + 1, total)

The ProgressObserver is a simple class that takes the current and total count, calculates the percentage, and stores it:

class ProgressObserver:

    def __init__(self, task):
        self.task = task

    def update(self, current, total):
        percent = (current / total) * 100
        self.task.update_state(
            state=‘PROGRESS‘,
            meta={
                ‘current‘: current,
                ‘total‘: total,
                ‘percent‘: percent
            }
        )

Celery provides the update_state method to modify task metadata. We store the current and total count as well as the completion percentage in the task‘s state.

With this observer hooked up, each iteration of the task will push an update. The next step is to expose that data to the frontend.

Communicating Progress

In order for the web frontend to know the progress, it needs some way to fetch task status from Django. We‘ll create a JSON endpoint that returns the task metadata.

First, define a URL pattern and view:

# urls.py
path(‘task_status/<str:task_id>/‘, views.task_status, name=‘task_status‘)

# views.py
from celery.result import AsyncResult

def task_status(request, task_id):
    task = AsyncResult(task_id)

    if task.state == ‘PENDING‘:
        response = {
            ‘state‘: task.state,
            ‘current‘: 0,
            ‘total‘: 1,
            ‘percent‘: 0,
        }
    elif task.state == ‘PROGRESS‘:
        response = {
            ‘state‘: task.state,
            ‘current‘: task.info.get(‘current‘, 0),
            ‘total‘: task.info.get(‘total‘, 1),
            ‘percent‘: task.info.get(‘percent‘, 0),
        }

    return JsonResponse(response)

This view uses Celery‘s AsyncResult to retrieve the task by ID. Celery stores tasks and their metadata in its "result backend", which is a separate database like Redis.

The possible task states are:

• PENDING – task is waiting to be executed
• STARTED – task has begun processing
• PROGRESS – task is in progress and sending updates
• SUCCESS – task completed successfully
• FAILURE – task failed

If the task is in the PROGRESS state, we extract the current, total, and percent values from the task.info dict. Otherwise, we just report the state with some placeholder progress values.

On the frontend, we‘ll poll this endpoint and update the progress bar. Here‘s a basic polling function:

function pollProgress(taskId) {
  fetch(`/task_status/${taskId}/`)
    .then(response => response.json())
    .then(data => {
      updateProgress(data);

      // poll every half second if task is in progress  
      if (data.state === ‘PROGRESS‘) {
        setTimeout(pollProgress, 500, taskId);
      }
    });
}

This sends a request to the /task_status/ endpoint every 500ms as long as the task is running. When a response arrives, it extracts the JSON data and passes it to our updateProgress function from earlier.

To kick off the polling, we‘ll start it when the form is submitted. The view that handles the form should return the ID of the created task:

def my_view(request):
    # ...
    task = do_work.delay(ProgressObserver)
    return JsonResponse({‘task_id‘: task.id})

Then the frontend can grab the ID and begin polling:

const form = document.getElementById(‘my-form‘);
form.addEventListener(‘submit‘, event => {
  event.preventDefault();

  const formData = new FormData(form);

  fetch(‘/submit/‘, {
    method: ‘POST‘,
    body: formData
  })
    .then(response => response.json()) 
    .then(data => {
      pollProgress(data.task_id);
    });
});

Putting It All Together

We‘ve covered all the key aspects of building a progress bar with Django and Celery. Here‘s how it flows end-to-end:

1. User submits a form
2. Django view creates a Celery task to handle the work and returns the task ID
3. Frontend begins polling /task_status/task_id/ for updates
4. Celery workers process the task in the background, notifying the ProgressObserver as they go
5. /task_status/ endpoint reads progress data from Celery and returns it as JSON
6. Frontend updateProgress function receives the data and adjusts the progress bar

This basic implementation can be extended in many ways. Some possibilities:

• Use WebSockets instead of polling for real-time progress updates
• Have the frontend cancel the task if the user navigates away
• Display a completion message or redirect when the task is finished
• Handle and display task failure states

I hope this post has given you a solid foundation for implementing progress bars in your own Django projects. Celery is an incredibly powerful tool for background processing, and we‘ve only scratched the surface of what it can do. Combine it with a bit of frontend JavaScript, and you can create a much nicer experience for users waiting on long operations.

Now go forth and put progress bars all the things! Your users will thank you.