How to Manipulate Data with Django Migrations

As a Django developer, you have likely faced the challenge of evolving your database schema to keep up with changes to your models as your application grows. Fortunately, Django provides a powerful tool called migrations to help manage these database changes and manipulate your application‘s data.

In this in-depth guide, we‘ll explore what Django migrations are, how they work under the hood, and walk through examples of using them to rename models, add and remove fields, and handle more complex schema changes. By the end, you‘ll have the knowledge and best practices to confidently update your Django models and database without fear of losing data. Let‘s dive in!

What are Django Migrations?

At their core, Django migrations are Python files that describe changes to your database schema. They work by comparing your current models to the existing state of your database, and generating the necessary SQL commands to update the database to match your models.

When you first create a Django app, the database starts off with no tables. As you add model classes and fields in your models.py file, Django can automatically generate the migration files required to create the corresponding database tables and columns.

Each migration file contains one or more Operations that tell Django what actions need to be performed on the database. Examples of migration operations include creating and deleting models/tables, adding or removing fields, altering field attributes, and even running custom SQL.

By applying migrations, Django updates your database schema to match the current state of your models. Migrations keep a history of all the database changes so you can track how the schema evolved over time.

In addition to updating the database schema, migrations are also useful for manipulating existing data when you need to rename models/tables, move columns from one table to another, split columns into multiple new ones, convert data to a new format, etc.

Understanding Database Tables and Django Models

Before we dive into an example of using migrations to manipulate data, it‘s important to understand the relationship between database tables and Django models.

If you‘re familiar with SQL databases, you can think of a database table like a spreadsheet. Each table has a set number of columns and any number of rows. The columns define the structure, or schema, of the data being stored. Each row represents an individual record or instance of data.

In Django, each model class maps to its own database table. The model fields map to columns in the database table. So when you define a model in Django like this:

from django.db import models

class Recipe(models.Model):
  name = models.CharField(max_length=200)  
  ingredients = models.TextField()
  instructions = models.TextField()
  created_at = models.DateTimeField(auto_now_add=True)

Django will create a corresponding "recipes" table in the database with columns for an auto-incrementing primary key, name, ingredients, instructions, and created_at.

Each row in the "recipes" table represents a single Recipe object or instance of the Recipe model. The data in the columns are the field values for that particular Recipe instance.

With this understanding of how models and database tables are related, let‘s look at an example of evolving a model using migrations.

Renaming a Model and Updating the Database Schema

Imagine you have an existing Django app with the Recipe model defined above. Perhaps after the app has been deployed for a while, you realize that "Recipe" is too generic of a name. Your app has grown and now includes non-food instructions like how-to guides and tutorials.

To update the name to something more appropriate, you‘ll need to:

1. Rename the model class in the models.py file
2. Generate a migration file to rename the database table
3. Apply the migration to update the database schema

Here are the steps:

First, open your models.py file and rename the Recipe class to something like Instructions:

class Instructions(models.Model):
  name = models.CharField(max_length=200)
  ingredients = models.TextField() 
  instructions = models.TextField()
  created_at = models.DateTimeField(auto_now_add=True)

Save the file, then open a terminal and navigate to your Django project‘s directory. Run the makemigrations command to generate the migration file:

python manage.py makemigrations

Django will detect that the model name has changed and prompt you to confirm the rename:

Did you rename the recipes.Recipe model to Instructions? [y/N] y
Migrations for ‘recipes‘:
  recipes/migrations/0002_auto_20200922_2331.py
    - Rename model Recipe to Instructions

Enter "y" to confirm and let Django generate the migration. The migration file will contain the RenameModel operation to handle renaming the database table:

from django.db import migrations

class Migration(migrations.Migration):

  dependencies = [
    (‘recipes‘, ‘0001_initial‘),
  ]

  operations = [
    migrations.RenameModel(
      old_name=‘Recipe‘,
      new_name=‘Instructions‘,
    ),
  ]

To apply the migration and update your database schema, run:

python manage.py migrate

And that‘s it! The "recipes" table in your database will be renamed to "instructions" and your app will use the new Instructions model going forward. All the existing rows of data will be preserved.

This same process can be used any time you need to rename a model. Just be sure to run the migrations command to update the database and avoid inconsistencies between your models and schema.

Adding, Removing, and Altering Fields

In addition to renaming models, you can also use migrations to add, remove, or modify fields on an existing model.

For example, let‘s say you want to add a new "author" field to the Instructions model to track who submitted each one. To do this, you would:

1. Add the new field to the model in models.py
2. Generate a migration file
3. Apply the migration to update the database schema

Here are the steps:

First, edit the Instructions model to add the new author field:

class Instructions(models.Model):
  name = models.CharField(max_length=200)
  ingredients = models.TextField()
  instructions = models.TextField() 
  author = models.CharField(max_length=100)
  created_at = models.DateTimeField(auto_now_add=True)

Then generate the migration file:

python manage.py makemigrations

This will create a new migration with the AddField operation:

from django.db import migrations, models

class Migration(migrations.Migration):

  dependencies = [
    (‘recipes‘, ‘0002_auto_20200922_2331‘),
  ]

  operations = [
    migrations.AddField(
      model_name=‘instructions‘,
      name=‘author‘,
      field=models.CharField(max_length=100),
    ),
  ]

Finally, apply the migration:

python manage.py migrate

Now your database table will have a new "author" column and you can start tracking who submitted each set of instructions.

Removing a field is just as straightforward. Simply delete the field from the model definition, generate a new migration, and apply it. For example, to remove the ingredients field:

class Instructions(models.Model):
  name = models.CharField(max_length=200)
  instructions = models.TextField()
  author = models.CharField(max_length=100)
  created_at = models.DateTimeField(auto_now_add=True)

python manage.py makemigrations
python manage.py migrate

The generated migration file will contain the RemoveField operation this time:

migrations.RemoveField(
  model_name=‘instructions‘,
  name=‘ingredients‘,
),

After applying the migration, the "ingredients" column will be deleted from the database table.

To modify a field, such as changing its data type, name, or configuration options, you can update the field definition in the model and generate/apply a migration in the same way. Django will detect what has changed and generate the appropriate operations.

Planning and Executing Migrations

Now that you‘ve seen how to use migrations to rename models and evolve fields, let‘s discuss some best practices to keep in mind as you work with migrations.

First and foremost, it‘s critical to always backup your database before performing any migrations. Migrations modify your database schema and data, so if something goes wrong, you‘ll want a way to revert back to a previous state. Make sure you have a recent backup that you can restore.

When you‘re planning a migration, take the time to think through all the changes you want to make and determine the appropriate steps. It‘s often best to make one small change at a time, generate a migration, and test it before moving on to the next change. Doing too many things at once risks mistakes and makes it harder to troubleshoot issues.

Be mindful of any code in your application that interacts with the fields/models being changed. Updating a field name or type could break application logic that depends on the old name/type. It‘s a good idea to update your application code to work with the new structure before generating and applying the migration.

After making your changes and generating the migration files, carefully review the generated operations to make sure they align with your expectations. The sqlmigrate command is useful for seeing the actual SQL that will be executed:

python manage.py sqlmigrate yourapp 0003

Once you‘re confident the migrations are correct, apply them to your development database and thoroughly test your application to spot any problems. It‘s much better to identify issues in development than after deploying to production.

After testing the migrations locally, you can push your code changes and generated migration files to version control. Be sure to apply the migrations after deploying the new code to production. It‘s wise to take a final backup of the production database right before doing this.

Taking a methodical approach like this minimizes the risk of data loss or downtime. While it takes a bit more planning and care, it‘s well worth it to keep your data safe!

Handling More Complex Migrations

As your application evolves, you may run into cases where you need to do more complex things like converting data to a new format, combining multiple columns into one, or splitting one columns into many. These types of changes can also be handled in migration files.

When you can‘t express the change you need via the built-in migration operations, you can use the RunPython and RunSQL operations to execute custom Python and SQL code respectively.

For example, imagine we want to split the name field on the Instructions model into separate first_name and last_name fields. We would first update our model definition:

class Instructions(models.Model):
  first_name = models.CharField(max_length=100)
  last_name = models.CharField(max_length=100)
  instructions = models.TextField()
  author = models.CharField(max_length=100) 
  created_at = models.DateTimeField(auto_now_add=True)

Then generate an empty migration file:

python manage.py makemigrations --empty yourapp

In the empty migration file, we can use RunPython to write a custom function that splits the name values and saves them to the new first_name and last_name fields:

from django.db import migrations

def split_names(apps, schema_editor):
  Instructions = apps.get_model(‘yourapp‘, ‘Instructions‘) 
  for inst in Instructions.objects.all():
    names = inst.name.split(‘ ‘)
    inst.first_name = names[0]
    inst.last_name = names[-1]
    inst.save()

class Migration(migrations.Migration):

  dependencies = [
    (‘yourapp‘, ‘0003_auto_20200131_1622‘),
  ]

  operations = [
    migrations.RunPython(split_names),
    migrations.RemoveField(
      model_name=‘instructions‘,
      name=‘name‘,
    ),
  ]

Applying this migration will iterate through all the existing Instructions instances, split the name value on spaces, save the first part to first_name and second part to last_name, and finally drop the original name column altogether.

Whenever you use RunPython or RunSQL, be very careful to test the migration thoroughly, as it‘s easy to make mistakes in custom code that could lead to data loss.

Conclusion

Learning how to use Django migrations to evolve your database schema and manipulate data is an essential skill for any Django developer. They are an incredibly powerful tool when used correctly.

Remember that migrations generate permanent changes to your database, so it‘s important to plan them carefully, review the generated operations, and test extensively before applying to a production database. Always make sure you have a recent backup in case you need to roll back.

Through our example of renaming models, adding/removing fields, and exploring more advanced cases, you should now have a solid foundation for working with migrations in your own Django projects.

Embrace migrations and use them to keep your database up-to-date as your application grows and evolves over time!