How to Create a Self-Documenting Makefile: An Expert Guide

As a seasoned full-stack developer, you‘ve likely worked with dozens, if not hundreds, of different projects over the years. Each project has its own unique setup, dependencies, build process, and development workflow. Remembering all the various commands and steps for each project can be a daunting task, even for the most experienced developers.

This is where Makefiles come in. While traditionally used as a build automation tool, Makefiles can also be leveraged to encode project-specific workflows and tasks in a clean, concise, and self-documenting way. In this in-depth guide, we‘ll explore the concept of self-documenting Makefiles, dive into Makefile syntax and best practices, and showcase real-world examples of this technique in action.

The Resurgence of Makefiles

Makefiles have been around since the 1970s, originally developed as part of the Unix ecosystem for automating software builds. For many years, Makefiles were the de facto standard for build automation, especially in the C/C++ world. However, with the rise of newer build tools like Ant, Maven, Gradle, and others in the early 2000s, Makefiles fell somewhat out of favor.

In recent years, though, Makefiles have seen a resurgence in popularity, particularly in the world of polyglot programming and microservices. As developers work across many different languages and frameworks, each with their own build tools and conventions, the simplicity and universality of Makefiles has become appealing again.

This Google Trends data shows the relative search interest in ‘makefile‘ over time. While interest has declined from its peak in the early 2000s, it has remained steady and even slightly increased in recent years.

Makefile Syntax Basics

Before we dive into self-documenting Makefiles, let‘s review the basics of Makefile syntax. A Makefile consists of a series of rules, each of which defines a target, its dependencies, and the commands to build or run the target.

Here‘s the general syntax of a Makefile rule:

target: prerequisites
    recipe

• The target is the name of the rule. It‘s usually the name of a file that is generated by the recipe (like a compiled binary), but it can also be a phony target that just runs a set of commands.
• The prerequisites are files or other targets that need to exist before the target can be built. Make will ensure these are built first if needed.
• The recipe is a series of shell commands to run to build the target. These commands must be indented with a tab character (not spaces).

Here‘s a simple example of a Makefile with two rules:

hello: hello.c
    gcc -o hello hello.c

clean:
    rm -f hello

The first rule, hello, has a prerequisite of hello.c and a recipe that compiles hello.c into an executable named hello. The second rule, clean, has no prerequisites and a recipe that removes the hello executable.

You can invoke these rules by running make <target> in the directory containing the Makefile. For example, make hello would compile the hello executable, and make clean would remove it.

The Problem with Complex Makefiles

As projects grow in complexity, so do their Makefiles. It‘s not uncommon for a large project‘s Makefile to contain dozens or even hundreds of rules, covering everything from building and testing to deploying and generating documentation.

While this can be very powerful, it also leads to a discoverability problem. With so many rules in a Makefile, it can be difficult for developers (especially new ones) to know what targets are available and what they do. This often leads to the Makefile becoming a sort of mystical artifact – everyone knows it‘s important, but few understand its inner workings.

Inline comments can help, but they‘re not a perfect solution. You still need to open up the Makefile and read through it to understand what‘s available. What we really need is a way for the Makefile to document itself.

The Solution: Self-Documenting Makefiles

This is where the concept of a self-documenting Makefile comes in. The idea is simple: we use a special syntax in the Makefile to annotate each rule with a brief description of what it does. Then, we create a special help rule that parses these annotations and prints out a user-friendly list of available targets and their descriptions.

Here‘s an example of what this might look like:

.PHONY: help
help: ## Print this help message
    @grep -E ‘^[a-zA-Z_-]+:.*?## .*$$‘ $(MAKEFILE_LIST) | awk ‘BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}‘

run: ## Run the application
    python main.py

test: ## Run the test suite
    pytest tests/

lint: ## Lint the codebase
    flake8 src/

format: ## Auto-format the codebase
    black src/

deploy: ## Deploy the application
    ansible-playbook deploy.yml

In this example, we use a double-hash ## syntax to annotate each rule with a brief description. The help rule then uses some grep and awk magic to parse out these descriptions and print them in a nicely-formatted way.

Now, whenever a developer wants to see what‘s available in the Makefile, they can just run make help:

$ make help
help                 Print this help message
run                  Run the application
test                 Run the test suite
lint                 Lint the codebase
format               Auto-format the codebase  
deploy               Deploy the application

Voila! A self-documenting Makefile. Now the Makefile itself serves as the documentation for what‘s available and how to use it.

Real-World Examples

This self-documenting Makefile technique is used by a number of high-profile open source projects. Here are a few examples:

• Kubernetes: The popular container orchestration platform. Their root Makefile is over 1000 lines long, but includes a well-documented help target.
• OpenSSL: The ubiquitous cryptography library. Their top-level Makefile uses a similar help target to document the available build options.
• Redis: The popular in-memory data store. Their Makefile includes a help target that prints out available build and test options.

These examples demonstrate that self-documenting Makefiles are a proven technique, used by some of the most widely-used and critically-important software projects in the world.

Advanced Makefile Techniques

In addition to the self-documenting annotations, there are a number of other techniques that can make your Makefiles more readable, maintainable, and powerful:

• Variables: You can define variables in your Makefile to hold commonly-used values, paths, or options. This makes your Makefile more DRY and easier to change. For example:

  PYTHON = python3
  PYTEST = $(PYTHON) -m pytest
  PYLINT = $(PYTHON) -m pylint
  
  test:
      $(PYTEST) tests/
  
  lint:
      $(PYLINT) src/

• Automatic Variables: Make provides a number of automatic variables that can be used in your recipes. These are set by Make for each rule, and provide access to things like the target name, the prerequisites, and the recipe command. For example:

  %.o: %.c
      gcc -c $< -o $@

  Here, $< is the name of the first prerequisite (the .c file), and $@ is the name of the target (the .o file).

• Shell Functions: You can use shell functions in your Makefile recipes to encapsulate common bits of functionality. For example:

  define run_test
      @echo "Running test: $1"
      @$(PYTHON) -m pytest tests/$1
  endef
  
  test_foo:
      $(call run_test,test_foo.py)
  
  test_bar:
      $(call run_test,test_bar.py)

  Here, the run_test function is defined using a Make define directive, and then called for each test target.

These are just a few examples of the advanced techniques available in Makefiles. The GNU Make manual provides a comprehensive reference for all of Make‘s features and capabilities.

Makefile Best Practices

Here are some best practices to keep in mind when working with Makefiles:

1. Use phony targets for non-file targets: If you have a target that doesn‘t produce an output file (like test or clean), declare it as a .PHONY target. This tells Make not to worry about file timestamps for these targets.

2. Prefix your recipes with @: By default, Make prints each command in a recipe before it‘s executed. If you want to suppress this echo, prefix the command with @. This can help declutter the output, especially for targets that run many commands.

3. Use variables for repeated values: If you find yourself using the same flags, paths, or other values in multiple rules, pull them out into variables at the top of your Makefile. This makes your Makefile more DRY and easier to maintain.

4. Split large Makefiles into includes: If your Makefile starts to get too large and unwieldy, consider splitting it up into smaller files and include-ing them into the main Makefile. This can help keep things organized and modular.

5. Provide a help target: As we‘ve seen, a help target that lists out the available targets and their descriptions is invaluable for making your Makefile self-documenting and user-friendly.

6. Use consistent naming and formatting: Establish a consistent naming scheme and format for your Makefile rules and variables, and stick to it. This makes your Makefile easier to read and understand.

Conclusion

In this guide, we‘ve explored the concept of self-documenting Makefiles as a powerful technique for encoding project-specific workflows and tasks in a maintainable and user-friendly way. By using a simple annotation syntax and a clever help rule, we can make our Makefiles self-describing and easy to use for all developers on a project.

We‘ve also looked at some advanced Makefile techniques and best practices that can make our Makefiles even more powerful and maintainable, and seen real-world examples of self-documenting Makefiles in action in some of the most popular open source projects.

Whether you‘re working on a small personal project or a large-scale enterprise application, investing time in creating a well-structured, self-documenting Makefile can pay huge dividends in terms of developer productivity and happiness. By making it easy for developers to see what‘s available and how to use it, you can turn your Makefile from a source of confusion into a powerful tool for enabling and streamlining development workflows.

So go forth and make your Makefiles self-documenting! Your teammates (and your future self) will thank you.

Further Reading

• GNU Make Manual – The comprehensive manual for GNU Make, covering all aspects of Makefile syntax and functionality.
• Self-Documented Makefile – A blog post by Marmelab that goes into more detail on the self-documenting Makefile technique.
• Makefile Tutorial by Example – A step-by-step tutorial on Makefile syntax and best practices, with many practical examples.
• Managing Projects with GNU Make – A comprehensive book on using Make for project management, with coverage of advanced techniques and real-world case studies.

With these resources and the techniques covered in this guide, you‘ll be well on your way to Makefile mastery. Happy coding!