How I Contributed to a Major Open Source Project Without Writing Any Code

As a full-stack developer and open source maintainer, I know firsthand how challenging it can be to manage a popular project. With the growth in users and contributors comes an increase in complexity, especially around building, testing and releasing new versions.

Recently, I had the opportunity to make a significant contribution to the Phoenix Framework, one of the most popular web frameworks in the Elixir ecosystem, without writing a single line of application code. In this post, I‘ll share the story of how I used an open source tool called Earthly to radically simplify Phoenix‘s testing workflow and make the development process more welcoming to contributors.

The Rise of Phoenix

To understand the impact of this change, it helps to start with some context about Phoenix and its community. Phoenix is a modern web framework built on the Elixir programming language, which itself is built on the battle-tested Erlang VM (known as the BEAM).

Since its initial release in 2014, Phoenix has seen tremendous adoption and growth. As of 2022, the project has:

• Over 18,000 GitHub stars and 2,000 forks
• More than 3 million downloads on the Hex package manager
• 100k+ monthly NPM downloads of phoenix_live_view, a key component
• An active community of over 5,000 developers on the Elixir forum

Part of the reason for Phoenix‘s popularity is its innovative approach to building interactive, real-time web applications without the complexity of client-side JavaScript frameworks. By leveraging WebSockets and server-side rendering through the LiveView library, developers can create dynamic, engaging UIs with a fraction of the code of a traditional single-page app.

Here‘s an example of what a LiveView component looks like in Phoenix:

defmodule DemoWeb.ThermostatLive do
  use Phoenix.LiveView

  def render(assigns) do
    ~L"""
    <div>
      <h2>Current temperature: <%= @temperature %>°</h2>
      <button phx-click="inc_temperature">+</button>
      <button phx-click="dec_temperature">-</button>
    </div>
    """
  end

  def handle_event("inc_temperature", _, socket) do
    {:noreply, update(socket, :temperature, &(&1 + 1))}
  end

  def handle_event("dec_temperature", _, socket) do
    {:noreply, update(socket, :temperature, &max(&1 - 1, 0))}
  end
end

In less than 20 lines of code, we have a functioning thermostat interface complete with real-time updates. Whenever the user clicks the increment or decrement buttons, a message is sent to the server, which updates the temperature state and re-renders the view. No JavaScript needed!

The Challenges of Testing at Scale

As powerful and productive as the Phoenix framework is, it‘s not without its challenges, especially when it comes to testing. Like any web framework, Phoenix needs to support multiple databases, each with their own quirks and APIs. The framework also needs to maintain backwards compatibility with older versions of Elixir and OTP (the Open Telecom Platform that powers the Erlang VM).

This results in a combinatorial explosion of test configurations. To be confident that any given change won‘t break the framework, the maintainers need to run the test suite against:

• 3 supported databases (PostgreSQL, MySQL, and MSSQL)
• 2 versions of Elixir (current stable and previous stable)
• 2 versions of OTP (current stable and previous stable)

That‘s 12 different configurations in total! And while Phoenix has an extensive suite of unit tests that can be run quickly in any environment, it also has dozens of integration tests that exercise the full stack, from the database all the way up to the view layer. These tests are critical for ensuring the overall health and functionality of the framework, but they take much longer to run and require access to live databases.

Historically, the Phoenix team ran these tests in a GitHub Actions matrix build. Each time a pull request was opened or updated, the CI system would provision the necessary databases using Docker Compose, install the appropriate versions of Elixir and OTP, fetch dependencies, and run the entire test suite. This process often took 20-30 minutes per configuration, leading to lengthy build times and consuming a significant amount of compute resources.

Worse, there was no easy way for contributors to run these tests locally before opening a pull request. The setup process was complex and error-prone, requiring multiple databases and specific versions of Elixir/OTP to be installed on the developer‘s machine. As a result, contributors often resorted to pushing changes and waiting for the CI build to catch any issues, leading to a frustrating and inefficient development cycle.

Simplifying the Workflow with Earthly

As a developer advocate at Earthly, I spend a lot of time working with open source maintainers to help them streamline their development workflows. Earthly is a build automation tool that allows developers to define their build and test process in a simple, declarative language called an Earthfile. Unlike traditional CI configuration files, an Earthfile can be run anywhere – on a developer‘s laptop, in a CI pipeline, or even in a Dockerfile.

When I heard about the challenges the Phoenix team was facing with their testing workflow, I knew Earthly could help. I opened a pull request to introduce an Earthfile that would codify the build and test process in a single, unified script. Here‘s a simplified version of what I proposed:

VERSION 0.6
FROM elixir:1.13-alpine

test-db:
  FROM postgres:14
  EXPOSE 5432

build:
  FROM +base
  RUN apk --update add git build-base nodejs npm
  WORKDIR /app
  COPY . .
  RUN mix local.hex --force && mix local.rebar --force
  RUN mix deps.get
  RUN npm install --prefix assets
  RUN mix compile

test-unit:
  FROM +build
  RUN mix test

test-integration:
  FROM +build
  WITH COMPOSE
    RUN docker-compose up -d 
    RUN mix test --only integration
  END

Let‘s break down what‘s going on here:

1. We define the base image for our build using the FROM directive. In this case, we‘re starting with the official Elixir image from Docker Hub.

2. We define a test-db target that specifies the PostgreSQL image to use for integration testing. The EXPOSE directive makes the default PostgreSQL port available to other targets.

3. The build target is where the real work happens. We install the necessary dependencies (git, build tools, Node.js), copy the application code into the image, fetch the Elixir and JavaScript dependencies, and compile the application.

4. The test-unit target runs the fast unit tests against the compiled application. This target can be run in isolation without needing any external dependencies.

5. Finally, the test-integration target sets up the integration testing environment. We use the WITH COMPOSE directive to specify the Docker Compose file to use, which includes configuration for the test database. We then run the integration tests using the mix test --only integration command.

With this Earthfile in place, running the full Phoenix test suite is as simple as typing earthly -P +test-unit +test-integration. This will execute the unit and integration test targets in parallel, spinning up the necessary database containers and running the tests against them.

But the real power of the Earthfile is that it can be parameterized to support different versions of Elixir and OTP. By using build arguments, we can dynamically swap out the base image and dependencies to test against different configurations:

earthly -P --build-arg ELIXIR_VERSION=1.12 --build-arg OTP_VERSION=24 +test-integration

This command runs the integration tests using Elixir 1.12 and OTP 24, without needing any changes to the Earthfile itself. This makes it trivial to test against the full matrix of supported configurations, both locally and in CI.

The Results

After some iteration and testing, the Phoenix team accepted my pull request to adopt the Earthfile. The results have been transformative for the project‘s development workflow:

• Contributors can now run the full test suite locally with a single command, without needing to set up databases or install specific versions of Elixir/OTP.
• The CI build time has been reduced from hours to minutes, thanks to the ability to run tests in parallel across multiple containers.
• The maintainers can easily test changes against the full matrix of supported configurations, catching issues before they‘re merged into the main branch.

In the words of Phoenix creator Chris McCord:

The Earthly integration has been a huge quality-of-life improvement for Phoenix development. Contributors can now get up and running with a single command, and our CI build times are faster than ever. We‘re excited to see how this new workflow will accelerate development and improve the stability of the framework.

Get Started with Earthly Today

If you maintain an open source project and are looking for ways to streamline your development workflow, I highly recommend giving Earthly a try. The tool is completely open source and free to use, and the documentation is top-notch.

To get started, simply install the earthly command-line tool and create an Earthfile in your project‘s root directory. You can use the Earthfile reference to learn about the available commands and syntax.

If you need help getting set up or have questions about how to integrate Earthly into your project, don‘t hesitate to reach out. I‘m always happy to lend a hand to fellow developers and maintainers. You can find me on the Earthly Community Slack or on Twitter at @adamgordonbell.

Together, we can build a more productive and sustainable open source ecosystem, one project at a time.