Build a Heroku Clone – Provision Infrastructure Programmatically with Pulumi

The way we build and deploy applications is rapidly evolving. Not too long ago, infrastructure provisioning involved filing tickets with central IT and waiting weeks for resources. But in today‘s fast-paced, software-driven world, that doesn‘t cut it anymore.

Developers need to be able to self-service their infrastructure, spinning up environments on-demand to test and deploy changes. At the same time, platform engineering teams need to maintain governance and control over an organization‘s infrastructure. The question is, how do you balance developer agility with centralized oversight?

The Rise of Self-Service Infrastructure

Over the past decade, public cloud platforms like AWS, Azure, and Google Cloud have revolutionized infrastructure provisioning. With just a few clicks in a web console, developers can spin up virtual machines, databases, load balancers, and more. This has been a huge boost for developer productivity.

However, the console-clicking approach doesn‘t scale well. It‘s prone to human error, lacks reproducibility, and can quickly lead to inconsistent, snowflake environments. That‘s where infrastructure as code (IaC) comes in.

With IaC, you define your infrastructure using definition files or code, and use that to provision and manage environments. This allows for consistent, repeatable deployments and makes it easy to version and peer review infrastructure changes just like you would application code.

Benefits of Programmatic Infrastructure

But the real power of IaC is realized when you provision infrastructure programmatically via APIs. This enables use cases like self-service infrastructure portals and ChatOps provisioning flows. Here are some of the key benefits:

• Empower developers – Developers can rapidly spin up the infrastructure they need without having to file tickets or wait on central IT.
• Standardize deployments – By codifying your infrastructure, you can enforce standards and best practices across environments.
• Increase velocity – Automating infrastructure provisioning removes manual bottlenecks and enables teams to ship faster.
• Improve reliability – Infrastructure is provisioned consistently every time, reducing the risk of configuration drift and failures.

Don‘t just take my word for it. According to the 2020 State of DevOps Report from DORA and Google Cloud, elite performers who have embraced these practices are crushing it in terms of speed and stability. They deploy 208 times more frequently and have 106 times faster lead time from commit to deploy compared to low performers.

Pulumi – Infrastructure as Code for Developers

So how do you actually implement programmatic infrastructure provisioning? That‘s where Pulumi comes in. Pulumi is an open source infrastructure as code platform that allows you to use familiar programming languages to define and deploy cloud infrastructure.

Why Pulumi?

With Pulumi, you write code to define your desired infrastructure state, and Pulumi figures out how to make the cloud resources match that state. You can use TypeScript, JavaScript, Python, Go, or .NET – no more learning custom DSLs or templating languages.

This means you can use all the same software engineering practices you know and love – code reuse, abstractions, IDE integration, testing, etc. – and apply them to infrastructure. Pulumi becomes an SDK and runtime for your infrastructure.

Let‘s look at a simple example. Say you want to define an AWS S3 bucket in TypeScript:

import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("my-bucket", {
    website: {
        indexDocument: "index.html",
    },
});

export const bucketName = bucket.id;

This creates an S3 bucket with website hosting enabled. Pulumi automatically figures out the dependencies between resources and builds a directed acyclic graph (DAG) to determine the order of operations. The bucketName is exported as an output so it can be referenced by other parts of the program.

Componentizing Infrastructure

But Pulumi really shines when it comes to componentizing infrastructure. You can define reusable abstractions and package them up for easy consumption. For example, here‘s how you might define a reusable WebsiteBucket component:

export class WebsiteBucket extends pulumi.ComponentResource {
    public readonly bucketName: pulumi.Output<string>;
    public readonly websiteUrl: pulumi.Output<string>;

    constructor(name: string, args: WebsiteBucketArgs, opts?: pulumi.ComponentResourceOptions) {
        super("pkg:index:WebsiteBucket", name, args, opts);

        const bucket = new aws.s3.Bucket(`${name}-bucket`, {
            website: {
                indexDocument: args.indexDocument || "index.html",
            },
        }, { parent: this });

        this.bucketName = bucket.id;
        this.websiteUrl = bucket.websiteEndpoint;
    }
}

This component encapsulates the configuration of the S3 bucket and exports the bucketName and websiteUrl for convenience. You can now reuse this component across projects:

const website = new WebsiteBucket("my-website", {
    indexDocument: "index.html",
});

export const websiteUrl = website.websiteUrl;

By componentizing your infrastructure, you can enforce best practices and create an internal library of approved infrastructure patterns for your organization.

Testing Infrastructure Code

Another key capability Pulumi enables is testing your infrastructure code. Just like with application code, you should have unit tests for your infrastructure to ensure it works as expected.

Pulumi provides testing frameworks for each supported language. For example, here‘s how you might test the WebsiteBucket component in TypeScript using the @pulumi/pulumi/testing library:

import * as pulumi from "@pulumi/pulumi";
import "mocha";
import { WebsiteBucket } from "./website";

pulumi.runtime.setMocks({
    newResource: function(args: pulumi.runtime.MockResourceArgs): {id: string, state: any} {
        return {
            id: args.inputs.name + "_id",
            state: args.inputs,
        };
    },
    call: function(args: MockCallArgs) {
        return args.inputs;
    },
});

describe("Infrastructure", function() {
    describe("#WebsiteBucket", function() {
        const website = new WebsiteBucket("test", {
            indexDocument: "index.html",
        });

        it("should create a website bucket", function(done) {
            pulumi.all([website.bucketName, website.websiteUrl]).apply(([bucketName, websiteUrl]) => {
                assert.strictEqual(bucketName, "test-bucket_id");
                assert.strictEqual(websiteUrl, "test-bucket_id.s3-website.us-east-1.amazonaws.com");
                done();
            });
        });
    });
});      

This test mocks out the Pulumi resource provider and asserts that the WebsiteBucket component creates an S3 bucket with the expected name and website URL.

Self-Service Infrastructure with Pulumi

So how does this all tie back to the Heroku clone example we walked through earlier? By defining your infrastructure as code with Pulumi, you can build self-service portals and APIs that empower developers to provision the infrastructure they need, when they need it.

Instead of clicking around a web console or filing tickets with central IT, developers can simply fill out a form or issue a ChatOps command to spin up a new environment. Behind the scenes, Pulumi is doing the heavy lifting of provisioning and configuring the necessary cloud resources.

Infrastructure Guardrails

But what about governance and control? This is where policy as code comes into play. With tools like Pulumi‘s CrossGuard, you can codify your infrastructure policies and enforce them on every deployment.

For example, you could enforce tagging standards, require encryption for data at rest, or limit which instance types can be provisioned. Developers have the freedom to self-service within the guardrails you set.

Here‘s an example of an Azure policy that requires a specific tag on every resource:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

const requiredTag = new azure.core.ResourceGroup("required_tags", {
    name: "required_tags",
    tags: {
        "Environment": "Production",
    },
}, { protect: true });

new pulumi.policy.PolicyPack("required_tags", {
    policies: [{
        name: "required-tags",
        description: "Enforces required tags on all resources.",
        enforcementLevel: "mandatory",
        validateResource: validateResourceOfType(azure.core.Resource, (resource, args, reportViolation) => {
            if (!resource.tags || !resource.tags["Environment"]) {
                reportViolation(`Missing tag ‘Environment‘ on resource ‘${resource.name}‘.`);
            }
        }),
    }],
});

This policy requires that every Azure resource have an Environment tag. If a developer tries to provision a resource without the tag, the deployment will fail.

By combining self-service infrastructure with policy as code, you can strike the right balance between developer agility and centralized governance.

ChatOps Provisioning

Another powerful pattern is ChatOps-driven infrastructure provisioning. With Pulumi‘s Webhooks, you can trigger infrastructure provisioning directly from Slack or Microsoft Teams.

For example, a developer could issue a command like /provision staging in Slack, which would kick off a Pulumi program to spin up a new staging environment. The program would report back the status in the channel, keeping everyone in the loop.

Here‘s a code snippet demonstrating a simple Pulumi Webhook handler:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as express from "express";
import * as bodyParser from "body-parser";
import * as nacl from "tweetnacl";

const secretKeyBytes = process.env.PULUMI_SECRET_KEY;
if (secretKeyBytes === undefined) { throw new Error("missing PULUMI_SECRET_KEY"); }

const app = express();
app.use(bodyParser.json());
app.post("/webhook", async (req, res) => {

    const signature = req.headers["pulumi-signature"] as string;
    const hmac = nacl.sign.detached.verify(
        Buffer.from(JSON.stringify(req.body)),
        Buffer.from(signature, "hex"),
        Buffer.from(secretKeyBytes, "hex")
    );

    if (!hmac) {
        console.log("Unauthorized request");
        return res.status(401).send("Unauthorized");
    }

    const webhookID = req.body.id as string;
    const webhookKind = req.body.kind as string;
    const commit = req.body.commit as string;

    try {
        await provisionInfrastructure();
        await notifySlack(`Successfully provisioned infrastructure for commit ${commit}`);
        res.json({ id: webhookID, status: "success" });
    } catch (err) {
        console.error(`Failed to provision infrastructure: ${err}`);
        await notifySlack(`Failed to provision infrastructure for commit ${commit}`);
        res.status(500).json({ id: webhookID, status: "failure", error: err.stack });
    }

});

// Helper function to provision a new staging environment with Pulumi
async function provisionInfrastructure() {
    // Pulumi code to provision infrastructure goes here 
}

// Helper function to post a message to Slack  
async function notifySlack(message: string) {
    // Code to post message to Slack goes here
}

export const endpoint = new aws.apigateway.x.API("pulumi-webhook", {
    routes: [{
        path: "/webhook",
        method: "POST",
        eventHandler: app
    }]  
});

export const webhookEndpoint = pulumi.interpolate`${endpoint.url}/webhook`;

This code sets up an API Gateway endpoint that listens for Pulumi Webhook events. When a new commit is pushed, it triggers the provisionInfrastructure function to spin up a new staging environment using Pulumi. It then posts a message to Slack indicating whether the provisioning succeeded or failed.

By embracing ChatOps, you can bring infrastructure provisioning directly into the conversation and make it more accessible to developers.

Infrastructure as a Product

As we‘ve seen, programmatic infrastructure provisioning is a key enabler for developer self-service and platform engineering. But it‘s only one piece of the puzzle. To truly succeed with infrastructure as code, you need to treat your infrastructure like a product.

Just like with any software product, infrastructure should have its own development lifecycle – planning, design, implementation, testing, deployment, and maintenance. It should be built and managed by a dedicated team with its own roadmap and backlog.

Metric	Without IaC	With IaC
Deployment Frequency	1/month	10/day
Lead Time for Change	1 week	1 hour
Change Failure Rate	50%	5%
Mean Time to Recover	1 day	1 hour

Sample metrics illustrating the impact of infrastructure as code

The table above shows some sample metrics comparing infrastructure delivery with and without IaC. By adopting programmatic provisioning and treating infrastructure as code, teams can achieve orders of magnitude improvements in deployment frequency, lead time for changes, and mean time to recovery.

Conclusion

In the cloud era, infrastructure is no longer just a cost center – it‘s a key enabler for application delivery and business agility. By provisioning infrastructure programmatically and enabling developer self-service, platform engineering teams can unlock new levels of velocity and innovation.

Pulumi‘s infrastructure as code platform is purpose-built for this new reality. With support for familiar programming languages, reusable components, policy as code, testing frameworks, and more, Pulumi empowers developers and infrastructure teams to work better together.

If you‘re interested in learning more about programmatic infrastructure provisioning with Pulumi, check out the following resources:

• Get Started with Pulumi
• Pulumi Case Studies
• Pulumi Blog
• Pulumi Community Slack

The future of infrastructure is code, and the time to start your journey is now!