Learn Infrastructure as Code by Building a Custom Machine Image in AWS

As a software developer or DevOps engineer, you‘ve likely heard the term "Infrastructure as Code" (IaC) tossed around. But what exactly does it mean, and why is it so important in modern cloud-native application development?

In this in-depth guide, we‘ll break down the key concepts of IaC and walk through a hands-on tutorial on using HashiCorp Packer to build a custom machine image in AWS. By the end, you‘ll have the skills and knowledge to start applying IaC practices in your own projects. Let‘s dive in!

What is Infrastructure as Code?

At its core, Infrastructure as Code is the practice of managing and provisioning computing infrastructure through machine-readable definition files or code, rather than physical hardware configuration or interactive configuration tools.

With IaC, you define your desired infrastructure state declaratively and let automated processes handle the implementation details. This allows you to treat your infrastructure the same way you treat your application code – storing it in version control, testing it, and deploying it in a repeatable and predictable manner.

The key benefits of adopting IaC include:

• Consistency and reliability – eliminate configuration drift and "snowflake" servers
• Speed and agility – rapidly spin up infrastructure for development, testing, and production
• Risk mitigation – recover more easily from failures and outages
• Cost optimization – reduce overprovisioning and take advantage of spot instances
• Collaboration and accountability – infrastructure changes go through the same review and approval processes as code changes

Mutable vs Immutable Infrastructure

Within the realm of Infrastructure as Code, there are two main approaches: mutable and immutable.

With mutable infrastructure, servers are continually updated and modified in place after they are deployed. Configuration management tools like Chef, Puppet, and Ansible excel in mutable environments, ensuring consistent configurations across servers over time.

Immutable infrastructure, on the other hand, is never changed after deployment. If an update or fix is needed, a new version of the server is built from a common template or image and the old version is discarded. This approach reduces inconsistencies and makes deployments more predictable.

As we‘ll see, building custom machine images is a key enabler of immutable infrastructure. Tools like Docker, Vagrant, and Packer allow you to create your own images with your application and configurations "baked in", so you can be confident that what you tested is exactly what gets deployed.

Server Templating with Packer

Packer is an open source tool developed by HashiCorp for creating identical machine images for multiple platforms from a single source configuration. Some of the key features that make Packer so powerful include:

• Support for a wide range of platforms, including AWS EC2, Google Cloud, Azure, Docker, and more
• Ability to use any major configuration management tool for provisioning
• Post-processors for tagging images, uploading artifacts, and more
• Multi-provider parallel builds

The core Packer workflow involves:

1. Defining your image template in a JSON file
2. Configuring one or more "builders" that create the base machine image for your desired platform(s)
3. Provisioning the image using "provisioners" to install software, copy files, etc.
4. Applying any "post-processors" to the artifact
5. Building the image
6. Launching the image and testing

To illustrate these concepts, let‘s walk through an example of building a custom AWS AMI that has Node.js and a sample application pre-installed, all configured through code.

Step 1: Prerequisites

Before we get started, you‘ll need:

• An AWS account
• AWS CLI installed and configured with your credentials
• Packer installed on your local machine

Step 2: Define the Template

Create a file named node_app.json and add the following:

{
  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-east-1",
    "source_ami_filter": {
      "filters": {
        "virtualization-type": "hvm",
        "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",
        "root-device-type": "ebs"
      },
      "owners": ["099720109477"],
      "most_recent": true
    },
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "node-app-{{timestamp}}"
  }]
}

Here‘s what‘s happening:

• The variables block defines inputs that can be used elsewhere in the template. In this case, we‘re reading the AWS access key and secret key from environment variables for security.

• The builders block configures one or more builders that create the base image. We‘re using the amazon-ebs builder to create an Amazon Machine Image (AMI) backed by an EBS volume.

• The source_ami_filter section tells Packer to start from the latest Ubuntu 16.04 AMI. We could also specify an explicit source AMI ID.

• The instance_type specifies the instance type to use when launching the source AMI for building. We‘re using a t2.micro to qualify for the AWS free tier.

• ssh_username sets the username to connect to SSH with. For Ubuntu AMIs this is the ubuntu user.

• ami_name defines the name of our output AMI, appending a timestamp for uniqueness.

Step 3: Add Provisioners

Now let‘s add a provisioner to install Node.js and our application code onto the image:

{
  "variables": {...},
  "builders": [...],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -",
      "sudo apt-get install -y nodejs",
      "sudo apt-get install -y git",
      "git clone https://github.com/yourusername/node-sample-app.git",
      "cd node-sample-app",
      "npm install",
      "npm run build"
    ]
  }]  
}

This shell provisioner executes the specified commands in order on the running instance. The commands will:

1. Add the NodeSource APT repository for Node.js 14.x
2. Install Node.js and Git
3. Clone a sample Node.js application from GitHub
4. Install the application‘s dependencies
5. Build the application

Step 4: Build the Image

With our template defined, we‘re ready to build the image. Run:

packer build node_app.json

Packer will launch a source instance, run the provisioners, create an AMI from the instance, and then terminate the instance. The output will look something like:

==> amazon-ebs: Creating temporary keypair: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
==> amazon-ebs: Creating temporary security group for this instance: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs: Launching a source AWS instance...
    amazon-ebs: Instance ID: i-xxxxxxxxxxxxxxxxx
==> amazon-ebs: Waiting for instance (i-xxxxxxxxxxxxxxxxx) to become ready...
==> amazon-ebs: Using ssh communicator to connect: 11.222.333.444
...
==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI node-app-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx from instance i-xxxxxxxxxxxxxxxxx
    amazon-ebs: AMI: ami-xxxxxxxxxxxxxxxxx
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build ‘amazon-ebs‘ finished after 4 minutes 7 seconds.

==> Wait completed after 4 minutes 7 seconds

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-xxxxxxxxxxxxxxxxx

Step 5: Launch and Test

Let‘s launch an EC2 instance using our new custom AMI and verify everything is working. Head over to the AWS Management Console, navigate to EC2, and launch a new instance.

Under "Application and OS Images", select "My AMIs" and choose the AMI we just created. The rest of the settings can be left at their defaults.

Once the instance is running, grab its public IP address and open it in a browser. You should see the sample Node.js application!

Best Practices

As you start incorporating Packer and IaC into your workflow, keep these tips in mind:

• Protect your credentials: Use Packer‘s support for reading variables from environment variables or encrypted files. Avoid committing plaintext credentials to version control.

• Keep your images slim: Start with a minimal base image and only install what‘s necessary. Smaller images are faster to build and deploy.

• Use variables and functions: Packer supports variables and functions for runtime parameters and reusability across templates.

• Leverage multi-platform builds: Take advantage of Packer‘s ability to generate images for multiple platforms in parallel.

• Integrate with your CI/CD pipeline: Automatically build and test your images as part of your Continuous Integration workflow for faster feedback.

Conclusion

In this guide, we explored the key concepts behind Infrastructure as Code and walked through a concrete example of using Packer to build an immutable custom machine image for AWS.

By adopting IaC and image building practices, you can make your infrastructure more reliable, consistent, and scalable. No more delicate snowflake servers to maintain by hand!

I encourage you to try building your own custom images and experimenting with more advanced techniques like:

• Running configuration management tools during the image build
• Applying custom tags to your AMIs
• Setting up automated AMI pruning to retire old images

The AWS and Packer documentation are great resources to dive even deeper. You may also want to look into tools like Terraform for provisioning the infrastructure around your images.

Now go forth and build some awesome immutable infrastructure! The cloud is the limit.