How to Automate REST API End-to-End Tests in a CI Environment with Postman and Newman

REST APIs power the modern web. As a developer, you will likely spend a lot of time integrating with upstream APIs or providing your own APIs for web, mobile, and third-party clients to consume. An API is a contract – you expect certain request parameters and headers, and in return, promise specific response schemas and behaviors. Breaking this contract can lead to bugs, errors and ultimately, unhappy users and lost business.

This is why testing your REST APIs is absolutely critical, and should be an integral part of your development workflow. In this guide, I‘ll show you how to use Postman, a popular tool for exploring and testing APIs, together with Newman, Postman‘s command-line test runner, to build automated end-to-end API tests that run in a Continuous Integration (CI) environment.

The Case for Automated REST API Testing

Before we dive into the how, let‘s talk about the why. Why should you invest time in writing end-to-end API tests, especially if you‘re already doing unit and integration testing?

First, end-to-end tests validate that the API behaves as expected from the client‘s perspective. It treats the entire API as a black box and doesn‘t care about the implementation details. This is important because refactoring your API‘s underlying code shouldn‘t break clients relying on the existing contract.

Second, automating these tests in a CI pipeline acts as a safeguard against regressions. Every time you commit code or deploy to staging/production, your end-to-end tests will run and immediately inform you if something broke the API contract. It‘s much better to catch bugs early than to have a customer find them for you!

Lastly, a comprehensive suite of API tests can serve as living documentation. Instead of wiki pages that go out of date, your tests are always up-to-date with the actual behavior of the API. Developers and even non-technical stakeholders can refer to the tests to understand what the API does.

Postman Primer

Postman is an API platform with a sleek user interface that allows developers to design, debug, test, document, and publish APIs all in one place. One of its most popular features is the ability to send HTTP requests to an API endpoint and inspect the response.

Beyond one-off requests, Postman lets you save related requests together into a Collection. For example, you might have a "Users API" collection with requests to create, read, update, and delete users.

Collections can be further organized into Folders. Folders are not only great for grouping related requests, but also enable you to define setup and teardown scripts that run before and after each request in the folder.

Another key Postman concept is Environments. An environment is a set of variables (key-value pairs) that allow you to parameterize your requests. Instead of hardcoding values like API host, API key, access token, etc. in the request, you reference the variable name, for example {{apiKey}}. This makes it easy to switch between different environments like local, development, staging, and production.

Saving requests to a collection and parameterizing with environment variables sets a great foundation for API testing, which we‘ll explore next.

Newman – Postman for CI Environments

While Postman is great for manually exploring APIs, at some point you‘ll want to automate your tests and integrate them into your CI pipeline. That‘s where Newman comes in. Newman is Postman‘s command-line Collection Runner. It allows you to run API tests directly from the command-line, and integrates nicely with build systems and CI servers.

Newman is distributed as an npm package, so assuming you have Node.js installed, you can simply run:

npm install -g newman

To execute a collection with Newman, you need two things:

1. The Collection JSON file exported from Postman
2. An environment JSON file with variable definitions

Here‘s an example command to run a collection against a local environment:

newman run mycollection.json -e local_env.json

Newman will send each request in the collection, run any tests, and report back the results. You can also generate an HTML report and export environment variables using additional command-line flags:

newman run mycollection.json -e local_env.json -r html --reporter-html-export results.html --export-environment outenv.json

With just these simple commands, you now have a repeatable way to fully test your API from end-to-end, which is a huge step towards automating quality assurance!

Writing Effective API Tests

Of course, running a collection is only useful if it contains well-designed test cases. Postman provides a pm.test() function that allows you to write test specifications inside the Tests tab of a request.

The test function takes two arguments – a string for the test name, and a function that returns a boolean indicating if the test passed or failed. You can write assertions using ChaiJS BDD syntax, which provides an expressive language for defining expected outcomes.

Some common REST API assertions include:

• Asserting specific status codes like 200 OK or 201 Created
• Verifying the presence or absence of certain response headers
• Checking the response time is below some threshold
• Validating the response JSON schema
• Verifying the response body contains expected values

Let‘s say you have an API endpoint GET /users/:id to fetch a user by ID. Here are some test cases you might write:

// Test for 200 OK status code
pm.test("Status code is 200", function () {
  pm.response.to.have.status(200);
});
// Test the response contains the expected user email 
pm.test("Response contains user email", function () {
var jsonData = pm.response.json();
pm.expect(jsonData.email).to.eql("[email protected]");
});

With tests in place, Newman will run them for each request and report successes and failures. This is immensely valuable for catching API bugs and regressions before they make it to production!

Building Request Workflows

Many REST APIs have requests that rely on data from previous requests. For example, you may need to first authenticate and get an access token before you can hit protected endpoints. Or you may have an endpoint to create a resource that returns an ID, which you then use to retrieve, update, or delete that resource instance in subsequent requests.

Postman provides a few ways to chain requests together into workflows:

1. Extracting response data into environment variables. You can use the pm.environment.set() function to save data from a response, then reference it in later requests. This is useful for things like access tokens and resource IDs.

// Extract the access token and save to an environment variable 
var jsonData = pm.response.json();
pm.environment.set("accessToken", jsonData.token);

2. Using postman.setNextRequest() to dynamically determine the next request to execute. For example, you could have a request that fetches a list of resources, then loops through and executes a request for each resource.

// Get the list of resource IDs from the response
var jsonData = pm.response.json();
var resourceIds = jsonData.map(resource => resource.id);
// Execute the next request for each resource ID
if (resourceIds.length > 0) {
pm.environment.set("resourceId", resourceIds.shift());
postman.setNextRequest("Get Resource by ID");
} 

3. Setting up Pre-request Script on a Folder. Any code in this script will execute before every request in the folder. This is a good place to set initial variable values or sign requests.

By effectively leveraging these techniques, you can model complete user flows and scenarios, providing even greater API test coverage.

Integrating Newman into a CI Pipeline

Now that you have a Postman Collection with solid API tests, it‘s time to integrate it into your CI pipeline. The specifics will depend on your CI server, but the general steps are:

1. Export your Collection and Environment JSON from Postman
2. Commit the JSON files to your code repository
3. Install Newman in your CI environment (e.g. a build agent or Docker image)
4. Add a build step that runs Newman with your Collection and Environment files
5. Configure Newman to generate JUnit XML or HTML reports
6. Save the reports as build artifacts and/or send notifications on failures

For example, here‘s what a Newman step might look like in a Jenkins pipeline:

stage(‘API Tests‘) {
  agent {
    docker { 
      image ‘postman/newman‘
    }
  }
  steps {
    sh ‘newman run mycollection.json -e ci_env.json -r junit --reporter-junit-export results.xml‘
  }
  post {
    always {
      junit ‘results.xml‘
    }
  }
}

With this setup, your Newman tests will run on every build, asserting the functionality and performance of your API. If there are any failures, the build will be marked as failed and your team will be notified to investigate and resolve the issue.

Newman Test Reports

Newman can output test reports in several formats, but two of the most useful are:

1. HTML – A visually rich, interactive report great for sharing with non-technical people. It includes request and response details, test assertions, and even graphs and timelines.

2. XML (JUnit format) – A machine-readable format that integrates with most CI servers‘ built-in test reporting and visualization tools.

You can generate either or both of these report types by including the -r flag and specifying the reporter name and output file.

newman run mycollection.json -e env.json -r html,junit --reporter-html-export results.html --reporter-junit-export results.xml

In a CI environment, archiving the HTML report as a build artifact makes it easy to review test results, especially if tests failed. The JUnit XML output can be fed into your CI server‘s test reporting system, allowing you to see trends over time, identify flaky tests, and even integrate with dashboards and monitoring tools.

Best Practices and Pro Tips

Here are a few tips I‘ve learned from using Postman and Newman extensively to test REST APIs:

• Keep sensitive data out of collections. Don‘t store secrets like API keys or passwords in your Collection or Environment files, as these often get committed to source control. Instead, use variables and set the actual values through environment variables or secret management tools in your CI system.
• Keep collections and environments DRY. Use variables liberally to avoid duplicating values across many requests. Also consider using a separate environment file for each stage (dev, staging, prod) with the appropriate config values, instead of separate collections entirely.
• Use folders to organize requests and write setup/teardown scripts. A bit of organization goes a long way, especially as your API surface grows. Folders provide a natural way to group related endpoints. Adding Pre-request Script to a folder to set variables or initialize state can greatly reduce redundancy.
• Write tests for both happy and unhappy paths. It‘s important to not only test that APIs return expected payloads with valid inputs, but also properly handle things like missing parameters, invalid auth, and edge cases.
• Use Collection Runner to debug flaky tests locally. Before committing collections, use Postman‘s built-in Collection Runner to execute the entire suite multiple times locally. This can help identify any flaky or order-dependent tests that may fail in CI.

Alternatives and Complementary Approaches

Postman + Newman is a powerful combo for automating API tests, but it‘s not the only option. Other alternatives worth considering:

• Swagger / OpenAPI – If you already have a Swagger spec for your API, there are many open source tools that can generate tests from the spec and execute them in a CI pipeline. The upside is less manual test writing, but the downside is less flexibility and customization.
• Gatling / Apache JMeter – These are popular open-source load testing tools. While more focused on performance than functionality, you can write API test scenarios and assertions. They can complement Newman by handling the heavy lifting of simulating many concurrent users.
• Cypress / Playwright – These are end-to-end testing frameworks that run in an actual browser. While typically used for UI testing, they can also send HTTP requests and process responses. The main benefit over API-only tools like Postman is the ability to test the full stack in a single flow.

Conclusion

REST APIs are central to modern web and mobile apps. Testing them thoroughly is critical for delivering high-quality, reliable software. Postman is already in most developers‘ toolbelts for manually exploring and debugging APIs. By leveraging Newman, you can take those same Postman Collections and integrate them into an automated CI pipeline.

With a well-designed collection, you can achieve extensive test coverage of your API‘s functionality, performance, and error handling. You can build complete flows that mirror how real clients and users will interact with your API. And most importantly, you can catch errors and breaking changes before they ever reach end-users.

API testing is a complex topic, and we‘ve really only scratched the surface in this guide. However, I hope it has provided a solid foundation to begin your API testing journey with Postman and Newman. The investment in building automated API tests will pay dividends in more reliable software that your users can depend on.