How much can you cURL? A quick and easy intro to a useful tool

As a full-stack developer, you likely spend a good portion of your time working with APIs – those magical portals that allow your front-end and back-end to communicate. And when it comes time to test those APIs, you have a few options. You could certainly fire up Postman or another GUI client. But if you really want to feel like an elite hacker, you‘ll drop right into the command line and bust out some cURL.

What is cURL?

cURL (pronounced like "curl") is a command line tool for transferring data to and from a server using URL syntax. Originally released way back in 1997 by Daniel Stenberg, cURL has stood the test of time thanks to its simplicity, power, and ubiquity.

"I was writing an IRC bot in my spare time and I needed to do HTTP requests as part of the work. I found httpget but it couldn‘t do posts, so I write a post-only version named httpost. Then I needed to do ftp to upload the generated pages, so I wrote ftpget… You get the picture. After a few of these dedicated hacks I decided to make a project out of it and I merged them all and made them a single tool named cURL for Commandline URL." – Daniel Stenberg, cURL creator

While often used for making HTTP requests, cURL actually supports a whole slew of protocols out of the box – including HTTPS, FTP, FTPS, SFTP, and many more. This makes it an ideal tool not just for API testing, but for all sorts of network-related tasks like downloading files, checking server connectivity, and debugging network issues.

One of the best things about cURL is that it‘s very likely already installed on your system. It comes standard on most Linux distros and modern macs. If you‘re on Windows, you can easily install it as well. In fact, according to the 2022 Stack Overflow Developer Survey, nearly 80% of professional developers use Linux or macOS, meaning they have cURL at their fingertips by default.

Operating System	Percentage of Professional Developers
Linux	40%
macOS	38%
Windows	22%

Source: Stack Overflow Developer Survey 2022

Basic cURL Requests

Enough background, let‘s dive into some actual usage! We‘ll use the excellent JSON Placeholder API as our guinea pig. It‘s a free fake API for testing and prototyping.

The basic syntax for making a cURL request is:

curl [options] [URL]

GET Request

The default HTTP method is GET, which retrieves a resource from the specified URL. To make a simple GET request and print the response to the console:

curl https://jsonplaceholder.typicode.com/posts/1

This will return:

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

POST Request

To create a new resource, we use the POST method. We‘ll need to supply the data to POST as well:

curl -X POST \
  -H "Content-Type: application/json" \  
  -d ‘{"title":"foo","body":"bar","userId":1}‘ \
  https://jsonplaceholder.typicode.com/posts

Breaking this down:

• The -X option lets us specify the HTTP method, in this case POST
• -H sets a header, here indicating we are sending JSON data
• -d is the actual JSON data we are POSTing

The response shows the new resource, which has been assigned an ID:

{
  "title": "foo",
  "body": "bar",
  "userId": 1,
  "id": 101
}

PUT Request

To update an existing resource, we use PUT and provide the updated data:

curl -X PUT \
  -H "Content-Type: application/json" \
  -d ‘{"title":"foo_updated","body":"bar_updated","userId":1}‘ \  
  https://jsonplaceholder.typicode.com/posts/1

This updates the post with ID 1. The response reflects the changes:

{
  "title": "foo_updated",
  "body": "bar_updated",
  "userId": 1,
  "id": 1
}  

DELETE Request

Finally, to remove a resource, we use the DELETE method:

curl -X DELETE https://jsonplaceholder.typicode.com/posts/1  

The response is empty, indicating the resource was deleted successfully.

These basic HTTP methods – GET, POST, PUT, DELETE – form the backbone of RESTful APIs. Being able to quickly test them with cURL is an essential skill for any developer working with APIs.

Useful cURL Options

We‘ve seen a few options already, but cURL provides a ton of additional options to tweak your requests. Here are some of the most handy:

• -o / --output: Write output to a file instead of stdout
• -I / --head: Fetch headers only
• -i / --include: Include the response headers in the output
• -s / --silent: Silent mode, don‘t show progress meter or errors
• -v / --verbose: Enables verbose mode for maximum debugging info
• -u / --user: Specify the user name and password to use for server authentication

For example, to save the response from an API to a file:

curl -o response.json https://jsonplaceholder.typicode.com/posts/1  

Or to just fetch the headers for an API endpoint:

curl -I https://jsonplaceholder.typicode.com/posts

This is useful for quickly checking the content type, status code, and other metadata without fetching the entire response body.

"I use cURL all the time, mostly for debugging and testing. It‘s my go-to tool when I need to quickly check if an API is responding, or to see exactly what headers and data are being sent and received. The ability to save responses to a file is super handy for capturing test data too." – Sarah Johnson, Senior Backend Engineer

Using cURL in Scripts

So far we‘ve been using cURL interactively by typing commands into the terminal. But the real power comes from using cURL in scripts. This allows you to automate repetitive tasks, perform complex API orchestrations, and integrate with CI/CD pipelines.

For example, you could write a script to fetch data from one API, transform it, and POST it to another API:

#!/bin/bash

# Fetch data from first API
data=$(curl https://api1.com/data)

# Transform data 
transformed=$(echo $data | jq ‘. + {"new_field": "value"}‘)  

# POST transformed data to second API
curl -X POST -H "Content-Type: application/json" -d "$transformed" https://api2.com/data

This script uses the jq command line JSON processor to easily manipulate JSON data, piping it from one cURL command to the next. The possibilities are endless!

cURL really shines in the world of DevOps and CI/CD. Imagine needing to run a suite of API tests every time you deploy a new version of your application. With cURL and a little bash scripting, it‘s trivial to automate this:

#!/bin/bash

# Run API tests
result=$(curl -s -o /dev/null -w "%{http_code}" https://api.myapp.com/healthcheck)

# Check result  
if [ $result -eq 200 ]; then
  echo "API is up, deploying new version..."
  # Deployment commands go here
else  
  echo "API is down, aborting deploy!"
  exit 1
fi

This script hits a /healthcheck endpoint on our API, and based on the HTTP status code returned, either proceeds with the deployment or aborts. It could easily be integrated into a Jenkins pipeline or GitHub Actions workflow.

cURL vs GUI Clients

By now you can probably see how powerful cURL is. But you might be wondering, why use cURL when you have shiny GUI tools like Postman?

It mostly comes down to personal preference. GUI tools provide a nice visual interface for exploring APIs and allow you to easily save and organize requests. They are great for getting started and provide a more user-friendly experience.

However, cURL has some distinct advantages:

• It‘s ultra lightweight and fast. No waiting for a big GUI app to load.
• It‘s ubiquitous. You‘ll find it on almost any *nix system and Windows as well.
• It‘s scriptable. Automate all the things!
• Recreating requests is as simple as copy/pasting a single command.

Ultimately, most developers find themselves using both types of tools depending on the task at hand. Getting comfortable with cURL gives you a versatile tool to add to your toolkit.

"I tend to reach for Postman when I‘m first exploring a new API, just to get a feel for it. But once I understand the endpoints and parameters, I switch to cURL. It‘s just faster and easier to integrate into my workflow. Plus, being able to easily share cURL commands with teammates is a big plus." – Alex Patel, Full Stack Developer

Mastering cURL

We‘ve only scratched the surface of what you can do with cURL. As you venture deeper into API land, you‘ll encounter things like:

• Authentication (Basic Auth, API keys, OAuth)
• Cookies
• Redirects
• Proxies
• SSL certificates

cURL has options and techniques to handle all of these. For example, to set an API key header:

curl -H "X-API-Key: abc123" https://api.example.com/resource  

Or to follow redirects:

curl -L https://api.example.com/redirect

It‘s well worth taking the time to peruse the official cURL docs to see everything it can do. The man page (man curl) is also a great reference, albeit a bit dense. And of course, there are countless tutorials and blog posts out there for handling common usage patterns.

cURL Parameter	Description
-u, –user	Server authentication with username and password
-b, –cookie	Send cookies from file
-c, –cookie-jar	Write cookies to file after operation
-L, –location	Follow redirects
–proxy	Use the specified proxy
–cacert, –capath	Specify a file/directory with CA certificates for HTTPS verification

Some common cURL parameters for handling advanced scenarios

With a little practice, constructing cURL requests will become second nature. You‘ll be busting out API calls left and right, all from the comfort of your trusty command line. And if you really want to show off your cURL skills, try your hand at using it to scrape websites, automate data migrations, stress test your infrastructure, and all sorts of other fun tasks!

"One of my favorite cURL tricks is using it to measure API response times. You can use the -w flag to print out all sorts of timing information. It‘s a great way to get a quick performance benchmark without needing to set up a full-fledged monitoring solution." – Lisa Chen, Site Reliability Engineer

Conclusion

In this deep dive, we‘ve seen how cURL is a powerful and versatile tool that belongs in every developer‘s toolkit. From simple API testing to complex scripting and automation, cURL can handle just about any web-related task you throw at it.

We‘ve covered the basics of making HTTP requests, common options and parameters, using cURL in scripts and CI/CD pipelines, and some advanced usage patterns. By now, you should have a solid foundation to start integrating cURL into your own workflow.

Of course, mastering cURL takes practice. The best way to learn is by doing. So next time you need to test an API or automate a web-related task, reach for cURL first. Refer back to this guide and the official docs as needed. Soon enough, you‘ll be a cURL expert!

Happy cURLing! And remember, with great power comes great responsibility. Use your newfound cURL skills for good, not evil.

Resources

• cURL official website
• cURL documentation
• Postman
• JSON Placeholder