Python Requests – How to Interact with Web Services using Python

In today‘s interconnected software landscape, it‘s hard to find an application that doesn‘t interact with a web service or API in some way. As a full-stack developer, I spend a significant portion of my time working with APIs – designing, building, integrating, and consuming them. Python, with its rich ecosystem of libraries, provides powerful tools for working with web services. In this in-depth guide, we‘ll dive into using the requests library – the de facto standard for making HTTP requests in Python.

The Rise of APIs

Before we jump into the code, let‘s take a step back and look at the bigger picture. Over the past decade, APIs have seen explosive growth. According to the ProgrammableWeb, the number of APIs in their directory has grown from just over 5,000 in 2012 to more than 22,000 in 2019 – a more than 4x increase.

This growth underscores the critical role that APIs play in modern software development. APIs allow applications to communicate with each other, enabling developers to leverage existing services and data to build new applications quickly and efficiently.

Setting Up

With that context in mind, let‘s get started with the requests library. First, you‘ll need to install it:

pip install requests

Then, import it in your Python script:

import requests

Making HTTP Requests

The requests library provides a separate function for each HTTP method. The most commonly used ones are:

• requests.get() for GET requests
• requests.post() for POST requests
• requests.put() for PUT requests
• requests.patch() for PATCH requests
• requests.delete() for DELETE requests

Let‘s look at each of these in more detail.

GET Requests

GET requests are used to retrieve data from an API. Here‘s an example that gets the current price of Bitcoin from the CoinDesk API:

import requests

response = requests.get(‘https://api.coindesk.com/v1/bpi/currentprice.json‘)

if response.status_code == 200:
    data = response.json()
    print(f"The current price of Bitcoin is: {data[‘bpi‘][‘USD‘][‘rate‘]}")
else:
    print(f"Failed to get price. Status code: {response.status_code}")

POST Requests

POST requests are used to send data to an API. Often, this is used to create a new resource. Here‘s an example that creates a new paste on Pastebin:

import requests

api_key = ‘YOUR_API_KEY‘
data = {
    ‘api_dev_key‘: api_key,
    ‘api_option‘: ‘paste‘,
    ‘api_paste_code‘: ‘print("Hello, world!")‘,
    ‘api_paste_name‘: ‘Hello from Python‘,
    ‘api_paste_format‘: ‘python‘
}

response = requests.post(‘https://pastebin.com/api/api_post.php‘, data=data)

if response.status_code == 200:
    print(f"Paste created at: {response.text}")
else:
    print(f"Failed to create paste. Status code: {response.status_code}")

PUT and PATCH Requests

PUT and PATCH requests are used to update an existing resource. A PUT request replaces the entire resource, while a PATCH request only modifies the provided fields. Here‘s an example that updates a GitHub gist:

import requests

gist_id = ‘YOUR_GIST_ID‘
url = f‘https://api.github.com/gists/{gist_id}‘
headers = {‘Accept‘: ‘application/vnd.github.v3+json‘}
data = {
    ‘description‘: ‘Updated gist description‘,
    ‘files‘: {
        ‘hello.py‘: {
            ‘content‘: ‘print("Hello, World!")‘
        }
    }
}

response = requests.patch(url, headers=headers, json=data)

if response.status_code == 200:
    print("Gist updated successfully")
else:
    print(f"Failed to update gist. Status code: {response.status_code}")

DELETE Requests

DELETE requests are used to delete a resource. Here‘s an example that deletes a GitHub gist:

import requests

gist_id = ‘YOUR_GIST_ID‘
url = f‘https://api.github.com/gists/{gist_id}‘
headers = {‘Accept‘: ‘application/vnd.github.v3+json‘}

response = requests.delete(url, headers=headers)

if response.status_code == 204:
    print("Gist deleted successfully")
else:
    print(f"Failed to delete gist. Status code: {response.status_code}")

Authentication

Many APIs require authentication to control access to their resources. There are several types of authentication commonly used with APIs:

• API keys
• Basic Auth
• Bearer Tokens (JWT)
• OAuth

The requests library provides ways to handle each of these authentication types.

API Keys

API keys are the simplest form of authentication. They‘re typically provided as a query parameter or in a header. Here‘s an example using the NASA API, which requires an API key:

import requests

api_key = ‘YOUR_API_KEY‘
url = f‘https://api.nasa.gov/planetary/apod?api_key={api_key}‘

response = requests.get(url)

if response.status_code == 200:
    print(response.json()[‘explanation‘])
else:
    print(f"Failed to get APOD. Status code: {response.status_code}")

Basic Auth

Basic Auth involves providing a username and password. With requests, you can provide these in a tuple to the auth parameter:

import requests

response = requests.get(‘https://api.github.com/user‘, auth=(‘username‘, ‘password‘))

if response.status_code == 200:
    print(response.json()[‘login‘])
else:
    print(f"Authentication failed. Status code: {response.status_code}")

Bearer Tokens

Bearer tokens, like JWT, are often used for authentication with APIs. They‘re typically included in the Authorization header. Here‘s an example:

import requests

token = ‘YOUR_TOKEN‘
headers = {‘Authorization‘: f‘Bearer {token}‘}

response = requests.get(‘https://api.example.com/user‘, headers=headers)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Authentication failed. Status code: {response.status_code}")

OAuth

OAuth is a more complex authentication protocol that involves multiple steps. The requests_oauthlib library provides tools for handling the OAuth flow. Here‘s a simplified example:

from requests_oauthlib import OAuth2Session

client_id = ‘YOUR_CLIENT_ID‘
client_secret = ‘YOUR_CLIENT_SECRET‘
redirect_uri = ‘YOUR_REDIRECT_URI‘

oauth = OAuth2Session(client_id, redirect_uri=redirect_uri)
authorization_url, _ = oauth.authorization_url(‘https://provider.com/oauth2/authorize‘)

print(f‘Please go here and authorize: {authorization_url}‘)

# Get the authorization verifier code from the callback url
redirect_response = input(‘Paste the full redirect URL here:‘)

# Fetch the access token
token = oauth.fetch_token(
    ‘https://provider.com/oauth2/token‘,
    client_secret=client_secret,
    authorization_response=redirect_response
)

# Fetch a protected resource
protected_url = ‘https://provider.com/user‘
response = oauth.get(protected_url)

if response.status_code == 200:
    print(response.json())
else:
    print(f"Failed to get resource. Status code: {response.status_code}")

Handling Errors

Handling errors is a critical part of working with APIs. The requests library provides several ways to handle errors.

The response object has a status_code attribute which returns an HTTP status code. You can use this to check if a request was successful. Here are the general guidelines:

• 2xx status codes indicate success
• 4xx status codes indicate a client error (e.g., bad request, unauthorized, not found)
• 5xx status codes indicate a server error

Here‘s a table of some common status codes:

Status Code	Meaning
200	OK
201	Created
204	No Content
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
500	Internal Server Error

You can use conditionals to check the status code and handle errors accordingly:

response = requests.get(‘https://api.github.com/user‘)

if response.status_code == 200:
    print(‘Request successful‘)
elif response.status_code == 404:
    print(‘Resource not found‘)
else:
    print(f"Request failed with status code: {response.status_code}")

You can also use the raise_for_status() method, which will raise an exception if the request was unsuccessful:

try:
    response = requests.get(‘https://api.github.com/user‘)
    response.raise_for_status()
except requests.exceptions.HTTPError as err:
    print(f"HTTP error occurred: {err}")
except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")

Advanced Features

The requests library provides many advanced features to help with more complex use cases.

Sessions

If you‘re making several requests to the same API, you can use a Session object. This allows you to persist certain parameters across requests (like authentication tokens), and also provides performance benefits by reusing the underlying TCP connection.

import requests

session = requests.Session()
session.auth = (‘username‘, ‘password‘)

response = session.get(‘https://api.github.com/user‘)
print(response.json())

Timeouts

You can use the timeout parameter to set a time limit for a request. If the request takes longer than the timeout value, an exception will be raised.

try:
    response = requests.get(‘https://api.github.com/user‘, timeout=1)
except requests.exceptions.Timeout:
    print(‘The request timed out‘)

Posting Form Data and Files

You can use the data parameter to send form-encoded data in a POST request:

data = {‘key1‘: ‘value1‘, ‘key2‘: ‘value2‘}
response = requests.post(‘https://httpbin.org/post‘, data=data)

To send files, you can use the files parameter:

files = {‘file‘: open(‘report.pdf‘, ‘rb‘)}
response = requests.post(‘https://httpbin.org/post‘, files=files)

Best Practices

Here are some best practices I‘ve learned from my experience as a full-stack developer working with APIs:

1. Always check status codes and handle errors. Don‘t assume a request was successful. Check the status code and handle errors gracefully.

2. Use environment variables for secrets. Never hard-code sensitive data like API keys, tokens, or passwords in your code. Instead, use environment variables.

3. Use a timeout. Always set a timeout for your requests. This prevents your program from hanging indefinitely if the API server is unresponsive.

4. Respect rate limits. Many APIs have rate limits. Respect these limits to avoid getting your requests blocked.

5. Use a Session for multiple requests. If you‘re making multiple requests to the same API, use a Session object to persist parameters and improve performance.

6. Use the appropriate HTTP method. Use GET for retrieving data, POST for creating data, PUT for replacing data, PATCH for modifying data, and DELETE for deleting data.

7. Validate responses. Don‘t assume the API always returns the data in the format you expect. Validate the response before using it in your application.

8. Log requests and responses. Logging your API interactions can be invaluable for debugging and monitoring.

9. Version your API. If you‘re building an API, include a version number in the URL. This allows you to update your API without breaking existing clients.

10. Provide good documentation. If you‘re providing an API, make sure to provide clear, comprehensive documentation. Include details about authentication, rate limits, endpoints, request and response formats, and error codes.

Conclusion

In this comprehensive guide, we‘ve covered everything you need to know to start interacting with web services using Python‘s requests library. We‘ve looked at how to make different types of HTTP requests, handle authentication, deal with errors, and use some of the library‘s more advanced features.

Remember, the requests library is a powerful tool, but it‘s just one part of interacting with APIs. To be an effective API developer, you also need to understand HTTP, REST principles, authentication mechanisms, and API design best practices.

APIs are the backbone of modern web development, and Python with requests provides a powerful and intuitive way to interact with them. I hope this guide has given you the knowledge and confidence to start integrating APIs into your own projects.

Happy coding!