How to Use Puppeteer With Node.js: The Ultimate Guide

As a full-stack developer, being able to automate repetitive web browser tasks like scraping, taking screenshots, and running tests can save you countless hours. Puppeteer is a powerful Node.js library that makes browser automation a breeze. In this comprehensive guide, I‘ll teach you everything you need to know to start using Puppeteer with Node.js like a pro.

What is Puppeteer?

Puppeteer is a Node.js library developed by Google that provides a high-level API for controlling a headless Chrome or Chromium browser. In other words, it allows you to automate web browser interactions programmatically using JavaScript. Some common use cases for Puppeteer include:

• Web scraping – extracting data from websites
• UI testing – automating form submission, clicking buttons, taking screenshots
• Generating PDFs of web pages
• Crawling sites to test performance or find broken links

Puppeteer works by communicating with a real browser instance under the hood. By default, it runs the browser in "headless" mode, meaning the GUI is not shown. But you can also configure it to run a full "headful" browser that you can watch execute your scripts.

Setting Up a Node.js Project With Puppeteer

Before we dive into examples, let‘s go through the process of setting up a new Node.js project with Puppeteer installed. I‘ll assume you already have Node.js and npm on your system.

1. Create a new directory for the project and navigate into it:

   mkdir puppeteer-example
   cd puppeteer-example

2. Initialize a new Node.js project and install Puppeteer:

   npm init -y
   npm install puppeteer

3. Create a new file called example.js and add this boilerplate code:

const puppeteer = require(‘puppeteer‘);

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Puppeteer script will go here

  await browser.close();
})();

This script launches a new browser instance, opens a new page, and then closes the browser at the end. The async keyword and parentheses around the immediately-invoked function are required to use await with Puppeteer methods.

Basic Usage and Common Methods

Now that we have a basic project set up, let‘s go over some of the most important Puppeteer methods you‘ll be using:

• page.goto(url) – navigate to a URL
• page.type(selector, text) – type into an input element
• page.click(selector) – click an element
• page.waitForSelector(selector) – wait for element to appear in page
• page.screenshot({ path: ‘image.png‘ }) – take screenshot of page
• page.pdf({ path: ‘page.pdf‘ }) – generate PDF of page
• page.evaluate(() => { ... }) – run code in page context
• browser.close() – close the browser

These are just a handful of available methods, but they cover the majority of use cases. Refer to the Puppeteer API docs for the full list.

Let‘s demonstrate a few of these with a simple example. Add this code to your example.js file:

await page.goto(‘https://duckduckgo.com‘);
await page.type(‘#search_form_input_homepage‘, ‘Puppeteer‘);
await page.click(‘#search_button_homepage‘);
await page.waitForSelector(‘.results‘);
await page.screenshot({ path: ‘duckduckgo.png‘ });

This navigates to DuckDuckGo, searches for "Puppeteer", waits for the results to load, and then takes a screenshot. Run the script with:

node example.js

You should see a new file duckduckgo.png with the screenshot.

Automating Web Scraping

A common application of Puppeteer is extracting structured data from web pages, also known as web scraping. Let‘s walk through an example of using Puppeteer to scrape top news stories from a news website.

const puppeteer = require(‘puppeteer‘);

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(‘https://www.cnn.com/‘);

  const stories = await page.evaluate(() => {
    let results = [];
    let items = document.querySelectorAll(‘h3.cd__headline a‘);

    items.forEach(item => {
       results.push({
         url: item.getAttribute(‘href‘),
         headline: item.innerText
       });
    });

    return results;
  });

  console.log(stories);

  await browser.close();
})();

Here‘s what‘s happening:

1. We navigate to the CNN home page
2. We use page.evaluate to run code in the context of the page. This allows us to query DOM elements with document.querySelectorAll.
3. We extract the href and innerText of each <a> element contained within an <h3> with a class of cd__headline. This targets the story headlines.
4. We build up an array of story objects with URL and headline, and return it.
5. Outside of page.evaluate, we log out the extracted stories array.

When you run this script, you should see an array of top news stories logged to the console.

Keep in mind, web scraping should be done respectfully and in accordance with a site‘s robots.txt rules. Don‘t overwhelm sites with automated requests, or you could get blocked.

Automating UI Testing

Another useful application of Puppeteer is UI testing – writing scripts that automatically interact with your web app to verify functionality. Let‘s go through a simple example of testing login functionality.

const assert = require(‘assert‘);
const puppeteer = require(‘puppeteer‘);

const user = {
  email: ‘[email protected]‘,
  password: ‘password123‘
};

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.goto(‘https://my-site.com/login‘);

  await page.type(‘#email‘, user.email);
  await page.type(‘#password‘, user.password);  

  await page.click(‘#login-btn‘);

  await page.waitForSelector(‘.dashboard‘);

  const url = await page.url();

  assert(url === ‘https://my-site.com/dashboard‘);

  console.log(‘Login test passed!‘);

  await browser.close();
})();

This test does the following:

1. Launches the browser in "headful" mode so we can watch it run
2. Navigates to the login page URL
3. Fills in the email and password fields
4. Clicks the login button
5. Waits for an element only shown on the dashboard page
6. Asserts the URL has changed to the dashboard URL after logging in
7. Logs a success message if the assertion passes

You can expand this example to handle cases like incorrect passwords, disabled accounts, etc. Puppeteer‘s API provides all the basic functionality you need to thoroughly test your app‘s UI.

Best Practices and Common Pitfalls

Here are some tips I‘ve learned to make using Puppeteer easier and avoid common issues:

• Use waitForSelector or waitForNavigation to avoid flakiness caused by elements not loading in time. Puppeteer methods are asynchronous and return Promises.

• If your script throws an error, Puppeteer may leave the browser hanging without exiting. Use browser.close() in a finally block or process event handler to clean up.

• Set a default viewport size using page.setViewport so your script runs consistently across different environments.

• Configure a launchTimeout and timeout in the puppeteer.launch options if your scripts are hanging indefinitely. This sets an upper limit on how long Puppeteer will wait to launch the browser or for a script to complete.

• Use page.evaluate to run any code that interacts with the page‘s DOM. You can pass arguments to the function, but they must be JSON-serializable.

• Need to sign into a site? You can use page.cookies to set pre-defined cookies, page.authenticate for HTTP authentication, or log in manually if security settings allow.

Limitations and Alternatives

While Puppeteer is extremely versatile, it does have some limitations:

• It only supports Chrome/Chromium browsers, so you can‘t test other browsers like Firefox or Safari directly.

• Headless mode can make it difficult to debug scripts, since you can‘t watch what the browser is doing. You can use slowMo mode to slow down execution.

• Since it‘s based on a real browser, Puppeteer can consume significant system resources and slow down under heavy load. Be cognizant of how many pages/tabs you leave open.

• Websites using advanced anti-bot techniques may detect and block Puppeteer traffic due to missing header and cookie data that browsers typically send.

Some alternatives to Puppeteer include:

• Cypress – another popular Node.js-based browser automation tool focused on front-end testing
• Playwright – a newer cross-browser automation library from Microsoft supporting Chromium, Firefox, and WebKit
• Selenium – the original web automation tool supporting many languages and browsers

Conclusion

With its intuitive API and well-documented examples, Puppeteer makes browser automation with Node.js easy. Whether you‘re scraping websites, testing your UI, or generating PDFs, Puppeteer can help eliminate tedious manual work from your development workflow. I hope this guide has given you all the knowledge you need to start using Puppeteer with confidence!