How to Set Up Application Performance Monitoring for Node.js Applications Using Elastic

As Node.js continues to grow in popularity for building scalable server-side applications, monitoring the performance and health of these applications becomes increasingly critical. Without proper monitoring in place, Node.js developers can find themselves flying blind – unaware of errors, performance bottlenecks, or other issues impacting their applications and end users.

This is where application performance monitoring (APM) solutions like Elastic APM come into play. Elastic APM provides deep visibility into Node.js applications, allowing developers to identify and troubleshoot performance issues, errors, and anomalies in real-time.

In this article, we‘ll walk through how to set up Elastic APM to monitor a Node.js application from end to end. By the end, you‘ll have a robust monitoring solution in place to ensure your Node.js applications are always available, fast, and error-free. Let‘s get started!

Why Monitor Node.js Application Performance?

Before we dive into the technical details, let‘s briefly discuss why monitoring Node.js application performance is so crucial. There are several key reasons:

1. Ensure a positive user experience – Performance issues like slow response times or errors can quickly lead to frustration for end users. Monitoring allows you to identify and resolve these issues before they significantly impact the user experience.

2. Identify performance bottlenecks – Elastic APM provides granular data on which parts of your application are taking the longest or consuming the most resources. With this data, you can optimize the performance of your Node.js applications.

3. Resolve errors quickly – APM solutions automatically capture errors and provide stack traces pinpointing where the error occurred. This accelerates troubleshooting and ensures errors are fixed promptly.

4. Capacity planning – By monitoring resource utilization and traffic patterns over time, you can better plan your infrastructure capacity to meet demand.

5. Monitoring service level agreements (SLAs) – For those providing Node.js applications or services to customers, monitoring is key to ensuring SLAs around performance, uptime, etc. are met.

Now that we understand the importance, let‘s look at how Elastic APM works and how to set it up for Node.js.

Overview of Elastic APM

Elastic APM is a fully-featured application performance monitoring solution provided by Elastic, the company behind the popular Elasticsearch, Logstash, and Kibana (ELK) stack. Some of the key capabilities of Elastic APM include:

• Distributed tracing to track requests across services
• Code-level performance insights and bottleneck detection
• Error tracking and logging with stack traces
• Application metrics like throughput, response times, error rates, etc.
• Server/infrastructure metrics like CPU, memory usage, etc.
• Deep frontend (real user monitoring) and backend visibility
• Customizable dashboards for visualizing data
• Alerting on performance, error, and infrastructure issues

Elastic APM has agents available for most popular languages and frameworks, including Node.js and Express.js. The basic architecture consists of three main components:

1. APM Agents – Lightweight plugins that instrument your application code to collect performance data, errors, etc. The Node.js APM agent is installed as a dependency in your application.

2. APM Server – A separate component that receives data from the agents and processes it for storage in Elasticsearch. The APM Server is typically run alongside your other Elastic Stack components.

3. Elasticsearch and Kibana – The APM data is stored and indexed in Elasticsearch, allowing it to be queried and visualized using Kibana. This provides an intuitive, real-time view into your application performance.

With this high-level understanding in place, let‘s walk through the process of setting up Elastic APM for a Node.js application.

Setting Up Elastic APM for Node.js

There are three main steps to setting up Elastic APM for a Node.js application:

1. Set up an Elastic Stack environment with the APM Server
2. Install and configure the Elastic APM Node.js agent
3. Instrument your Node.js application code

Let‘s walk through each of these steps in detail.

Step 1: Set Up an Elastic Stack Environment with APM Server

The first step is setting up an Elastic Stack environment that includes Elasticsearch, Kibana, and the APM Server. You can install these components in a few different ways:

• Manually on one or more servers
• Using Docker containers
• Using a cloud-managed Elastic Stack service

For simplicity, we‘ll use a Docker-based setup in this tutorial. Assuming you have Docker and docker-compose installed, create a new directory for the project and add the following docker-compose.yml file:

version: ‘2.2‘
services:
  apm-server:
    image: docker.elastic.co/apm/apm-server:7.8.0
    depends_on:
      - elasticsearch
    cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
    cap_drop: ["ALL"]
    ports:
    - 8200:8200
    networks:
    - elastic
    command: >
       apm-server -e
         -E output.elasticsearch.hosts=["elasticsearch:9200"]

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    environment:
    - bootstrap.memory_lock=true
    - discovery.type=single-node
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
    - esdata:/usr/share/elasticsearch/data
    ports:
    - 9200:9200
    networks:
    - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:7.8.0
    depends_on:
      - elasticsearch
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
    - 5601:5601
    networks:
    - elastic

volumes:
  esdata:
    driver: local

networks:
  elastic:
    driver: bridge

This file defines services for Elasticsearch, Kibana, and the APM Server, all running on a shared Docker network. To start the environment, run:

docker-compose up

Once the services are up and running, you should be able to access Kibana at http://localhost:5601 and the APM Server at http://localhost:8200.

Step 2: Install and Configure the Elastic APM Node.js Agent

With the Elastic Stack environment in place, the next step is installing the Elastic APM Node.js agent in your application. You can install it using npm:

npm install elastic-apm-node --save

Next, require and configure the agent at the very top of your application‘s entry point (before any other require statements):

// Add this to the VERY top of the first file loaded in your app
const apm = require(‘elastic-apm-node‘).start({
  // Override service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: ‘‘,

  // Use if APM Server requires a token
  secretToken: ‘‘,

  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: ‘‘,
})

The three most important configuration options are:

• serviceName – The name of your application/service
• secretToken – Token for authenticating with the APM Server (if required)
• serverUrl – URL of your APM server

There are many other configuration options available – see the elastic-apm-node documentation for full details.

Step 3: Instrument Your Node.js Application Code

With the Node.js agent installed and configured, the last step is instrumenting your application code to send data to the APM Server. The simplest way to do this is using the elastic-apm-node API.

Here‘s an example of how you might instrument an Express.js application:

const express = require(‘express‘)
const app = express()

// Add this to the VERY top of the first file loaded in your app
const apm = require(‘elastic-apm-node‘).start({
  serviceName: ‘my-express-app‘,
  serverUrl: ‘http://localhost:8200‘,
})

app.get(‘/‘, function (req, res) {    
  // Log a custom event
  apm.captureError(     
    ‘Custom event successfully logged: ‘ + Date.now(),
    ‘log-my-custom-event‘   
  )

  res.send(‘Hello World!‘)
})

app.get(‘/api/foo‘, function (req, res) { 
  // Record a custom transaction
  const transaction = apm.startTransaction(‘GET /api/foo‘)

  // Wrap a MySQL query with a custom span
  const span = transaction.startSpan(‘MySQL SELECT x FROM y‘)
  mysqlClient.query(‘SELECT x FROM y‘, (err, result) => { 
    if (err) {
      // Record the error against the current transaction
      transaction.captureError(err)
    }
    else {
      // Update the span with response data
      span.end({ result: result.rows })  
    }

    // End the transaction
    transaction.end()
  })
})

app.listen(3000)

This example demonstrates a few key concepts:

• apm.captureError() allows you to log a custom error or event
• apm.startTransaction() allows you to record a custom transaction (e.g. a specific route or operation)
• transaction.startSpan() allows you to record a specific operation (e.g. a database query) within a transaction
• transaction.end() finalizes the transaction and sends the data to the APM server
• Errors can be automatically recorded using transaction.captureError()

With your app instrumented, start it up and generate some traffic by hitting the endpoints. After a minute or two, you should start seeing data appearing in Kibana under the "APM" tab.

Monitoring Node.js Applications with Elastic APM

Once you have data flowing into Elastic APM, you can take advantage of its powerful features for monitoring the performance and health of your Node.js applications. Some key areas to monitor include:

• Transaction response times – Look for slow transactions or sudden increases in latency. Elastic APM automatically categorizes transactions by URL, so it‘s easy to identify problem areas.

• Throughput – Monitor the number of requests per minute/second to understand traffic patterns and identify potential bottlenecks.

• Error rates – Keep an eye out for spikes in error rates, which could indicate a major issue. You can drill into individual errors to get stack traces and other details to help with debugging.

• App and server metrics – Track metrics like CPU usage, memory usage, and garbage collection stats to get a sense of how hard your servers are working.

• Slow database queries – Use database query spans to surface slow SQL queries for further optimization.

Elastic APM provides customizable dashboards in Kibana, allowing you to visualize and analyze all of this data in one place. You can also set up alerts to notify you when certain thresholds are crossed (e.g. error rate > 5%, response time > 2s, etc.)

Best Practices for Monitoring Node.js Applications with Elastic APM

To get the most value out of Elastic APM for monitoring your Node.js applications, follow these best practices:

1. Use descriptive names for custom transactions and spans – This makes your data easier to navigate and understand in Kibana.

2. Capture errors with context – Rather than just capturing the error message, include additional metadata about what the user was doing, system state, etc.

3. Monitor all layers of your stack – Don‘t just instrument your application code, but also any databases, caches, or external services it depends on. Elastic APM has integrations for most popular technologies.

4. Establish performance baselines – Know what "normal" looks like for your application in terms of response times, error rates, etc. This makes it easier to spot anomalies.

5. Set up alerts for key metrics – Don‘t just rely on manual monitoring. Set up automated alerts to notify you when something is wrong.

6. Use APM data to drive optimizations – Regularly review your APM data to identify opportunities to optimize performance, reduce errors, etc.

By following these practices and leveraging Elastic APM‘s powerful features, you can ensure your Node.js applications are consistently fast, stable, and error-free.

Conclusion

Monitoring the performance and health of Node.js applications is critical for ensuring a positive user experience and quickly resolving any issues. Elastic APM provides a comprehensive solution for end-to-end monitoring of Node.js apps, from tracking individual requests to surfacing high-level trends and anomalies.

In this article, we‘ve walked through how to set up Elastic APM for a Node.js application, including:

1. Setting up an Elastic Stack environment with the APM Server
2. Installing and configuring the Node.js APM Agent
3. Instrumenting a Node.js app to send custom transactions, spans, errors, and metrics

We‘ve also covered some best practices for effectively monitoring Node.js applications using Elastic APM.

While we‘ve focused on Node.js in this article, it‘s worth noting that Elastic APM supports a wide variety of other languages and frameworks as well. So regardless of what your stack looks like, Elastic APM likely has you covered.

If you‘re not already monitoring your Node.js applications with Elastic APM, I‘d highly encourage you to give it a try. With a relatively small amount of setup, you can gain deep visibility into your Node.js applications that will help you ensure a consistently excellent user experience.