An Introduction to Logging for Programmers

Logging is one of the most important yet often overlooked aspects of building robust software systems. When done well, logging provides invaluable diagnostic information for troubleshooting issues, auditing user actions, and gaining visibility into an application‘s behavior. However, many developers treat logging as an afterthought, littering their code with ad hoc print statements.

In this guide, we‘ll cover the fundamentals of logging that every programmer should know. You‘ll learn what logging is, why it‘s critical, best practices and frameworks to use, and see real-world examples. By the end, you‘ll be equipped to implement effective logging in your own projects.

What is Logging?

At its core, logging is the practice of recording information about an application‘s execution. A log message might capture things like:

• Notable events that occur (e.g. a user logging in)
• Detailed debugging information (function parameters, return values, etc.)
• Error conditions and stack traces
• Performance metrics and timings
• External service call requests/responses

Log data is typically stored in a structured format like JSON and aggregated in a centralized logging system for persistence and analysis. This information serves as an audit trail that can answer key questions like:

• What was the application doing at a given time?
• What sequence of events led up to an error?
• How is the application performing over time?
• What actions are users taking within the app?

Why Logging Matters

Logging is a critical practice for several reasons:

1. Debugging and Troubleshooting – Logs provide a detailed record that makes it much easier to diagnose issues. Rather than debugging blind, a developer can consult the logs to trace an error back to its source.

2. Auditing and Compliance – In industries like healthcare and finance, audit logging is essential for tracking critical actions for security and regulatory requirements. Logs establish an immutable activity record.

3. Monitoring and Alerting – Logs provide valuable telemetry that can give insight into application health and detect anomalies. Spikes in errors, traffic patterns, and performance metrics captured in logs serve as the basis for setting up alerts.

4. Business Intelligence – User activity logs can help answer questions like: Who are the most active users? What features are people engaging with? This informs product decisions.

Despite these benefits, a 2020 survey conducted by Scalyr found that 96% of developers rely solely on manual searches through their logs to troubleshoot issues. Only 4% are using log management tools to their full potential.

Types of Logging

There are two broad categories of logging:

1. Diagnostic Logging – These are logs meant to aid in debugging and tracing execution flow. They tend to be more low-level and technical in nature. Examples:

   • Error stack traces
   • Function input/output values
   • Timestamps for measuring durations
   • System resource utilization metrics

2. Audit Logging – Audit logs capture meaningful business events and focus less on the technical details. The goal is establishing a record of "who did what when." Examples:

   • User authentication attempts (logins/logouts)
   • Admin privileged actions
   • Key transactions (e.g. orders, payments)
   • Data access/modification
   • Configuration changes

Structured Logging Formats

Traditionally, logs were written as unstructured text data. An error message might look like:

[2022-04-15 09:14:27] ERROR CommonTools.scala:38 - Parsing failed due to malformed JSON 

While a human can read this, it‘s difficult for computers to reliably extract fields for analysis. That‘s where structured formats like JSON come in:

{
  "timestamp": "2022-04-15T09:14:27Z",
  "level": "ERROR",
  "file": "CommonTools.scala",
  "line": 38, 
  "message": "Parsing failed due to malformed JSON"
}

With JSON, each bit of data is organized into key-value pairs. This consistent schema allows for easy querying, filtering, and aggregation. You can answer questions like:

• How many errors of type X happened in the past hour?
• What‘s the 95th percentile response time for endpoint Y?
• Show me a histogram of log events over time grouped by level.

Tools like ElasticSearch, Splunk, and Datadog have made JSON the defacto standard for log ingestion.

Logging Frameworks

While you can certainly roll your own logging implementation, it‘s advisable to leverage an existing framework. These will provide features like log levels, formatters, handlers, and appenders out of the box. Some popular options:

• Winston (Node.js)
• Bunyan (Node.js)
• Log4j (Java)
• Logback (Java)
• Serilog (.NET)
• Python Logging
• Monolog (PHP)

Logging frameworks allow for granular control over how and what gets logged. You can specify a destination (e.g. write to console vs file vs remote server), format, log level, and even attach additional metadata.

Here‘s a simple example using Python‘s built-in logging module:

import logging

# Basic configuration  
logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)

# Example messages  
logger.debug("This is a debug message")
logger.info("This is an informational message")
logger.warning("This is a warning")
logger.error("This is an error message")

When run, this outputs:

DEBUG:__main__:This is a debug message
INFO:__main__:This is an informational message  
WARNING:__main__:This is a warning
ERROR:__main__:This is an error message

Logging Best Practices

Here are some tips for effective logging:

1. Use Log Levels Judiciously – Most frameworks define several log levels (debug, info, warning, error, fatal). Use them consistently to convey severity. Avoid overusing debug logs in production.

2. Be Concise Yet Descriptive – Log messages should be brief yet informative. Include context like function names, line numbers, and specific error descriptions when applicable.

3. Leverage Structured Data – As previously mentioned, use a structured format like JSON. Include relevant event metadata in each log entry.

4. Don‘t Log Sensitive Data – Be mindful not to log passwords, access tokens, PII, or other sensitive information.

5. Use Unique Identifiers – When logging related events, include a unique ID that allows for easy correlation. This is critical for distributed tracing.

6. Implement Log Rotation – Logs can quickly consume disk space if not controlled. Use an automated rotation scheme to compress and archive old logs.

7. Monitor and Alert on Logs – Effective monitoring means proactively identifying issues from logs before they impact users. Set up alerts for high error rates, spikes in traffic, and other key events.

Real-World Examples

Let‘s look at a few scenarios that demonstrate the value of good logging practices:

Example 1: Diagnosing a Memory Leak

Suppose an application‘s performance has degraded over time, with users reporting slow response times. Without logging, a developer would be hard pressed to pinpoint the issue.

However, if the code includes memory usage logging, the logs could reveal a clue:

{"timestamp": "2022-04-01T10:32:11Z", "level": "INFO", "message": "Memory usage: 150MB"}
{"timestamp": "2022-04-02T11:14:53Z", "level": "INFO", "message": "Memory usage: 1.5GB"}
{"timestamp": "2022-04-03T09:46:32Z", "level": "WARNING", "message": "Memory usage: 6GB"}

The logs show memory usage growing unbounded over time, a telltale sign of a memory leak. Armed with this information, the developer can focus their troubleshooting on memory allocation code.

Example 2: Tracing a Failed Payment

An e-commerce company relies heavily on audit logging to diagnose payment failures. When a customer complains their order wasn‘t processed, support can consult the logs to see what went wrong:

{"timestamp": "2022-04-04T14:11:39Z", "userId": 12345, "action": "InitiateCheckout", "orderId": "abc123"}
{"timestamp": "2022-04-04T14:12:01Z", "userId": 12345, "action": "SubmitPayment", "orderId": "abc123", "paymentId": "xyz456", "amount": 99.99}
{"timestamp": "2022-04-04T14:12:03Z", "level": "ERROR", "message": "PaymentGatewayError - Connection timed out", "orderId": "abc123", "paymentId": "xyz456"}

Here the audit trail shows the user initiated checkout and submitted payment, but the third log indicates the payment failed due to a gateway timeout error. Support can refund the customer and notify engineering of the integration issue.

Example 3: Detecting Suspicious Activity

A financial application uses auth access logs to detect fraudulent activity:

{"timestamp": "2022-04-05T08:30:14Z", "userId": 56789, "event": "login_succeeded"}
{"timestamp": "2022-04-05T08:32:47Z", "userId": 56789, "event": "sensitive_data_accessed"}  
{"timestamp": "2022-04-05T08:33:01Z", "userId": 56789, "event": "login_succeeded", "ipAddress": "200.140.250.114"}
{"timestamp": "2022-04-05T08:33:53Z", "userId": 56789, "event": "high_value_transfer_created", "amount": 100000}  

These logs capture a sequence of suspicious actions from a user:

1. A login from the user‘s typical device
2. Sensitive data accessed, indicating the session was hijacked
3. A new login from an unrecognized foreign IP address
4. An attempt to create a large money transfer

Security teams can use algorithms and machine learning to detect anomalous patterns like this across terabytes of log data. Logging provides the foundation for proactive threat detection.

Conclusion

We‘ve covered the importance of logging, different types of logs, best practices, and real-world scenarios that highlight its utility. Logging is a powerful tool, but it requires a thoughtful approach to yield actionable insights.

When implemented properly, it provides an essential feedback loop – logs capture an application‘s behavior, developers gain visibility to make improvements, and the cycle continues. Without logging, you‘re flying blind.

As the scale and complexity of software systems grows, logging has never been more critical. By embracing a culture of disciplined logging, your organization will reap the benefits of more stable, performant, and secure applications. The next time you‘re writing code, remember to log early and log often!