How to Find and Fix Docker Container Vulnerabilities in 2020

Containers have become the de facto standard for packaging and deploying applications in the cloud-native era. Technologies like Docker and Kubernetes enable organizations to release software faster, more consistently, and at greater scale than ever before.

However, this agility comes at a price. The 2020 Cloud Native Security Report found that 91% of organizations had at least one container with high or critical severity vulnerabilities, while 60% had at least one container that was running as root. The average company had 58 vulnerable images or containers, illustrating just how widespread the issue is.

As a full-stack developer turned DevOps engineer, I‘ve seen firsthand how easily container vulnerabilities can slip through the cracks and put applications and data at risk. In this post, I‘ll share my learnings on the most pervasive Docker security risks and provide practical guidance for detecting and mitigating them.

Anatomy of a Container Vulnerability

Container vulnerabilities can stem from a variety of factors, but most fall into one of three buckets:

1. Vulnerable Packages: The open source software supply chain is rife with known vulnerabilities. According to Snyk‘s 2020 State of Open Source Security report, the top ten most popular Docker images contain more than 1,300 vulnerabilities. When you docker pull an image, you‘re inheriting all of its baggage.

2. Misconfigurations: Many container and orchestrator settings have security implications. For example, running a container as --privileged or with the --cap-add=SYS_ADMIN flag gives it essentially root access to the host. Allowing hostPath volume mounts or hostNetwork mode can also enable container breakouts.

3. Exposed Secrets: Hardcoded secrets like API keys, database credentials, and SSH keys are a common issue in container images. A study by Palo Alto Networks found that 44% of images in public registries contained at least one secret. Leaking secrets can allow an attacker to gain unauthorized access to other systems and data.

To illustrate, here‘s a breakdown of the distribution of container vulnerabilities by type based on data from a large enterprise environment:

Vulnerability Type	% of Total
Vulnerable Packages	68%
Misconfigurations	25%
Exposed Secrets	7%

Source: Internal vulnerability scan data, Q2 2020

As you can see, vulnerable packages are by far the most common issue, followed by misconfigurations. However, exposed secrets can be the most damaging since they often provide a direct path to sensitive data.

Container vs Host Vulnerabilities

One common misconception is that container vulnerabilities are less severe than host-level vulnerabilities since containers run in an isolated sandbox. However, a vulnerability in a container can be just as impactful as one on the host.

Consider the following scenario:

$ docker run -it --rm -v /:/host alpine:latest sh
/ # chroot /host
/ # echo "You‘ve been pwned!" > /etc/motd
/ # exit
$ ssh localhost
You‘ve been pwned!

Here, we‘ve started a container with a --volume mount to the host‘s root filesystem. Using a chroot, we‘ve switched the container‘s filesystem to the host‘s and then modified the SSH login banner. When a user SSH‘s into the host, they‘ll see our message.

This is a contrived example, but it demonstrates how container breakouts are possible with the right mixture of vulnerabilities and misconfigurations. Sophisticated attackers can use techniques like container escape, kernel exploits, and SSH key theft to pivot from a compromised container to the underlying host.

As a result, container vulnerabilities need to be treated with the same level of severity and urgency as host-level issues. Fortunately, there are a number of best practices we can follow to reduce risk.

Detecting Container Vulnerabilities

The first step in managing container risk is establishing visibility. You can‘t protect what you can‘t see. There are several approaches for identifying vulnerabilities in your container environment:

Image Scanning

Image scanning involves analyzing the contents of a container image to identify known vulnerabilities in its OS packages and application dependencies. This can be done:

• Pre-deployment via a CI/CD pipeline step that fails builds of images with fixable vulnerabilities
• At runtime by monitoring registries for new images and scanning them before deployment
• Continuously by scanning already-deployed images on a regular basis to surface newly disclosed vulnerabilities

Some popular open source image scanning tools include:

• Clair – Static analysis tool for detecting known vulnerabilities in appc and Docker containers
• Trivy – Simple and comprehensive vulnerability scanner for containers and other artifacts
• Anchore Engine – Centralized service for inspecting images and applying user-defined acceptance policies

In addition, many container registries like Docker Hub, Google Container Registry, and Amazon ECR, provide built-in image scanning capabilities.

Runtime Monitoring

While image scanning can help prevent vulnerable containers from being deployed, it‘s equally important to monitor the behavior of containers at runtime. This allows you to detect anomalous activity that could indicate an attempted exploit.

Some key signals to watch include:

• Process activity: Monitor for processes spawning shells, making outbound network connections, or accessing sensitive files
• Filesystem changes: Watch for modifications to system binaries, libraries, and configuration files
• Network traffic: Look for unexpected inbound or outbound connections, port scanning, or C2 communication
• Kernel calls: Hook into the kernel to detect attempted container escapes, privilege escalations, or loading of malicious modules

Open source projects like Falco and Sysdig Inspect provide frameworks for defining runtime security rules and alerting on violations. Commercial platforms like Aqua Security, Twistlock, and StackRox offer more advanced capabilities like machine learning and network visualization.

Here‘s an example Falco rule to detect a shell being spawned inside a container:

- rule: Terminal shell in container
  desc: A shell was spawned by a non-shell program in a container with an attached terminal.  
  condition: >    
    spawned_process and container 
    and shell_procs and proc.tty != 0 
    and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal (user=%user.name %container.info
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty)
  priority: WARNING

Penetration Testing

Penetration testing involves proactively trying to exploit vulnerabilities in your container environment. This can help uncover misconfigurations, permissions issues, and weaknesses in network segmentation.

Some common container pen testing techniques include:

• Attempting to break out of a container and access the host filesystem
• Scanning for open ports and unprotected network interfaces
• Fuzzing container runtimes and orchestrators for vulnerabilities
• Testing whether a compromised container can be used to access others on the same host or cluster

Open source tools like kube-hunter and kubeaudit can help automate discovery of misconfigurations and vulnerabilities in Kubernetes clusters. For a deeper dive, check out this presentation on Attacking and Defending Kubernetes from Black Hat 2019.

Remediating Container Vulnerabilities

Once you‘ve identified vulnerabilities in your containers, the next step is figuring out how to fix them. Here are some best practices to consider:

Rebuild Images

The immutable nature of containers means that vulnerabilities in running containers can‘t be patched directly. Instead, you need to rebuild the container image with updated packages and redeploy.

This can be done by:

1. Updating the image‘s Dockerfile to use a version of the base image (e.g. debian:stretch) that contains fixes
2. Rerunning apt-get update && apt-get upgrade in the image build to pull in updates to system packages
3. Updating the versions of any application dependencies installed via a language package manager (e.g. pip, npm, gem)

The updated image should be given a new semantic version tag and pushed to the registry. Then any deployments or services using the image need to be updated to reference the new version.

Automating this rebuild-and-replace loop via a CI/CD pipeline can help ensure images are always up-to-date with the latest security fixes. Anchore‘s inline_scan tool integrates with Jenkins to fail builds that introduce new vulnerabilities.

Apply Principle of Least Privilege

The principle of least privilege dictates that containers should run with the minimum capabilities required for their workload. This reduces the blast radius in case of exploitation.

Some ways to enforce least privilege include:

• Avoiding running containers as root and dropping unneeded capabilities with --cap-drop
• Configuring containers to run as read-only and remount filesystems as read/write only when necessary
• Setting seccomp, AppArmor, and SELinux profiles to restrict available syscalls and accesses
• Implementing network policies to limit ingress/egress traffic and using encryption for all inter-container communication

Docker and Kubernetes provide a number of security contexts and policies for hardening containers. For more, check out the CIS Benchmarks for Docker and Kubernetes.

Rotate Secrets Frequently

Secrets management is a challenging problem for containerized apps since they often need access to sensitive credentials like API keys and database passwords. Embedding secrets in images or passing them via environment variables significantly increases the risk of exposure.

Instead, containers should retrieve secrets at runtime from a secure secrets store like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. These vaults provide features like:

• Secure storage with encryption at rest and in transit
• Automatic rotation of secrets on a scheduled basis
• Dynamic secrets that are issued on-demand and have a limited TTL
• Fine-grained access policies and audit logging

For Kubernetes-based workloads, projects like Sealed Secrets and Kubernetes External Secrets provide a declarative way to manage secrets using native K8s resources.

Use Behavioral Analytics

In addition to detecting known vulnerabilities, it‘s important to monitor for unknown threats. This requires establishing a baseline of normal container behavior and alerting on deviations.

Machine learning can be used to profile container behavior across multiple dimensions like:

• Process execution: Unusual process names, arguments, or execution frequencies
• File access: Anomalous reads/writes of sensitive files and directories
• Network connections: Unexpected network flows between containers or to external IPs
• Resource utilization: Spikes in CPU, memory, or I/O that could indicate cryptomining or DoS attempts

Security platforms like Sysdig Secure and Aqua Dynamic Threat Analysis use ML to build behavioral models for each container and alert on abnormal activity in real-time. Catching "unknown unknowns" is critical for detecting zero-day exploits and insider threats.

Implement Defense-in-Depth

Relying on a single control is a recipe for disaster. Effective container security requires a layered, defense-in-depth approach.

In addition to the techniques described above, organizations should consider:

• Segregating environments: Use separate clusters and networks for dev, test, and prod to limit blast radius
• Implementing admission controllers: Enforce security policies on Kubernetes API objects in-line with the control plane
• Enabling audit logging: Record all container-related events and API calls to a secure, centralized log store for forensics
• Securing the software supply chain: Digitally sign container images and use a policy engine to ensure only trusted images are deployed

The NIST Application Container Security Guide provides a comprehensive reference on container security best practices.

Looking Ahead: The Future of Container Security

Container security is a fast-moving space. Here are some emerging trends and technologies shaping the future:

Cloud-Native Application Protection Platforms (CNAPP)

There‘s a growing convergence of technologies for securing containerized apps across the full development lifecycle. This includes tools for vulnerability scanning, compliance, runtime protection, and network security.

Gartner has coined the term Cloud-Native Application Protection Platform (CNAPP) to describe these integrated platforms. Leading vendors include Palo Alto Networks (Prisma Cloud), Trend Micro (Cloud One), Aqua Security, Sysdig, and StackRox.

eBPF

Extended Berkeley Packet Filter (eBPF) is a revolutionary technology that allows running custom code in the Linux kernel without changing kernel source code or loading modules.

For security, eBPF offers a powerful way to instrument running containers and collect granular telemetry with minimal overhead. For example, an eBPF program could trace all file accesses, network connections, and process executions made by a container.

The Cilium project uses eBPF to provide Kubernetes network security and observability. Tracee is an eBPF-based runtime security tool that detects abnormal behavior using kernel-level events.

Here‘s a simple eBPF program in Python to detect execution of a specific syscall (e.g. exec):

from bcc import BPF
bpf_text = """
int kprobe__sys_execve(struct pt_regs *ctx, const char __user *filename) {
    bpf_trace_printk("Exec called: %s\\n", filename);
    return 0;
};
"""
b = BPF(text=bpf_text)
b.trace_print()

Confidential Computing

Confidential computing is an emerging paradigm that uses hardware-based trusted execution environments (TEEs) to protect data in use.

TEEs like Intel SGX, AMD SEV, and AWS Nitro Enclaves create an encrypted memory region that is inaccessible to the host OS and hypervisor. This allows running sensitive workloads in the cloud without exposing data to the underlying infrastructure.

Confidential computing can help mitigate the impact of container escape vulnerabilities by preventing access to memory outside the TEE. The Asylo framework provides tools for running containers inside Intel SGX enclaves.

Conclusion

Container security is a complex but critical discipline. As a full-stack developer, it‘s important to understand the unique risks posed by containers and how to mitigate them through a combination of proactive engineering and reactive detection and response.

This post covered the most common types of container vulnerabilities, techniques for identifying them, and best practices for remediation. It also explored some cutting-edge technologies shaping the future of container security.

While there‘s no silver bullet, following the guidance outlined here can help reduce your container attack surface and keep your applications safe. Container security is a shared responsibility—by working together, developers, operations, and security teams can reap the benefits of containerization without putting the organization at risk.