How to Run Commands on Multiple Linux Hosts Using PSSH

Introduction

As a full-stack developer and devops engineer, a significant part of your time is spent managing the infrastructure that powers your applications. Whether you‘re deploying microservices across a Kubernetes cluster, updating configurations on a fleet of web servers, or running ad-hoc data analysis jobs on dozens of Hadoop nodes, the ability to efficiently execute commands and scripts on multiple hosts is essential.

This is where PSSH (Parallel SSH) comes in. PSSH is a powerful tool that allows you to run commands and copy files across large numbers of hosts in parallel, greatly speeding up common system administration and orchestration tasks. In this comprehensive guide, we‘ll dive deep into PSSH and explore how it can be used to streamline your infrastructure management workflows.

Installing PSSH

PSSH is available in the standard repositories for most major Linux distributions. To install on Ubuntu or Debian:

sudo apt update && sudo apt install pssh

For CentOS, Fedora, and other Red Hat-based distributions:

sudo yum install pssh

MacOS users can install via Homebrew:

brew install pssh

Basic PSSH Usage

Specifying Remote Hosts

The core PSSH utilities (parallel-ssh, parallel-scp, parallel-rsync, etc.) all operate on a common set of remote hosts. These are specified in a plaintext "hosts" file, with one hostname or IP address per line:

web1.example.com
web2.example.com 
192.168.1.101
192.168.1.102

You can specify a different hosts file with the -h flag:

parallel-ssh -i -h myhosts.txt uptime

Running Commands with parallel-ssh

The parallel-ssh command allows you to execute a command on all hosts in parallel. The basic syntax is:

parallel-ssh -i -h hosts_file command

For example, to check the uptime on all hosts:

$ parallel-ssh -i -h hosts uptime
[1] 17:32:00 [SUCCESS] web1.example.com 22  up 2 days,  4:38,  0 users,  load average: 0.00, 0.01, 0.05        
[2] 17:32:00 [SUCCESS] web2.example.com 22  up 14 days,  3:57,  0 users,  load average: 0.08, 0.03, 0.01        
[3] 17:32:00 [SUCCESS] 192.168.1.101 22  up 7 days,  6:30,  0 users,  load average: 0.00, 0.00, 0.00 
[4] 17:32:00 [SUCCESS] 192.168.1.102 22  up 7 days,  6:30,  0 users,  load average: 0.15, 0.04, 0.01

Some useful parallel-ssh options:

• -i: Interactive mode, prints output as commands finish
• -t: Set a timeout for commands
• -p: Adjust number of parallel connections (default is 32)
• -A: Enable SSH agent forwarding
• -x: Extra SSH arguments, e.g. -x "-o StrictHostKeyChecking=no"

Copying Files with parallel-scp

parallel-scp allows efficiently copying files to multiple hosts, similar to scp:

parallel-scp -h hosts local_file remote_destination

For example, to copy a configuration file to all web servers:

parallel-scp -h webhosts nginx.conf /etc/nginx/

Retrieving Files with parallel-slurp

parallel-slurp retrieves files from remote hosts to a local directory:

parallel-slurp -h hosts -L local_dir remote_file

To fetch all web server logs into timestamped directories:

parallel-slurp -h webhosts -L webhosts_logs_$(date +%F) /var/log/nginx/access.log 

Killing Processes with parallel-nuke

As the name implies, parallel-nuke can quickly terminate processes across hosts:

parallel-nuke -h hosts process_name

For example, to restart Nginx on all web servers:

parallel-nuke -h webhosts nginx
parallel-ssh -i -h hosts sudo systemctl start nginx

Orchestrating Application Deployments with PSSH

While ad-hoc command execution is useful, PSSH really shines when used to orchestrate complex operations like application deployments. Let‘s walk through an example of deploying a new version of a web application across a fleet of servers.

Example: Rolling Update of a Web Application

Consider a typical 3-tier web application with Nginx web servers, Java application servers, and a MySQL database. Our goal is to deploy a new version of the application with zero downtime using a rolling update strategy.

# Fetch application bundle
wget -O app.tar.gz http://build-server/app/app-v2.3.1.tar.gz

# Distribute to all app servers 
parallel-scp -h apphosts app.tar.gz /tmp/

# Stop application on 50% of servers
parallel-ssh -i -h apphosts[0:50%] ‘
  sudo systemctl stop myapp
  rm -rf /opt/myapp/*
  tar xf /tmp/app.tar.gz -C /opt/myapp 
  sudo systemctl start myapp
‘

# Check health of updated servers
# If healthy, proceed with remaining servers
# Else abort deployment, rollback to previous version

parallel-ssh -i -h apphosts[50%:] ‘
  sudo systemctl stop myapp
  rm -rf /opt/myapp/*
  tar xf /tmp/app.tar.gz -C /opt/myapp
  sudo systemctl start myapp  
‘

# Final health check of all servers
# Clean up
parallel-ssh -i -h apphosts ‘rm /tmp/app.tar.gz‘

Some key points:

• We fetch the new application bundle once and distribute it to all servers. This avoids each server downloading it independently which could overload the build server.
• Host lists can be sliced with Python-style notation, e.g. apphosts[0:50%] for the first half
• Complex logic can be expressed as a multi-line quoted script to parallel-ssh
• Health checks are crucial. Automate testing the new version on a subset of servers before proceeding with the full deploy

Generating Dynamic Host Inventories

So far we‘ve used static host files, but in cloud environments servers are often ephemeral. You can generate PSSH host files dynamically from cloud APIs or configuration management databases:

# Get all active web servers from AWS API
aws ec2 describe-instances \
  --filters "Name=tag:Role,Values=web" "Name=instance-state-name,Values=running" \
  --query "Reservations[].Instances[].PublicDnsName" --output text > webhosts

# Get database servers from Ansible inventory  
ansible db --list-hosts | tail -n +2 > dbhosts

Debugging and Troubleshooting

Inevitably, things will go wrong in distributed environments. Some debugging tips for PSSH:

• Use -i liberally to see command output interactively
• If a command hangs, try -t to set a timeout and move on
• Check return codes. By default PSSH continues on error (-s aborts on first failure)
• Use -o/-e to log stdout/stderr to per-host files for later analysis
• Send output to Logstash or Splunk for centralized debugging
• Write idempotent scripts that can be safely re-run if needed

PSSH Performance and Scalability

Compared to sequential SSH in a loop, PSSH can yield dramatic speedups by saturating your network bandwidth with parallel connections:

Hosts	Sequential SSH	PSSH
10	25 sec	3 sec
50	2 min 11 sec	8 sec
100	4 min 33 sec	16 sec
500	22 min 42 sec	1 min 18 sec

(Benchmarks from Tecmint)

The optimal value for -p (parallel connections) depends on your network latency and bandwidth. Start around 50-100 and increase until you see diminishing returns.

With careful tuning, PSSH can effectively manage up to low thousands of hosts. Beyond that, consider a tool like Ansible that can further parallelize operations using a fan-out tree of control nodes.

Integrating PSSH into a DevOps Workflow

As a full-stack developer, your goal should be to treat infrastructure as code – define it declaratively, version it alongside your application, and update it automatically as part of your Continuous Integration / Continuous Deployment pipeline.

PSSH is a handy tool for bridging the gap between declarative tools like Terraform and Kubernetes that define your desired infrastructure state, and the imperative actions needed to actually mutate servers, e.g. upgrading packages, running data migrations, etc.

Some tips for integrating PSSH into a modern devops workflow:

• Define your PSSH host inventories in version-controlled Ansible YAML files or Terraform templates
• Wrap common PSSH automations in Makefiles or Fabric scripts
• Invoke PSSH from your CI system (Jenkins, CircleCI, etc.) as deployment steps
• Expose key PSSH commands as chatops actions in Slack
• Create incident runbooks that use PSSH to quickly parallelize diagnostic data collection

Security Best Practices

While PSSH is fundamentally an orchestration tool, it‘s built on SSH, so following SSH security best practices is crucial:

• Enforce strong SSH private key security. Never share keys between users or servers.
• Use SSH agent forwarding (ssh-add; parallel-ssh -A) to avoid storing keys on intermediate hosts
• Segment your network with bastion hosts and only allow SSH from those trusted sources
• Use the principle of least privilege. Restrict PSSH to only the commands and servers needed for each automation.
• Log and audit PSSH usage centrally, tying commands back to responsible users

PSSH Alternatives and Complements

PSSH excels at running simple commands across hosts in parallel. For more complex configuration management and orchestration, consider:

• Ansible: YAML-based config management and orchestration
• Fabric: Pythonic remote execution and deployment
• Chef, Puppet: Ruby-based config management
• SaltStack: Python-based config management and event-driven automation

Key factors to consider when choosing an orchestration tool:

• Language preferences (Bash, Python, Ruby, YAML, etc.)
• Support for your target operating systems
• Scalability and performance
• Community and ecosystem (pre-built modules, playbooks, etc.)
• Learning curve and ease-of-use
• Mature vs. bleeding edge

Often a combination of tools works best. You might use Terraform and Packer to build base VM images, Ansible to declaratively configure servers, and PSSH for ad-hoc operations and reactive maintenance tasks.

Conclusion

In this deep dive, we‘ve seen how PSSH can greatly simplify managing Linux servers at scale. By allowing you to execute commands and copy files across hundreds of servers in seconds, PSSH is a powerful addition to a full-stack developer‘s infrastructure toolbox.

Some key takeaways:

• PSSH is a simple, flexible tool for running SSH commands on many servers
• It excels at gathering facts and metrics, distributing files, and coordinating rolling deployments
• PSSH is easily integrated into CI/CD pipelines and chatops workflows
• For managing more than a few hundred nodes, consider a tool like Ansible
• Always treat infrastructure as code and apply software development best practices

As cloud architectures continue to evolve towards containerized microservices and serverless functions, the role of server orchestration is changing. But the fundamental need to automate, audit, and remediate at scale will always be with us. Mastering tools like PSSH is a core skill for any DevOps practitioner.