RSync Examples – Rsync Options and How to Copy Files Over SSH

If you‘re a full-stack developer or DevOps engineer, mastering efficient file transfer between local and remote systems is a core skill. One of the most powerful tools in your arsenal for this task is rsync.

In this deep dive guide, we‘ll explore rsync from a technical perspective, looking at how it works under the hood, advanced command line options, and how to effectively leverage rsync in your development workflows. Whether you‘re a rsync veteran or just getting started, by the end of this article you‘ll have expert-level understanding of this essential utility.

How Rsync Works: A Technical Deep Dive

At a high level, rsync is a utility that efficiently syncs files between a source and a destination, either locally or to/from a remote system over SSH. But how does it actually work?

The secret to rsync‘s performance lies in its delta transfer algorithm. Instead of naively copying entire files, rsync employs a sophisticated diffing algorithm to determine which parts of a file have changed, and then only transmits those deltas.

The rsync algorithm[^1] works like this:

1. The sender splits the file into fixed-size blocks and computes rolling checksums for each block.
2. The sender sends these checksums to the receiver.
3. The receiver searches its version of the file for any blocks that match those checksums.
4. The receiver sends back a list of the blocks it needs, using the rolling checksums to identify them.
5. The sender sends over those requested blocks.
6. The receiver reconstructs the file using its existing blocks plus the delta blocks from the sender.

By only transmitting the differences, rsync can dramatically reduce the amount of data sent over the network. This makes it significantly faster than simpler tools like scp or cp, especially over slower network connections.

But don‘t just take my word for it. Let‘s look at some hard data. In a 2020 benchmark comparing file transfer tools[^2], rsync consistently outperformed scp, especially for larger file sizes:

File Size	rsync Time (s)	scp Time (s)	Speedup
100MB	1.2	5.6	4.7x
500MB	5.8	28.1	4.8x
1GB	11.5	56.2	4.9x

As you can see, rsync is nearly 5 times faster than scp for gigabyte-scale transfers. That speedup only becomes more pronounced over slower or higher latency network links.

Rsync in the Dev Workflow

So rsync is fast – but why should you as a developer care? The answer is that mastering rsync can significantly streamline your development workflow when working with remote servers or synchronizing files between multiple machines.

Here are a few examples of where rsync shines:

• Deploying code to staging/production servers
• Keeping dev environments in sync across multiple machines
• Performing backups of critical data
• Mirroring websites or repositories to redundant servers

For instance, let‘s say you‘re a full-stack dev working on a large web application. You have a local development environment, a staging server for testing, and a production deployment. With rsync, you can easily keep all three environments in sync with commands like:

# Deploy local code changes to staging
rsync -avz --delete /path/to/local/code/ [email protected]:/path/to/code/

# Pull production data to local for testing 
rsync -avz --progress [email protected]:/path/to/data/ /path/to/local/data/

By scripting these kinds of rsync commands, you can create smooth development workflows that eliminate manual error-prone steps. As Charity Majors, co-founder of Honeycomb.io, puts it: "Rsync is an amazing power tool for syncing data and code around. It‘s fast, it‘s efficient, and it‘s saved my bacon more times than I can count."[^3]

Advanced Rsync Command Line Options

While the basic syntax of rsync is straightforward, it‘s real power comes from its plethora of command line flags that modify its behavior. We already discussed some essentials like -a for archive mode and -z for compression – but there are dozens more to master.

Here are a few of my favorite advanced rsync options:

• --dry-run: Performs a trial run without actually transferring any files. Useful for testing commands before committing.

rsync -avz --dry-run /src/dir/ /dst/dir/

• --exclude-from: Specify a file containing exclude patterns, one per line. Helps keep your rsync command concise.

rsync -avz --exclude-from ‘exclude-list.txt‘ /src/dir/ /dst/dir/

• --link-dest: Enables incremental backups by hardlinking unchanged files to an existing backup directory[^4].

rsync -avz --link-dest=/path/to/prev/backup /src/dir/ /new/backup/

• --bwlimit: Limits the transfer bandwidth, helping throttle rsync on network links that can‘t handle full speed.

rsync -avz --bwlimit=1m /src/dir [email protected]:/dst/dir

The rsync man page[^5] contains an exhaustive list of options – I encourage you to read through it and experiment.

Securing Rsync Transfers

One key thing to keep in mind when using rsync, especially over networks, is security. By default, rsync uses SSH for its transport, which provides robust encryption. However, there are additional steps you can take to harden your rsync transfers:

1. Use SSH key pairs instead of passwords for authentication. This prevents password sniffing and brute force attacks.

2. Lock down the SSH server config on the receiving side to restrict rsync access. For instance, use the command option in authorized_keys to only allow a specific rsync command[^6].

3. Run rsync as a lower-privileged user where possible, using --chown to set file ownership as needed. This limits damage if a server is compromised.

4. Enable SSH logging and monitoring to detect any unusual rsync activity. Tools like fail2ban can automatically block suspicious IP addresses.

By following these security best practices, you can leverage rsync to its fullest while minimizing risk.

Rsync in the DevOps World

Rsync is not just a standalone tool – it plays nicely with the wider ecosystem of DevOps utilities. In fact, many higher-level tools use rsync under the hood for efficient file transfer.

For example, the wildly popular configuration management tool Ansible uses rsync to deploy files and templates to managed servers[^7]. The synchronize module in Ansible wraps an rsync call, allowing you to easily incorporate rsync transfers into your configuration playbooks.

Similarly, many Continuous Integration and Continuous Deployment (CI/CD) pipelines take advantage of rsync. Tools like Jenkins, GitLab CI, and CircleCI all support using rsync for efficient deployments[^8].

By understanding the ins and outs of rsync, you‘ll be better equipped to leverage these higher-level DevOps tools effectively. Rsync is a foundational building block for efficient automation.

Conclusion

We‘ve covered a lot of ground in this deep dive into rsync. From the technical details of the delta algorithm, to advanced command line options, to security best practices – you should now have an expert-level grasp on this powerful file syncing utility.

But the learning doesn‘t stop here. Rsync is under active development, with performance improvements and new features added regularly[^9]. I encourage you to stay up to date with the latest rsync releases and experiment with incorporating it into your daily development workflows. The time invested in mastering rsync will pay dividends in speed and efficiency.

As Bram Cohen, the creator of BitTorrent, said: "Few pieces of software are as well-written or useful as rsync. It‘s a pleasure to use a program that does exactly what you want, without a lot of fuss."[^10]

Here‘s to fewer fuss and more efficient file transfers in your stack. Happy syncing!

[^1]: Tridgell, A. & Mackerras, P., "The rsync algorithm", TR-CS-96-05, Australian National University, 1996
[^2]: Terada, M., "Benchmark Analysis of File Transfer Protocols: FTP vs SCP vs RSYNC vs BBCP", Journal of Network and Systems Management, 2020
[^3]: Majors, C., "My Go-To DevOps Tools", Charity.wtf Blog, 2019
[^4]: Sivaraman, R., "Efficient Incremental Backups with Rsync", Linux Journal, 2017
[^5]: "rsync(1) – Linux manual page", https://man7.org/linux/man-pages/man1/rsync.1.html
[^6]: Copeland, B., "Restricting rsync Over SSH", Spin.atomicobject.com Blog, 2012
[^7]: "Ansible synchronize module", https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html
[^8]: "Using Rsync and SSH for Deployments", CircleCI Docs, https://circleci.com/blog/using-rsync-and-ssh-for-deployments/
[^9]: "Rsync ChangeLog", https://download.samba.org/pub/rsync/src/rsync-changelog.gz
[^10]: Cohen, B., "Efficiency Hacks", Bramcohen.com, 2017