In mid-August, the first commercially available ZFS cloud replication target became available at rsync.net. Who cares, right? As the service itself states, "If you're not sure what this means, our product is Not For You."
Of course, this product is for someone—and to those would-be users, this really will matter. Fully appreciating the new rsync.net (spoiler alert: it's pretty impressive!) means first having a grasp on basic data transfer technologies. And while ZFS replication techniques are burgeoning today, you must actually begin by examining the technology that ZFS is slowly supplanting.
A love affair with rsync
Further Reading
We look at the amazing features in ZFS and btrfs—and why you need them.
Revisiting a first love of any kind makes for a romantic trip down memory lane, and that's what revisiting
rsync—as in "
rsync.net"—feels like for me. It's hard to write an article that's inevitably going to end up trashing the tool, because I've been wildly in love with it for more than 15 years. Andrew Tridgell (of Samba fame) first announced rsync publicly in June of 1996. He used it for three chapters of his PhD thesis three years later, about the time that I discovered and began enthusiastically using it. For what it's worth, the earliest record of my professional involvement with major open source tools—at least that I've discovered—is
my activity on the rsync mailing list in the early 2000s.
Rsync is a tool for synchronizing folders and/or files from one location to another. Adhering to true Unix design philosophy, it's a simple tool to use. There is no GUI, no wizard, and you can use it for the most basic of tasks without being hindered by its interface. But somewhat rare for any tool, in my experience, rsync is also very elegant. It makes a task which is humanly intuitive seem simple despite being objectively complex. In common use, rsync looks like this:
root@test:~# rsync -ha --progress /source/folder /target/
Invoking this command will make sure that once it's over with, there will be a /target/folder, and it will contain all of the same files that the original /source/folder contains. Simple, right? Since we invoked the argument -a (for archive), the sync will be recursive, the timestamps, ownership, permission, and all other attributes of the files and folders involved will remain unchanged in the target just as they are on the source. Since we invoked -h, we'll get human-readable units (like G, M, and K rather than raw bytes, as appropriate). Progress means we'll get a nice per-file progress bar showing how fast the transfer is going.
So far, this isn't much more than a kinda-nice version of copy. But where it gets interesting is when /target/folder already exists. In that case, rsync will compare each of those files in /source/folder with its counterpart in /target/folder, and it will only update the latter if the source has changed. This keeps everything in the target updated with the least amount of thrashing necessary. This is much cleaner than doing a brute-force copy of everything, changed or not!
It gets even better when you rsync to a remote machine:
root@test:~# rsync -ha --progress /source/folder user@targetmachine:/target/
When rsyncing remotely, rsync still looks over the list of files in the source and target locations, and the tool only messes with files that have changed. It gets even better still—rsync also tokenizes the changed files on each end and then exchanges the tokens to figure out which blocks in the files have changed. Rsync then only moves those individual blocks across the network. (Holy saved bandwidth, Batman!)
You can go further and further down this rabbit hole of "what can rsync do." Inline compression to save even more bandwidth? Check. A daemon on the server end to expose only certain directories or files, require authentication, only allow certain IPs access, or allow read-only access to one group but write access to another? You got it. Running "rsync" without any arguments gets you a "cheat sheet" of valid command line arguments several pages long.
To Windows-only admins whose eyes are glazing over by now: rsync is "kinda like robocopy" in the same way that you might look at a light saber and think it's "kinda like a sword."
If rsync's so great, why is ZFS replication even a thing?
This really is the million dollar question. I hate to admit it, but I'd been using ZFS myself for something like four years before I realized the answer. In order to demonstrate how effective each technology is, let's go to the numbers. I'm using rsync.net's new ZFS replication service on the target end and a Linode VM on the source end. I'm also going to be using my own open source orchestration tool syncoid to greatly simplify the otherwise-tedious process of ZFS replication.
First test: what if we copy 1GB of raw data from Linode to rsync.net? First, let's try it with the old tried and true rsync:
root@rsyncnettest:~# time rsync -ha --progress /test/linodetest/ root@myzfs.rsync.net:/mnt/test/linodetest/ sending incremental file list ./ 1G.bin 1.07G 100% 6.60MB/s 0:02:35 (xfr#1, to-chk=0/2) real 2m36.636s user 0m22.744s sys 0m3.616s
And now, with ZFS send/receive, as orchestrated by syncoid:
root@rsyncnettest:~# time syncoid --compress=none test/linodetest root@myzfs.rsync.net:test/linodetest INFO: Sending oldest full snapshot test/linodetest@1G-clean (~ 1.0 GB) to new target filesystem: 1GB 0:02:32 [6.54MB/s] [=================================================>] 100% INFO: Updating new target filesystem with incremental test/linodetest@1G-clean ... syncoid_rsyncnettest_2015-09-18:17:15:53 (~ 4 KB): 1.52kB 0:00:00 [67.1kB/s] [===================> ] 38% real 2m36.685s user 0m0.244s sys 0m2.548s
Time-wise, there's really not much to look at. Either way, we transfer 1GB of data in two minutes, 36 seconds and change. It is a little interesting to note that rsync ate up 26 seconds of CPU time while ZFS replication used less than three seconds, but still, this race is kind of a snoozefest.
So let's make things more interesting. Now that we have our 1GB of data actually there, what happens if we change it just enough to force a re-synchronization? In order to do so, we'll touch the file, which doesn't do anything but change its timestamp to the current time.
Just like before, we'll start out with rsync:
root@rsyncnettest:/test# touch /test/linodetest/1G.bin root@rsyncnettest:/test# time rsync -ha --progress /test/linodetest/ root@myzfs.rsync.net:/mnt/test/linodetest sending incremental file list 1G.bin 1.07G 100% 160.47MB/s 0:00:06 (xfr#1, to-chk=0/2) real 0m13.248s user 0m6.100s sys 0m0.296s
And now let's try ZFS:
root@rsyncnettest:/test# touch /test/linodetest/1G.bin root@rsyncnettest:/test# time syncoid --compress=none test/linodetest root@myzfs.rsync.net:test/linodetest INFO: Sending incremental test/linodetest@syncoid_rsyncnettest_2015-09-18:16:07:06 ... syncoid_rsyncnettest_2015-09-18:16:07:10 (~ 4 KB): 6.73kB 0:00:00 [ 277kB/s] [==================================================] 149% real 0m1.740s user 0m0.068s sys 0m0.076s
Now things start to get real. Rsync needed 13 seconds to get the job done, while ZFS needed less than two. This problem scales, too. For a touched 8GB file, rsync will take 111.9 seconds to re-synchronize, while ZFS still needs only 1.7.
Touching is not even the worst-case scenario. What if, instead, we move a file from one place to another—or even just rename the folder it's in? For this test, we have synchronized folders containing 8GB of data in /test/linodetest/1. Once we've got that done, we rename /test/linodetest/1 to /test/linodetest/2 and resynchronize. Rsync is up first:
root@rsyncnettest:/test# mv /test/linodetest/1 /test/linodetest/2 root@rsyncnettest:/test# time rsync -ha --progress --delete /test/linodetest/ root@myzfs.rsync.net:/mnt/test/linodetest/ sending incremental file list deleting 1/8G.bin deleting 1/ ./ 2/ 2/8G.bin 8.59G 100% 5.56MB/s 0:24:34 (xfr#1, to-chk=0/3) real 24m39.267s user 3m15.944s sys 0m30.056s
Ouch. What's essentially a subtle change requires nearly half an hour of real time and nearly four minutes of CPU time. But with ZFS...
root@rsyncnettest:/test# mv /test/linodetest/1 /test/linodetest/2 root@rsyncnettest:/test# time syncoid --compress=none test/linodetest root@myzfs.rsync.net:test/linodetest INFO: Sending incremental test/linodetest@syncoid_rsyncnettest_2015-09-18:16:17:29 ... syncoid_rsyncnettest_2015-09-18:16:19:06 (~ 4 KB): 9.41kB 0:00:00 [ 407kB/s] [==================================================] 209% real 0m1.707s user 0m0.072s sys 0m0.024s
Yep—it took the same old 1.7 seconds for ZFS to re-sync, no matter whether we touched a 1GB file, touched an 8GB file, or even moved an 8GB file from one place to another. In the last test, that's almost three full orders of magnitude faster than rsync: 1.7 seconds versus 1,479.3 seconds. Poor rsync never stood a chance.
Listing image by Flickr user: jonel hanopol
OK, ZFS is faster sometimes. Does it matter?
I have to be honest—I feel a little like a monster. Most casual users' experience of rsync will be "it rocks!" and "how could anything be better than this?" But after 15 years of daily use, I knew exactly what rsync's weaknesses were, and I targeted them ruthlessly.
As for ZFS replication's weaknesses, well, it really only has one: you need to be using ZFS on both ends. On the one hand, I think you should already want ZFS on both ends. There's a giant laundry list of features you can only get with a next-generation filesystem. But you could easily find yourself stuck with a lesser filesystem—and if you're stuck, you're stuck. No ZFS, no ZFS replication.
Aside from that, ZFS replication ranges from "just as fast as anything else" to "noticeably faster than anything else" to "sit down, shut up, and hold on." The particular use case that drove me to finally exploring replication—which was much, much more daunting before tools like syncoid automated it—was the replication of VM images.
Virtualization keeps getting more and more prevalent, and VMs mean gigantic single files. rsync has a lot of trouble with these. The tool can save you network bandwidth when synchronizing a huge file with only a few changes, but it can't save you disk bandwidth, since rsync needs to read through and tokenize the entire file on both ends before it can even begin moving data across the wire. This was enough to be painful, even on our little 8GB test file. On a two terabyte VM image, it turns into a complete non-starter. I can (and do!) sync a two terabyte VM image daily (across a 5mbps Internet connection) usually in well under an hour. Rsync would need about seven hours just to tokenize those files before it even began actually synchronizing them... and it would render the entire system practically unusable while it did, since it would be greedily reading from the disks at maximum speed in order to do so.
The moral of the story? Replication definitely matters.
rsync is great, but when you feel the need for (data transfer) speed...
Now that we know what ZFS replication is and why it matters, let's talk about rsync.net. I can't resist poking a little fun, but I like these folks. The no-nonsense "if you don't know what it is, it isn't for you" approach is a little blunt, but it makes a realistic assessment of what they're there for. This is a basic service offering extremely high-quality infrastructure to admins who know what they're doing and want to use standard system tools without getting hamstrung by "friendly" interfaces aimed at Joe and Jane Six-Pack. They've been around since 2001, and they are sporting some pretty big names in their "these are our customers" list, including Disney, ESPN, and 3M.
What you're actually getting with your rsync.net subscription is a big, honking VM with as much space on it as you paid for. You can use that VM as a target for rsync—the basic service they've been offering to the world for fourteen years now—or, now, for ZFS replication. It's kind of a sysadmin's dream. You can install what you want, however you want, without any "helpful" management interface getting in your way. Despite the fact that they'd never heard of my helper application Syncoid, I was able to get its dependencies installed immediately and get right to syncing without any trouble. As a veteran sysadmin... niiiiice.
I had a little more trouble testing their bandwidth, simply because it's hard to get enough bandwidth to really challenge their setup. I spun up a ridiculously expensive Linode instance ($960/mo, thank Ghu for hourly billing!) that claimed to offer 10Gbps outbound bandwidth, but it turned out to be... less impressive. Whether I sent to rsync.net or did a speed test with any speedtest.net provider within 200 miles of the exit point of Linode's network, the results turned out the same—about 57mbps. It's possible that Linode really is offering 10gbps outbound in aggregate but is using traffic-shaping to limit single pipes to what I saw. But I frankly didn't have the time, or the inclination, to test.
Did I mention speedtest.net? Did I mention that rsync.net offers you full root access to your VM and doesn't get in your way? I'm back to my happy place now. A couple of git clones later, I had a working copy of a command-line-only interface to speedtest.net's testing infrastructure on my rsync.net VM, and I could test it that way:
root@3730:/usr/local/bin # python ./speedtest_cli.py Retrieving speedtest.net configuration... Retrieving speedtest.net server list... Testing from Castle Access (69.43.165.28)... Selecting best server based on latency... Hosted by I2B Networks Inc (San Diego, CA) [57.35 km]: 4.423 ms Testing download speed........................................ Download: 501.63 Mbit/s Testing upload speed.................................................. Upload: 241.71 Mbit/s
502mbps down, 242mbps up. Am I limited by rsync.net there, or by the speedtest.net infrastructure? I honestly don't know. Results were pretty similar with several other speedtest.net servers, so these are some pretty reasonable numbers as far as I can tell. The TL;DR is "almost certainly more bandwidth than you can use," and that's good enough for me... particularly considering that my 1TB VM with rsync.net is only $60/mo.
For a backup target, though, network bandwidth isn't the only concern. What about disk bandwidth? Can you write those incoming ZFS snapshots as fast as the network will carry them? In order to find out, I got a little crafty. Since ZFS supports inline compression and I have no way to be sure rsync.net isn't using it where I can't see it, I wrote a few lines of Perl to generate 1GB of incompressible (pseudo-random) data in memory and then write it repeatedly to the disk.
#!/usr/local/bin/perl print "Obtaining 1G of pseudorandom data:\n"; my $G1 = `dd if=/dev/urandom bs=1024M count=1 2>/dev/null`; print "Beginning write test:\n"; my $loop; open FH, "| pv -s 10G > /mnt/test/linodetest/test.bin"; while ($loop <10) { print FH $G1; $loop++; } close FH; Looks good. So let's see what happens when we hammer the disks with a 10G stream of random data:
root@3730:/mnt/test/linodetest # perl ~/test.pl Obtaining 1G of pseudorandom data: Beginning write test: 10GiB 0:00:49 [ 208MiB/s] [==================================================>] 100%
Ultimately, I can receive data from the network at about 60MB/sec, which won't stress my storage since it can write at >200 MB/sec. We're solid... seriously solid. From our perspective as the user, it looks just like we have our own beefy machine with 8GB of RAM, a good CPU, and several high-quality hard drives all to ourselves. With a nice fat several-hundred-mbps pipe. In a datacenter. With support. For $60/mo.
Good
- That price is pretty amazing for what you get—it's about on par with Amazon S3, while offering you any number of things S3 won't and can't give you.
- rsync.net is so simple. If you know what you're doing, it's going to be extremely easy to work with and extremely difficult to compromise—there's no big complex Internet-facing Web interface to get compromised, it's just you, ssh, and the stuff you put in place.
- When vulnerabilities do pop up, they're going to be extremely easy to address. FreeBSD is going to patch them, rsync.net is going to apply them (making the vulnerability window extremely small).
- With no tier 1 support, anybody you talk to is going to be a serious *nix engineer, with serious security policies they understand. The kind of social engineering that owned Mat Honan's iCloud account will be extremely difficult to pull off.
Bad
- Some of the above strengths are also weaknesses. Again, there is no tier 1 support for rsync.net—if you need support, you're going to be talking to a real, no-kidding *nix engineer.
- If you have to use that support, well, it can get frustrating. I did have some back and forth with support while writing this review, and I learned some things. (I wasn't aware of the High Performance Networking fork of SSH until I reached out to rsync.net support during the course of reviewing the service.) Despite the fact that the folks at rsync.net knew I was writing a review and that it might end up on the front page of Ars Technica, it generally took anywhere from several hours to a day to get an e-mail response.
Ugly
- There's nothing else like rsync.net commercially available right now, but this is a pretty specialized service. Neither I nor rsync.net are likely to advocate it as a replacement for things like Dropbox any time soon.
Jim Salter (@jrssnet) is an author, public speaker, small business owner, mercenary sysadmin, and father of three—not necessarily in that order. He got his first real taste of open source by running Apache on his very own dedicated FreeBSD 3.1 server back in 1999, and he's been a fierce advocate of FOSS ever since. He also created and maintains http://freebsdwiki.net and http://ubuntuwiki.net.
In the name of full disclosure, the author developed and maintained the ZFS replication tool referenced above (Syncoid). And for this piece, the shell sessions would become a lot more cumbersome to follow if replication was instead done manually. While there are commercial options, Syncoid is a fully open source, GPL v3.0 licensed tool. The links above lead directly to the Github repo where it can be freely downloaded and used by anyone.
