Hack and / - Lightning Hacks
Recently, I was working on a remastered Knoppix DVD that I had sent out to a few people. After I had sent out the full remastered DVD, I found out that I needed to change a few small files on the DVD. Even though my home DSL speeds are pretty fast, the upload is still slow enough that it took overnight to transfer the 3GB+ DVD image. I didn't want to go through that again, especially as I had made only minor changes to the DVD.
I knew that binary diff tools existed, but I discovered that not all of them are equal. Some binary diff tools require enough RAM to store multiple copies of the file, which certainly wouldn't work with a 3GB image. Lucky for me, I found rdiff, a tool that works well with large files and doesn't require a lot of RAM. What's better is that rdiff works with any binary—you can use it for any large binary files from DVD images to virtual disks to multimedia files.
rdiff works via a three-stage process. In this example, I have two files, old.iso and new.iso, that have minor differences from each other. For the first stage, you create a signature file that rdiff uses to represent your original file:
$ rdiff signature old.iso old.signature
Now that you have a signature file, use it with rdiff to create a delta file that represents the differences between the old and new files:
$ rdiff delta old.signature new.iso new.delta
This new.delta file is now all that anyone needs to convert old.iso to new.iso. For me, this file ended up being around 150Kb, because I had made only a few changes. The delta file was much simpler to send around than the full image. If you want to test that the delta file will work, first create an md5sum of new.iso:
$ md5sum new.iso
Then, use rdiff to patch the old file with the delta to create the new file. This is the same command that everyone else with the original file will use:
$ rdiff patch odl.iso new.delta newtest.iso
Now that you have newtest.iso, create an md5sum of that file and compare it with the one you made for new.iso:
$ md5sum newtest.iso
As I said before, this method works not only with ISOs, but also with any binary file large or small. It's worth noting that rdiff works with the same binary diff method rsync uses. rdiff just lets you use the algorithm step by step on the command line.
Kyle Rankin is a Senior Systems Administrator in the San Francisco Bay Area and the author of a number of books, including Knoppix Hacks and Ubuntu Hacks for O'Reilly Media. He is currently the president of the North Bay Linux Users' Group.
Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.
Webinar: 8 Signs You’re Beyond Cron
11am CDT, April 29th
|Play for Me, Jarvis||Apr 16, 2015|
|Drupageddon: SQL Injection, Database Abstraction and Hundreds of Thousands of Web Sites||Apr 15, 2015|
|Non-Linux FOSS: .NET?||Apr 13, 2015|
|Designing Foils with XFLR5||Apr 08, 2015|
|diff -u: What's New in Kernel Development||Apr 07, 2015|
- Drupageddon: SQL Injection, Database Abstraction and Hundreds of Thousands of Web Sites
- Play for Me, Jarvis
- Non-Linux FOSS: .NET?
- Designing Foils with XFLR5
- Flexible Access Control with Squid Proxy
- Not So Dynamic Updates
- New Products
- Users, Permissions and Multitenant Sites
- New GeekGuide: Beyond Cron