Tarsnap: On-line Backups for the Truly Paranoid
Now let's list the archives Tarsnap has stored:
# tarsnap --list-archives docs.20120701 docs.20120702
To demonstrate Tarsnap's smart approach to storage further, I will delete the oldest backup:
# tarsnap df docs.20120701 Total size Compressed size All archives 2132325 1815898 (unique data) 2132325 1815898 This archive 2132325 1815898 Deleted data 445 1037
The "all archives" number has dropped because now I have only one archive, but the "unique data" has not changed much because it is still retaining all files necessary to satisfy my "docs.20120702" archive. If I list it, I can see my data is still there:
# tarsnap tvf docs.20120702 drwxrwxr-x 0 andrew 0 Jun 14 20:52 docs/ -rw------- 0 andrew 48568 Jun 14 16:41 docs/andrew_passwords.psafe3 -rw-rw-r-- 0 andrew 46014 Jun 14 16:35 docs/vacation_notes.doc -rw-rw-r-- 0 andrew 134959 Jun 14 16:44 docs/vacation_reservation.pdf -rw-rw-r-- 0 andrew 48128 Jun 14 16:41 docs/vacation_hotels.doc -rw------- 0 tina 14271 Jun 14 16:42 docs/tina_passwords.psafe3 -rw-rw---- 0 andrew 1833222 Jun 14 16:38 docs/2011 Tax Return.pdf
I use a date string for convenient versioning, but I could just as easily use any naming convention for the archive, such as "docs.1", "docs.2" and so on. For my personal backups, I have a cron job that invokes Tarsnap nightly with a date-string-named archive:
tarsnap cf docs.`date '%+Y%m%d'` /docs
If I have a local calamity and want to restore that data, it is just another simple Tarsnap command to get my files back. Note that like traditional tar, Tarsnap removes the leading slash so all files are restored relative to the current working directory:
# cd / # rm -rf docs # tarsnap xvf docs.20120702 x docs/ x docs/andrew_passwords.psafe3 x docs/vacation_notes.doc x docs/vacation_reservation.pdf x docs/vacation_hotels.doc x docs/tina_passwords.psafe3 x docs/2011 Tax Return.pdf
If you want to run Tarsnap as a nonroot user, create a .tarsnaprc file in your home directory. The syntax is identical to the tarsnap.conf discussed above. For example:
$ cat ~/tarsnap.conf cachedir /home/andrew/tarsnap-cache keyfile /home/andrew/tarsnap.key print-stats
If you have other services or users contending for your Internet
--maxbw-rate to specify a maximum bytes per second
that Tarsnap will be allowed to use.
print-stats command gives you account status information when used
interactively, but for batch operations (such as running Tarsnap out
of cron), you can suppress the output by removing that directive
from your tarsnap.conf or by invoking Tarsnap with
Finally, you can play with the
-v flags to simulate Tarsnap
backup operations without actually burning network and disk. Once you've
got your command constructed exactly as you want it, remove
Tarsnap is not distributed under an open-source license, although all client source is provided (and compiled by the user during install). However, the company regularly contributes back to projects whose code it utilizes, such as libarchive. Tarsnap also has open-sourced some of its own projects, including the scrypt package, the spiped secure pipe dæmon and the kivaloo NoSQL data store.
The Tarsnap home page (http://tarsnap.com) has a wealth of documentation and information, as well as links to the Tarsnap IRC channel, mailing list and FAQ. The "Technical Details" section is absorbing reading for those interested in the deep details of Tarsnap's cryptographic approach and history.
Tarsnap also pays significant cash bounties for bugs found in the product, ranging from a few dollars for small cosmetic bugs up to a couple thousand dollars if someone finds a serious security flaw. This transparent approach is further comfort for the truly paranoid.
Tarsnap's current version is 1.0.32, released on February 22, 2012, for Linux, BSD, OS X, Solaris, Minix and Cygwin.
Q&A with Dr Colin Percival
Dr Colin Percival
A: I've heard "startup company" defined as "time to double revenue or the number of users is measured in months", and I've heard "highly successful startup company" defined as "time to double revenue or the number of users is measured in weeks". By those definitions, Tarsnap is a startup company, but not a highly successful one.
And yes, that is dodging the question, but Tarsnap is my primary source of income, and I come from a culture that considers someone's income to be a private matter, so I don't want to publish precise numbers.
Q. Looking at your Bug Bounties page, you've paid out more than $2,000 to users who've submitted bug reports. Why a "bug bounty" system as opposed to the traditional bug reporting in open-source projects?
A: I gave a talk about this at BSDCan'12 and AusCERT'12 (in consecutive weeks, no less—a word of advice, don't ever try to attend back-to-back conferences 10,000 miles apart). In short, bug bounties help get more people looking at code, and help encourage people to report anything they see that seems wrong.
Probably a better question is why I offered bounties for all bugs rather than just for security bugs—aside from Knuth's famous prizes, I don't know of any other case where bounties extend beyond security vulnerabilities. The answer here is roughly the same though—there's a lot of people who won't look for security bugs because they don't have a security background, but offering cash for all bugs gets them interested...and my experience with FreeBSD is that many security vulnerabilities aren't found by security people, but rather by other developers just saying "something here looks weird".
Q. What do you think are the weakest points of the Tarsnap design, from a security perspective? Is there anything that should keep the truly paranoid up at night?
The weakest link in Tarsnap's security, without question, is me. I wrote the Tarsnap client code to encrypt and sign everything on the client side, but how do you know that it does what I claim?
If you're paranoid, you should look at the code yourself and make sure it does what I claim it does. And when I release a new version, you should look at that too—compare against the previous version and make sure that the changes are sensible.
If the CIA kidnaps me and threatens to torture me until I decrypt someone's data, I won't be able to do anything to help them. But if the CIA kidnaps me and threatens to torture me unless I insert a Trojan horse into the next version of Tarsnap...well, I'm not optimistic about my ability to withstand torture.
Q. Tarsnap's client is very UNIX-nerd-friendly, with its familiar tar syntax. Do you have any interest or plans for a graphical interface for less-sophisticated users?
A: This is absolutely something I plan on doing...in the future. I have very little experience with anything GUI, so I'll probably end up paying someone to produce a GUI—ideally an open-source GUI for tar, since Tarsnap uses almost exactly the same command-line interface, and I think a good GUI front end to tar would be useful for its own sake too.
Q. Why attodollars and not femtodollars or zeptodollars?
A: With attodollars, I can express everything as a pair of 64-bit integers.
Q. Do you have any new features in the works for upcoming Tarsnap releases?
A: I'm mostly working on performance improvements these days. There's a few frequently requested features that I might add, but in general, there are good reasons when features don't exist—either they're impossible (for example, server-side expiration of old archives—the server has no way to know which blocks should be freed when an old archive is deleted) or would require a substantial redesign (for example, renaming archives—the client-server protocol has write transactions and delete transactions, but renaming would need to atomically write some files and delete others).
Backup image via Shutterstock.com.
Andrew Fabbro is a senior technologist living in the Portland, Oregon, area. He's used Linux since Slackware came on floppies and presently works for Con-way, a Fortune 500 transportation company.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
|Introduction to MapReduce with Hadoop on Linux||Jun 05, 2013|
- Containers—Not Virtual Machines—Are the Future Cloud
- Non-Linux FOSS: libnotify, OS X Style
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Linux Systems Administrator
- Introduction to MapReduce with Hadoop on Linux
- RSS Feeds
- Weechat, Irssi's Little Brother
- Tech Tip: Really Simple HTTP Server with Python
- Validate an E-Mail Address with PHP, the Right Way
- Android's Limits
- Reply to comment | Linux Journal
39 min 10 sec ago
- Welcome to 1998
1 hour 27 min ago
- notifier shortcomings
1 hour 51 min ago
3 hours 28 min ago
- Android User
3 hours 29 min ago
- Reply to comment | Linux Journal
5 hours 22 min ago
8 hours 12 min ago
- This is a good post. This
13 hours 25 min ago
- Great, This is really amazing
13 hours 27 min ago
- These posts are really good
13 hours 28 min ago