Build a Home Terabyte Backup System Using Linux
Generating the Key Pair
On the machine you want to log in to (logged on as bob on bar in this case), type ssh-keygen -d to generate the key pair. Enter a password if the key will be accessible/readable to other users. Otherwise, press Return. Change into the .ssh directory and copy the public key to the allowed list:
cd ~/.ssh cp id_dsa.pub authorized_keys2
Copy the private key to the .ssh directory of the account on the machine you will be logging in from (for example, root user on foo). Remove the private key from bar (the machine you want to log on to):
scp id_dsa root@foo:~/.ssh/id_dsa rm id_dsa
On the machine you're logging in from, start the SSH agent, and add the key to the agent's list (ssh-add asks for a password if you typed one in the first step above):
eval `ssh-agent` ssh-add
You can now log in to account bob bar from foo without a password:
You can run a script on foo to replicate foo on bar using bob's account on bar. You should read the documentation for rsync, which has numerous features (more than 70 command-line options). In particular, the -delete option can have disastrous consequences if misused. Listing 1 shows a seven-day incremental backup. Files altered or deleted on each day of the week are deposited in directories named for the day (set by -backup-dir). The most recent backup is stored in the directory current.
Listing 1. Full and Incremental rsync
#!/bin/sh # This script does backups of foo to the backup server bar # in a 7 day rotating incremental backup. # Based on script by Andrew Tridgell # directory to backup BDIR=/home # Remote directory on backup server BACKUP_HOME=/data1/foo # Backup login account on remote server BACKUP_LOGIN=bob # the name of the backup server BSERVER=bar BACKUPDIR=`date +%A` OPTS="--force --ignore-errors --delete --backup -backup-dir=$BACKUP_HOME/$BACKUPDIR -av" export PATH=$PATH:/bin:/usr/bin:/usr/local/bin # Dump output to backup file date > /var/log/backup.$BACKUPDIR.log # the following line clears the last week's incremental directory [ -d /tmp/emptydir ] || mkdir /tmp/emptydir rsync --delete -a /tmp/emptydir/ BACKUP_LOGIN@$BSERVER:$BACKUP_HOME/$BACKUPDIR/ rmdir /tmp/emptydir # now the actual transfer rsync $OPTS $BDIR BACKUP_LOGIN@$BSERVER:$BACKUP_HOME/current >> /var/log/backup.$BACKUPDIR.log
If you prefer a compressed archive format, you still can run tar for a full backup over the network:
tar cvfz - /home | ssh bob@bar dd of=/data1/foo/current.tar.gz
and use the -newer option for an incremental tar backup.
rsync is more efficient than the tar command, because rsync copies only the differences between the current and previous copy of the data.
You can get by with rsync and SSH on most platforms (including MS Windows), but in reality, a fileserver setup is preferable, especially if you are running MS Windows clients. For MS Windows machines, a Windows backup application is preferable. The easiest way to do this is to run the backup to write to a share on the Samba server.
If your Linux installation supports SMB file sharing, Samba is probably installed. If not, binaries are included with virtually all distributions. If this isn't the case with your distribution, or if you prefer to use the very latest Samba version, download the source code and compile and install. Official Samba distributions are available from the Samba home page (see Resources). Refer to the documentation there for installing and initially configuring Samba.
Once your backup server has Samba server installed, all Samba configurations are made by editing the smb.conf file, which is usually in /etc/samba/smb.conf or /usr/local/samba/lib/smb.conf. Graphical configuration utilities like SWAT usually are included with Samba. See your documentation for information about starting or stopping Samba. You should configure your server to ensure that Samba starts when the server initially boots up.
Following our backup example above, on server bar, set up a simple smb.conf file or try appending the section below to the existing smb.conf file to define a share called bob:
[bob] comment = foo backup account path = /data1/foo valid users = bob public = no writable = yes
Next, add bob with any secure password as a Samba user (bob must have a Linux account as well as permission to read/write the /data1/foo directory):
smbpasswd -a bob New SMB password: somepassword Retype new SMB password: somepassword Added user bob
For MS Windows clients, map the share \\bar\bob as a network drive in MS Windows using the user name bob and the SMB password for the bob Samba account. You then should be able to run backups to the mapped network drive. I typically use the free ntbackup software and set it up to write .bkf files to network storage. ntbackup comes free with Windows 2000 and XP and can run automated, regularly scheduled backups from the Windows client. Windows client-based backups have the advantage of backing up the entire state of the system (including the Windows registry).
You also can use Samba to serve files to most UNIX or Mac OS X clients. The smb client is installed by default in Mac OS X. In Linux distributions, make sure that the smb client package is installed. The smb share should be mounted onto the /backup mountpoint of machine foo:
mount -t smbfs -o username=bob,password=somepassword //bar/foo /backup
To have the backup drive mount when the system boots, place a line such as the following in /etc/fstab:
//bar/data1/foo /backup smbfs rw,username=bob,password=somepassword 0 0
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Non-Linux FOSS: Caffeine!
- Managing Linux Using Puppet
- Tech Tip: Really Simple HTTP Server with Python
- Doing for User Space What We Did for Kernel Space
- Rogue Wave Software's Zend Server
- Parsing an RSS News Feed with a Bash Script
- SuperTuxKart 0.9.2 Released
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide