Pass on Passwords with scp
In this article, I show you how to use the scp (secure copy) command without needing to use passwords. I then show you how to use this command in two scripts. One script lets you copy a file to multiple Linux boxes on your network, and the other allows you to back up all of your Linux boxes easily.
If you're a Linux sysadmin, you frequently need to copy files from one Linux box to another. Or, you may need to distribute a file to multiple boxes. You could use FTP, but using scp has many advantages. For instance, scp is much more secure than FTP. scp travels across the LAN/WAN encrypted, while FTP uses clear text, even for passwords.
But what I like best about scp is it's easily scriptable. Suppose you need to distribute a file to 100 Linux boxes. I'd rather write a script to do this than type 100 sets of copy commands. If you use FTP in your script, things can get messy, because each Linux box you log into is going to ask for a password. But if you use scp in your script, you can set things up so the remote Linux boxes don't ask for a password. Believe it or not, this actually is much more secure than using FTP.
Here's an example demonstrating the most basic syntax for scp. To copy a file named abc.tgz from your local PC to the /tmp dir of a remote PC called bozo, use:
scp abc.tgz root@bozo:/tmp
You now are asked for bozo's root password, so we're not quite there yet. The system still is asking for a password, so it's not easily scriptable. To fix that, follow this one-time procedure, after which you can make endless password-less scp copies:
Decide which user on the local machine will be using scp later on. Of course, root gives you the most power, and that's how I personally have done it. I'm not going to give you a lecture here on the dangers of root, so if you don't understand them, choose a different user. Whatever you choose, log in as that user now and stay there for the rest of the procedure. Log in as this same user when you use scp later on.
Generate a public/private key pair on the local machine. Say what? If you're not familiar with public key cryptography, here's the 15-second explanation. In public key cryptography, you generate a pair of mathematically related keys, one public and one private. You then give your public key to anyone and everyone in the world, but you never ever give out your private key. The magic is in the mathematical makeup of the keys; anyone with your public key can use it to encrypt a message, but only you can decrypt it with your private key. Anyway, the syntax to create the key pair is:
ssh-keygen -t rsa
In response, you should see:
Generating public/private rsa key pair Enter file in which to save the key ...
Press Enter to accept this.
In response, you should see:
Enter passphrase (empty for no passphrase):
You don't need a passphrase, so press Enter twice.
In response, you should see:
Your identification has been saved in ... Your public key has been saved in ...
Note the name and location of the public key just generated. It always ends in .pub.
Copy the public key just generated to all of your remote Linux boxes. You can use scp or FTP or whatever to make the copy. Assuming you're using root--again, see my warning in step 1--the key must be contained in the file /root/.ssh/authorized_keys. Or, if you are logging in as a user, for example, clyde, it would be in /home/clyde/authorized_keys. Notice that the authorized_keys file can contain keys from other PCs. So, if the file already exists and contains text, you need to append the contents of your public key file to what already is there.
Now, with a little luck, you should be able to scp a file to the remote box without needing to use a password. So let's test it by trying our first example again. Copy a file named xyz.tgz from your local PC to the /tmp dir of a remote PC called bozo:
scp xyz.tgz root@bozo:/tmp
Wow--it copied with no password!
A word about security before we go on. This local PC just became pretty powerful, as it now has access to all the remote PCs with only the one local password. So that one password better be strong and well guarded.
Now for the fun part. Let's write a short script to copy a file called houdini from the local PC to the /tmp dir of ten remote PCs, located in ten different cities, in only five minutes of work. Of course, it would work the same with 100 or 1000 of PCs. Suppose the 10 PCs are called brooklyn, oshkosh, paris, bejing, winslow, rio, gnome, miami, minsk and tokyo. Here's the script:
#!/bin/sh for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo do scp houdini root@$CITY:/tmp echo $CITY " is copied" done
It works like magic. With the echo line in this script, you should be able to watch as each city's copy is completed one after the next.
By the way, if you're new to shell scripting, here is a pretty good tutorial.
As you may know, scp is only one part of the much broader SSH program. Here's the cool part: when you followed my six-step procedure above, you also gained the ability to sit at your local PC and execute any command you like on any of the remote PCs--without a password, of course. Here's a simple example to view the date and time on the remote PC brooklyn:
ssh brooklyn "date"
Let's now put these two concepts together for one final and seriously cool script. It's a down-and-dirty way to back up all of your remote Linux boxes. The example backs up the /home dir on each box. It's primitive compared to the abilities of commercial backup software, but you can't beat the price. Consider the fact that most commercial backup software charges license fees for each machine you back up. If you use such a package, instead of paying license fees to back up 100 PCs remotely, you could use the script to back up the 100 PCs to one local PC. Then, back up the local PC to your commercial package and save the license fees for 99 PCs. Anyway, the script below demonstrates the concepts, so you can write your own script to suit your own situation. Simply put this script in a cron job on your local PC; no script is required on the remote PCs. Please read the comments carefully, as they explain everything you need to know:
#!/bin/sh # Variables are upper case for clarity # before using the script you need to create a dir called '/tmp/backups' on each # remote box & a dir called '/usr/backups' on the local box # on this local PC # Set the variable "DATE" & format the date cmd output to look pretty # DATE=$(date +%b%d) # this 'for loop' has 3 separate functions for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo do # remove tarball on remote box from the previous time the script ran # to avoid filling up your HD # then echo it for troubleshooting # ssh -1 $CITY "rm -f /tmp/backups/*.tgz" echo $CITY " old tarball removed" # create a tarball of the /home dir on each remote box & put it in /tmp/backups # name the tarball uniquely with the date & city name # ssh $CITY "tar -zcvpf /tmp/backups/$CITY.$DATE.tgz /home/" echo $CITY " is tarred" # copy the tarball just create from the remote box to the /usr/backups dir on # the local box # scp root@$CITY:/tmp/backups/$CITY.$DATE.tgz /usr/backups echo $CITY " is copied" done # the rest of the script is for error checking only, so it's optional: # on this local PC # create error file w todays date. # If any box doesn't get backed, it gets written to this file # touch /u01/backup/scp_error_$DATE for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo do # Check if tarball was copied to local box. If not write to error file # note the use of '||' which says do what's after it if what's before it is not # true # ls /u01/backup/$CITY.$DATE.tgz || echo " $CITY did not copy" >> scp_error_$DATE # Check if tarball can be opened w/o errors. If errors write to error file. tar ztvf /u01/backup/$CITY.$DATE.tgz || echo "tarball of $CITY is No Good" >> scp_error_$DATE done
That's about it. In this article, I've tried to give examples that demonstrate the concepts and are not necessarily ready to be used "as is". Some of the syntax may not work for all distributions, but in the interest of brevity, I could not include all the possibilities. For example, if you are using Red Hat 6.2 or before, the syntax requires some changes. So be creative, and hopefully you can use some of this work in your own environment.
Copyright (c) 2004, Dave Sirof. Originally published in Linux Gazette issue 98. Copyright (c) 2004, Specialized Systems Consultants, Inc.
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
|Introduction to MapReduce with Hadoop on Linux||Jun 05, 2013|
- Containers—Not Virtual Machines—Are the Future Cloud
- Non-Linux FOSS: libnotify, OS X Style
- Linux Systems Administrator
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Validate an E-Mail Address with PHP, the Right Way
- Technical Support Rep
- Senior Perl Developer
- UX Designer
- Introduction to MapReduce with Hadoop on Linux
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?