The SSH Suite can be leveraged to perform some pretty amazing tasks. I maintain hundreds upon hundreds of open systems as part of my job, but a good deal of my experience with SSH for batch processing came from a brief stint at a startup company that had employed a few hundred OpenBSD systems in an application-clustered environment. Keeping them in sync wasn't too hard once I came up with a few creative ways to get everything straight.
Currently, the most popular (and arguably most secure) SSH suite is OpenSSH which ships with almost all Open-Source operating systems and some commercial ones. Packages exist for most others, or there's always the source code.
For the un-initiated, SSH itself replaces RSH and Telnet as remote-access protocols for interactive logins. SFTP replaces the insecure FTP protocol with an encrypted, SSH-tunneled file transfer protocol. SCP replaces RCP with a secure remote copy protocol. At its core, SSH can simply authenticate these tools against the host's existing password database. The real power of SSH comes from its ability to use public keys to authenticate accounts. This kind of authentication is fragile if implemented improperly. If one account is compromised and the account has weakly-protected keys distributed to other systems, the impact of that one account can be much further-reaching. It is for this reason that I recommend using un-privileged accounts when using passwordless SSH keys.
Generating SSH KeysTo generate SSH keys for your account, you login to the account on the system that you'll be running automated or batch tasks from, then run ssh-keygen and follow the prompts, entering a blank password for the key.
-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/axon/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/axon/.ssh/id_rsa.
Your public key has been saved in /home/axon/.ssh/id_rsa.pub.
The key fingerprint is:
5c:f4:64:3c:51:a3:a7:80:fd:fa:55:7a:06:f7:4d:84 axon@chimerabsd.labs.h-i-r.net
Distributing SSH keysFrom here, you have to push the key out to each remote host by adding the contents of the id_rsa.pub file to the end of "authorized_keys" in the .ssh directory of the account you're going to be connecting to. My usual method goes something like this:
ssh user@remote.host "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub
If this returns an error such as ".ssh/authorized_keys: No such file or directory", then you need to go to the remote host and create the .ssh directory manually. This ssh'es to the remote account, and runs "cat >> .ssh/authorized_keys", which takes any input and appends it to the authorized_keys file. At the same time, it's sending the contents of of id_rsa.pub to be appended.
The above command will prompt you for a password for the remote user, then exit immediately. From that point on "ssh user@remote.host" should no longer ask for a password, and then the account can be used for remote automation. Note: You can also enter a password during key generation, and this password will then be prompted for when ssh'ing to remote hosts. This hinders automation, however.
I chose, instead, to make a script to push the keys out to multiple systems, since I'll be demonstrating batch processing of remote commands. First, come up with a list of all the remote systems you wish to effect at the same time. I'm going to choose from a subset of HiR Lab boxes that I've got running right now:
File: labs.h-i-r.net.list
backtrack.labs.h-i-r.net
bouncer.labs.h-i-r.net
chimera.labs.h-i-r.net
chimerabsd.labs.h-i-r.net
lampdev.labs.h-i-r.net
The key push script:
File: key.pu.sh
#!/bin/sh
# mass ssh key push script by ax0n@h-i-r.net 2008-01-06
# Pushes ssh keys out to an entire list of hosts from a file
if [ -z $1 ]
then
echo "Syntax: $0 server-list "
exit 1
fi
boxlist=$1
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub done
Then run key.pu.sh [server-list-file]. It will ask for passwords for each machine. That's the breaks, kid.
-bash-3.2$ ./key.pu.sh labs.h-i-r.net.list
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
axon@backtrack.labs.h-i-r.net's password:
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
axon@bouncer.labs.h-i-r.net's password:
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
Password:
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
axon@chimerabsd.labs.h-i-r.net's password:
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
axon@lampdev.labs.h-i-r.net's password:
Testing the SSH keys
Now that the keys are pushed, remote work will be simple. Try to ssh to any of the systems in the list. It should just work without a password:
-bash-3.2$ ssh axon@backtrack.labs.h-i-r.net
Last login: Sun Jan 6 22:34:20 2008 from 192.168.0.107
Linux 2.6.21.5.
backtrack ~ $ logout
Connection to backtrack.labs.h-i-r.net closed.
If you wish to run a single command remotely, then exit, you can also add the command to the end of the command line like this:
-bash-3.2$ ssh bouncer.labs.h-i-r.net uname -a
OpenBSD bouncer.labs.h-i-r.net 4.2 GENERIC#851 sparc
Similar to my key push script is one that I wrote to run a command across all of the systems in a list file:
File: mass-ssh.sh
#!/bin/sh
# mass ssh script by ax0n@h-i-r.net 2008-01-06
# Runs the same command across an entire list of hosts from a file
if [ -z $2 ]
then
echo "Syntax: $0 server-list command-string"
exit 1
fi
boxlist=$1
shift
commandstring=$*
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "$*"
done
Let's see it in action by running the "uptime" command across my lab environment:
-bash-3.2$ ./mass-ssh.sh labs.h-i-r.net.list uptime
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
22:53:40 up 5 days, 5:21, 2 users, load average: 0.24, 0.05, 0.02
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
10:07PM up 2:31, 0 users, load averages: 0.28, 0.15, 0.10
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
16:53 up 1 day, 6:02, 3 users, load averages: 1.44 1.32 1.15
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
10:54AM up 2:32, 1 user, load averages: 0.32, 0.14, 0.10
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
16:53:19 up 2:08, 0 users, load average: 0.08, 0.02, 0.01
Notice how I don't get prompted for any passwords. It's important to note that my account on these systems is a non-privileged account without access to sudo or membership in any special system groups such as root, wheel, adm, or bin. You could use this script with a privileged account for adding or removing users, but I'd keep very tight security on any system with passwordless ssh keys. It's mostly useful for seeing who is logged in, checking for high system loads, and whatnot.
Transferring files
sftp works much like ftp. The syntax is "sftp user@host.name" and then it either prompts for a password, or if SSH keys are present, uses those. After that, most of your usual FTP commands work as usual. In the following example, I do not have SSH keys from chimera (my laptop) to itself (localhost):
chimera$ sftp axon@localhost
Connecting to localhost...
Password:
sftp> ls
Desktop Documents Downloads Library Movies Music Photos
Pictures Public Sites myphotos scp.sh
sftp> cd Photos
sftp> ls
06-06-07_1911.jpg 06-25-07_0600.jpg PowerLinkChain.jpg
sftp> get PowerLinkChain.jpg
Fetching /Users/axon/Photos/PowerLinkChain.jpg to PowerLinkChain.jpg
/Users/axon/Photos/PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00
Better yet, in my opinion, is scp, the secure replacement for rcp. The syntax is simply the same as "cp" with the exception of it accepting a user and host name and a colon in front of the filename. This allows you to copy files over the network via an encrypted channel from the command-line. I'll do the same thing as before, from chimerabsd (which has ssh keys to chimera) to my home directory locally:
-bash-3.2$ scp axon@chimera.labs.h-i-r.net:Photos/PowerLinkChain.jpg ~
PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00
And as you guessed, I also have a program for pushing a file to a list of hosts as well.
File: pu.sh
#!/bin/sh
# mass scp push script by ax0n@h-i-r.net 2008-01-06
# uses scp to push one file out to an entire list of hosts from a file
if [ -z $2 ]
then
echo "Syntax: $0 file server-list [remote-path]"
exit 1
fi
boxlist=$2
pushfile=$1
rempath=$3
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
scp $file $box:$rempath
done
TroubleshootingIf you have trouble getting SSH keys to work, make sure the "PubkeyAuthentication" option in the sshd configuration file (usually /etc/ssh/sshd_config) is set to "yes" and un-commented.
In Closing
My scripts were quick and dirty examples -- use the scripts as a template for more complex tasks, such as uploading and unpacking a tarball nightly, running a database backup via mysqldump, or any other thing you can imagine. You may wish to add an option for an alternate username, or hard-code a username into your own scripts. Once SSH keys are in place, you can automate many non-interactive tasks via Cron or At as well.
Posts
