Sysadmin Sunday: Quick & Dirty SSH Tunneling

Occasionally, you might need to tunnel some other traffic over SSH. This could be to get access to an external web proxy, to get a remote X display up, or to get around a firewall-blocked port that you must access.

The syntax (on the command-line OpenSSH client) for a Local forward is like this:

# ssh remote-example.h-i-r.net -L 3128:localhost:3128

This tells my SSH client to tunnel traffic to port 3128 on my workstation to port 3128 on my DMZ box. Port 3128 isn't accessible because of a firewall, but tunnelled over SSH it works fine. In this case, I'm running squid on the remote example host. Telling Firefox to use http://localhost:3128 as the proxy will now tunnel all of my web traffic over the SSH tunnel to the squid proxy behind the firewall. The reason it's called a local forward because it forwards a local port over the SSH connection.

A remote forward will open up a port on the remote machine and connect it to a port on the client's network. The syntax is similar:

# ssh remote-example.h-i-r.net -R 3306:dbserver:3306

This would open up port 3306 (the MySQL server port) on the remote host and tunnel it to the MySQL service on the host named dbserver on my local network.

While running forwarding of either type, you can enter the hotkey sequence "~#" to see all the open connections through the forwarded ports.

Sysadmin Sunday: Automate remote tasks with SSH

The SSH Suite can be leveraged to perform some pretty amazing tasks. I maintain hundreds upon hundreds of open systems as part of my job, but a good deal of my experience with SSH for batch processing came from a brief stint at a startup company that had employed a few hundred OpenBSD systems in an application-clustered environment. Keeping them in sync wasn't too hard once I came up with a few creative ways to get everything straight.

Currently, the most popular (and arguably most secure) SSH suite is OpenSSH which ships with almost all Open-Source operating systems and some commercial ones. Packages exist for most others, or there's always the source code.

For the un-initiated, SSH itself replaces RSH and Telnet as remote-access protocols for interactive logins. SFTP replaces the insecure FTP protocol with an encrypted, SSH-tunneled file transfer protocol. SCP replaces RCP with a secure remote copy protocol. At its core, SSH can simply authenticate these tools against the host's existing password database. The real power of SSH comes from its ability to use public keys to authenticate accounts. This kind of authentication is fragile if implemented improperly. If one account is compromised and the account has weakly-protected keys distributed to other systems, the impact of that one account can be much further-reaching. It is for this reason that I recommend using un-privileged accounts when using passwordless SSH keys.

Generating SSH Keys
To generate SSH keys for your account, you login to the account on the system that you'll be running automated or batch tasks from, then run ssh-keygen and follow the prompts, entering a blank password for the key.

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/axon/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/axon/.ssh/id_rsa.
Your public key has been saved in /home/axon/.ssh/id_rsa.pub.
The key fingerprint is:
5c:f4:64:3c:51:a3:a7:80:fd:fa:55:7a:06:f7:4d:84 axon@chimerabsd.labs.h-i-r.net

Distributing SSH keys
From here, you have to push the key out to each remote host by adding the contents of the id_rsa.pub file to the end of "authorized_keys" in the .ssh directory of the account you're going to be connecting to. My usual method goes something like this:

ssh user@remote.host "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub

If this returns an error such as ".ssh/authorized_keys: No such file or directory", then you need to go to the remote host and create the .ssh directory manually.  This ssh'es to the remote account, and runs "cat >> .ssh/authorized_keys", which takes any input and appends it to the authorized_keys file. At the same time, it's sending the contents of of id_rsa.pub to be appended.  

The above command will prompt you for a password for the remote user, then exit immediately. From that point on "ssh user@remote.host" should no longer ask for a password, and then the account can be used for remote automation. Note: You can also enter a password during key generation, and this password will then be prompted for when ssh'ing to remote hosts. This hinders automation, however.

I chose, instead, to make a script to push the keys out to multiple systems, since I'll be demonstrating batch processing of remote commands.  First, come up with a list of all the remote systems you wish to effect at the same time.  I'm going to choose from a subset of HiR Lab boxes that I've got running right now:

---

File: labs.h-i-r.net.list

---

backtrack.labs.h-i-r.net
bouncer.labs.h-i-r.net
chimera.labs.h-i-r.net
chimerabsd.labs.h-i-r.net
lampdev.labs.h-i-r.net

---

The key push script:

---

File: key.pu.sh

---

#!/bin/sh
# mass ssh key push script by ax0n@h-i-r.net 2008-01-06
# Pushes ssh keys out to an entire list of hosts from a file
if [ -z $1 ]
then
echo "Syntax: $0 server-list "
exit 1
fi
boxlist=$1
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub done

---

Then run key.pu.sh [server-list-file].  It will ask for passwords for each machine.  That's the breaks, kid.

-bash-3.2$ ./key.pu.sh labs.h-i-r.net.list
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
axon@backtrack.labs.h-i-r.net's password:
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
axon@bouncer.labs.h-i-r.net's password:
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
Password:
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
axon@chimerabsd.labs.h-i-r.net's password:
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
axon@lampdev.labs.h-i-r.net's password:

Testing the SSH keys

Now that the keys are pushed, remote work will be simple.  Try to ssh to any of the systems in the list.  It should just work without a password:

-bash-3.2$ ssh axon@backtrack.labs.h-i-r.net
Last login: Sun Jan 6 22:34:20 2008 from 192.168.0.107
Linux 2.6.21.5.
backtrack ~ $ logout
Connection to backtrack.labs.h-i-r.net closed.

If you wish to run a single command remotely, then exit, you can also add the command to the end of the command line like this:

-bash-3.2$ ssh bouncer.labs.h-i-r.net uname -a
OpenBSD bouncer.labs.h-i-r.net 4.2 GENERIC#851 sparc

Similar to my key push script is one that I wrote to run a command across all of the systems in a list file:

---

File: mass-ssh.sh

---

#!/bin/sh
# mass ssh script by ax0n@h-i-r.net 2008-01-06
# Runs the same command across an entire list of hosts from a file

if [ -z $2 ]
then
echo "Syntax: $0 server-list command-string"
exit 1
fi
boxlist=$1
shift
commandstring=$*
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "$*"
done

---

Let's see it in action by running the "uptime" command across my lab environment:

-bash-3.2$ ./mass-ssh.sh labs.h-i-r.net.list uptime
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
22:53:40 up 5 days, 5:21, 2 users, load average: 0.24, 0.05, 0.02
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
10:07PM up 2:31, 0 users, load averages: 0.28, 0.15, 0.10
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
16:53 up 1 day, 6:02, 3 users, load averages: 1.44 1.32 1.15
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
10:54AM up 2:32, 1 user, load averages: 0.32, 0.14, 0.10
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
16:53:19 up 2:08, 0 users, load average: 0.08, 0.02, 0.01

Notice how I don't get prompted for any passwords.  It's important to note that my account on these systems is a non-privileged account without access to sudo or membership in any special system groups such as root, wheel, adm, or bin.  You could use this script with a privileged account for adding or removing users, but I'd keep very tight security on any system with passwordless ssh keys.  It's mostly useful for seeing who is logged in, checking for high system loads, and whatnot.

Transferring files

sftp works much like ftp.  The syntax is "sftp user@host.name" and then it either prompts for a password, or if SSH keys are present, uses those.  After that, most of your usual FTP commands work as usual. In the following example, I do not have SSH keys from chimera (my laptop) to itself (localhost):

chimera$ sftp axon@localhost
Connecting to localhost...
Password:
sftp> ls
Desktop Documents Downloads Library Movies Music Photos
Pictures Public Sites myphotos scp.sh
sftp> cd Photos
sftp> ls
06-06-07_1911.jpg 06-25-07_0600.jpg PowerLinkChain.jpg
sftp> get PowerLinkChain.jpg
Fetching /Users/axon/Photos/PowerLinkChain.jpg to PowerLinkChain.jpg
/Users/axon/Photos/PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

Better yet, in my opinion, is scp, the secure replacement for rcp. The syntax is simply the same as "cp" with the exception of it accepting a user and host name and a colon in front of the filename. This allows you to copy files over the network via an encrypted channel from the command-line. I'll do the same thing as before, from chimerabsd (which has ssh keys to chimera) to my home directory locally:

-bash-3.2$ scp axon@chimera.labs.h-i-r.net:Photos/PowerLinkChain.jpg ~
PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

And as you guessed, I also have a program for pushing a file to a list of hosts as well.

---

File: pu.sh

---

#!/bin/sh
# mass scp push script by ax0n@h-i-r.net 2008-01-06
# uses scp to push one file out to an entire list of hosts from a file
if [ -z $2 ]
then
echo "Syntax: $0 file server-list [remote-path]"
exit 1
fi
boxlist=$2
pushfile=$1
rempath=$3
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
scp $file $box:$rempath
done

---

Troubleshooting
If you have trouble getting SSH keys to work, make sure the "PubkeyAuthentication" option in the sshd configuration file (usually /etc/ssh/sshd_config) is set to "yes" and un-commented.

In Closing

My scripts were quick and dirty examples -- use the scripts as a template for more complex tasks, such as uploading and unpacking a tarball nightly, running a database backup via mysqldump, or any other thing you can imagine.  You may wish to add an option for an alternate username, or hard-code a username into your own scripts.  Once SSH keys are in place, you can automate many non-interactive tasks via Cron or At as well.

Sysadmin Sunday: A somewhat secure quick and dirty network backup with ssh, rsync, tar and cron

Greetings and salutations.

In this article I will cover a method of backing up a small number of servers to a central storage server for daily, weekly and monthly archiving. This is good for under 10 machines needing a basic backup solution. It is not meant for large installations due to limitations of rsync and space considerations.

-=- Index -=-

1...... Summary
2...... Description of the process
3...... Server Setup Details
4...... Client setup details
5...... Sripts
6...... Security Concerns
7...... Informative resources

1. Summary
This article is intended for system administrators with intermediate experience with Un*x environments. You will require a server with a large stable amount of space. Bonus points for raid and other storage redundancy features. You will also need an up to date version of sshd installed on the server as well as tar, gzip and rsync. You also will need up to date versions of ssh and rsync on each of the machines to be backed up. I say somewhat secure because it uses ssh to shuttle the files to the server and thus is granted a level of security from eavesdropping. Its a simple way of backing up a server with out the hassle of implementing a full blown backup server software on a multi-unix-platform environment. SSH, rsync, gzip, tar and split are all usually available in a unix or unix-like installation and for the most part compatible with each other. Thus, very little compiling is needed to implement this.

2. Description of the process
Each client will be configured with a time slot and at the given time it will open a ssh connection to the backup server to an un-privileged account. It will then start rsync and synchronize a list of folders with a corresponding set of folders on the remote server. It will then disconnect and continue on its business.

The server, during off-peak hours will perform weekly differential tar backups of the
rsync archive. The server will roll the whole archive into a compressed tar archive and move it to an archive directory either on the server or using some other attached storage.

3. Server Setup Details
SSH needs to be available to the machines needing to backed up. I'm leaving the security details fuzzy here because there are so many ways to secure this setup, setting the ssh server to only use trusted host keys, certain usernames, groups ...etc In this case we are using public key authentication so this option needs to be enabled on the sshd server.

Our backup user is a regular user (no special groups, just the basic usergroup). For this example our backup user is "backupuser" who belongs to group "backupuser".

Our backup user's quota setup and home directory need to have lots of storage space available.

Inside their home directory we need a directory structure similar to this:

-SERVERBACKUPS
- SERVER1
+ Daily
+ Weekly
- SERVER2
+ Daily
+ Weekly
- SERVER3
+ Daily
+ Weekly

NOTE: Security precautions you should look at taking is switching on the no-execute feature on the file system (if the folder resides some where that wont need scripts being executed from). The backup user account needs to have a restricted shell (ex. /bin/rbash). Security is beyond the scope of this article so use your best judgement.

Then configure cron to run your roll scripts. (see section 5)

You will want to run crontab on the backup server as root or some other user with enough permissions to manipulate the backup data owned by our backup user.
Run :

#crontab -e

(It will then run vi or the other default editor)
... add the following lines

# 12:30pm on Saturdays
30 12 * * 6 /root/scripts/weekly_diff.sh
# 12:30pm on Sunday once every month
30 12 * 1-12 0 /root/scripts/monthly_roll.sh

(Save the file and exit the editor)

We will now need to make the ssh private key.

#ssh-keygen -t dsa

***You will not want to put a password on this key (just press enter).

Then place the public key into the backup users ~/.ssh/authorized_keys and chmod 700 on the .ssh directory and chmod 600 on the key itself. (this would be a good time to verify that sshd is configured for public key logins)

You can also put multiple public keys into the authorized keys list (one for each client).

4...... Client setup details

Each client needs to have a copy of the private key we made in section 3. You will want to run chmod 600 on the key to prevent other system users access to the key. Then put the backup.sh script and the private key into a folder accessible only to the root account (like /root/scripts) and run chmod 700 on the script so only the owner (root) can execute it.

make sure the file is where the script expects to to find it

You will then want to put in a cron job for the root account.

#crontab -e

(it will then launch the system default editor like vi)
insert the following commands:

# Backup the fs at 11:30pm every day of the week
30 23 * * * /root/scripts/backup.sh

Save the file and exit. Now it will now execute the /root/backup.sh script at 11:30pm every day.

5...... Scripts

#!/bin/sh
#--------SERVER WEEKLY ARCHIVING SCRIPT (weekly_diff.sh) -----------
#!/bin/bash
DATE=`date +%V%g`
cd /data/serverbackups
for file in *; do
tar --create \
-z \
--file=$file/weekly/$file$DATE.tar.gz \
-g $file/weekly/weekly-diff.snar \
$file/daily
done

#end weekly script

#--------SERVER MONTHLY ARCHIVING SCRIPT (monthly_roll.sh)-----------
#!/bin/bash
DATE=`date +%b%g`
cd /path/to/serverbackups
mkdir /path/to/archive/$DATE
for file in *; do
rm $file/weekly/*
FILENAME=$DATE.tgz
tar -zcvf $file/weekly/$FILENAME -g $file/weekly/weekly-diff.snar $file/daily/.
cp $file/weekly/*.tgz /path/to/archive/$DATE
done

#end monthly script

#--------SERVER OFF SITE ARCHIVING SCRIPT -----------
MOUNT_CMD="/path/to/mount"
MOUNT_DEV="/path/to/external/storage/device"
MOUNT_POINT="/path/to/mount/point"
MOUNT_FS="filesystem name -t "
MKNOD_CMD="/path/to/mknod /tmp/tar_pipe p"
SPLIT_CMD="/path/to/split -b 512000000"
ARC_PATH="/path/to/monthly/archive"
TAR="/path/to/tar -cvf"
GZ="/path/to/gzip"
DATE=`/path/to/date +%m%Y`

$MOUNT_CMD $MOUNT_FS $MOUNT_DEV $MOUNT_POINT
mkdir $MOUNT_POINT/$DATE
$TAR $MOUNT_POINT/$DATE/archive_$DATE.tgz /tmp/tar_pipe &
$SPLIT_CMD /tmp/tar_pipe $MOUNT_POINT/$DATE/archive_backup_$DATE.tar.
#end of archive script

#------------------CLIENT BACKUP.SH-------------------
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/path/to/key
RUSER=backupuser
#SERVER IP
RHOST="11.22.33.44"
RPATH="/path/to/serverbackups/servername/daily"
LPATH=/

$RSYNC -az --links --safe-links --exclude /dev --exclude /proc --exclude /mnt $LPATH -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH


#end of backup.sh

6...... Security Concerns
Obviously, having an account which contains all of your network data available to any one who has the secure key is a problem. Having an rsynced archive of everything also has other file related issues. You could use different backup server local users, keys or tack on some countermeasures which chmod the files to something less offensive.

Then we have concerns about ssh itself. Internet sites are vulnerable to scripted ssh probes using dictionary attacks ..etc
You could move ssh to a different port and avoid some of the scripted attacks. Once again this is all outside the scope of this article.

If these concerns are a bit too much for your environment consider using a backup system like Baccula or Amanda or one of the commercial backup solutions.

7...... Informative resources

Johnson, Troy. "Using Rsync and SSH." Accessed December 2007
http://troy.jdmz.net/rsync/index.html

Linux-Backup.net "Examples." Accessed December 2007
http://www.linux-backup.net/Example

OpenSSH.org "Manuals." Accessed December 2007
http://openssh.org/manual.html