awk over non-interactive ssh sessions

I had trouble wrapping my brain around this one today. I was attempting to grab the second field of ps(1) output to display only the PID of a given process. Of course, if we had pgrep and pkill, this would be a no-brainer. But we don't.

Note, when I use grep on ps(1) I also pipe it through grep -v grep (which ignores any line containing the pattern "grep") so that grepping for some_process doesn't end up like this:

root 751 200 0 Jun 25 ? 3:11 /usr/sbin/some_process -d
axon 2429 222 0 4:54:59 pts/1 0:00 grep some_process

First attempt:

ssh somebox "ps -ef | grep some_process | grep -v grep | awk '{print $2}' "
The output, though, was the whole line out of ps.

root 751 200 0 Jun 25 ? 3:11 /usr/sbin/some_process -d

I attempted escaping the ticks, double ticks, escaped double ticks, double quotes, and all kinds of madness. Nothing was working. I was getting either a whole ps line (as if awk wasn't even there) or syntax errors from awk.

Finally, I ask a co-worker (the biggest shell geek I know) pointed out that it was being frobbed by two separate shells. He gave me a somewhat complicated line to use, but I figured out an easier way. The second shell was interpreting $2, thinking I was referencing a shell argument, and was passing nothing to awk. Solution? Escape $2.

ssh somebox "ps -ef | grep some_process | grep -v grep | awk '{print \$2}' "
751

Hooray! Hopefully this helps some poor sysadmin somewhere when the time comes to reference variables remotely in something like perl or awk.

Reverse SSH Tunnel Watchdog

We've covered tunneling before on HiR. I even wrote a little about reverse tunneling in my quick-and-dirty tunneling howto. This time, I'm building a setup to make an always-on reverse tunnel with a cron-powered watchdog script. I've even coded a cron watchdog before, but this way is less hackish, in my opinion.

Here's what's needed to make it work:

• A Linux/BSD/Unix system on the inside of your target network that's capable of SSH-ing out to the Internet (even if via strange ports)
  
• An SSH server you control on the Internet. This can be at home, or elsewhere

The reverse tunnel can handle pretty much any protocol. To a squid proxy, for example. I'll be using SSH to reverse-tunnel SSH, though, to allow SSH access to a server behind a firewall that I do not control. Here's how it'll work:

Cron will run a script on the server on the inside. This script will check to see if the SSH tunnel is working properly. If it's not (or if it hasn't been started yet), it will start the reverse tunnel.

Here's the script, which I put in /usr/local/bin/rtunnel.sh. Obviously, you need to edit the first 4 variables to reflect your environment.

#!/bin/sh
USERHOST=axon@somewhere.labs.h-i-r.net # Login and External system
RPORT=22 # SSH Listener port on your external system
FPORT=1337 # Port that will be opened locally to tunnel SSH
CONN=localhost:22   # SSH Listener on the system behind the firewall

COMMAND="ssh -q -N -R $FPORT:$CONN $USERHOST -p $RPORT"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
ssh $USERHOST -p $RPORT netstat -an | egrep \
"tcp.*:$FPORT.*LISTEN">/dev/null 2>&1
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi

The above essentially runs this as the command line:

ssh -q -N -R 1337:localhost:22 axon@somewhere.labs.h-i-r.net -p 22

Then uses pgrep (ps command with grep functionality) to see if it's up and running, and tries to ssh to the outside system, using netstat to check the status on that end. If either of those fail, the process is killed with pkill (pgrep, with kill functionality) and restarted.

And then I put the entry in root's crontab, to run every 5 minutes:

*/5     *       *       *       *       /usr/local/bin/rtunnel.sh

In my implementation, the firewalled host (An OpenBSD box) is connecting to a Ubuntu desktop system at my home. I can just log into it, then use the tunnel on port 1337, using the -p [port] option.

axon@somewhere:~$ ssh localhost -p 1337
axon@localhost's password:
Last login: Sat Nov 14 00:01:04 2009 from localhost.labs.h-i-r.net
OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

-bash-3.2$

If the connection times out or fails for any other reason, the remote end should re-spawn the connection in the next 5 minutes. If you've waited, and don't get a response, something else might be amiss. DNS rules, a network admin that's blocked you, etc... You can try switching the port SSH runs on

The one problem I've had is that occasionally the session will be alive, the port will be forwarded, but I can't get it to log in. It just hangs then times out. If this happens, I use lsof on my workstation to find and kill off the process that's listening on the TCP port I am using for forwarding.

axon@somewhere:~$ sudo lsof -n | grep TCP.*:1337
sshd   20386   axon    9u   IPv6    2862057  TCP [::1]:1337 (LISTEN)
sshd   20386   axon   10u   IPv4    2862058  TCP 127.0.0.1:1337 (LISTEN)
axon@somewhere:~$ kill 20386

Of course, you can also tunnel stuff over this reverse tunnel. The possibilities are endless!

If you see Open0wn.c or 0wn0wn.c laying around, don't run it, mkay?

I continue to see this code masquerading as the Zero-Day OpenSSH Exploit. It showed up on Pastebin today so I reported it as abusive. If you haven't figured it out yet, this "exploit code" only exploits the person that runs it -- i.e. script kiddies and/or careless researchers.

Thierry Zoller already did a good a great job "disassembling" this and proving that it is being spread just to lay pwnage upon those who are trying to find and/or test the exploit.  It basically deletes everything you have in your homedir (and all of the root dir, if you're dumb enough to run it as root) as shown below. I modified the jmpcode[] to something an online hex-to-ascii converter can comprehend, and the result was "rm -rf ~ /* 2> /dev/null &" 

As a general rule, you should always be careful when tinkering with someone else's un-trusted code, never run stuff like this as root without checking the source code and (yes, I'll say it again) back up your data. It will save your ass.

Check out Thierry's post on secdev for a decode of the rest. It's a good read.

OpenSSH - The coast is clear. For now.

Remember, this was all rumor to begin with. Analysis of the logs has pretty much proven that they were just doctored to look like something new. This was just a vanilla brute-force attack tool.

The ISC wants everyone to quit spreading FUD. I agree.

I am still uneasy about strange OpenSSH bastardizations with old code being used as "enterprise" SSH implementations, and I still think that creating Google Alerts RSS feeds and pumping them into Google Reader is a great way to track the development of rumors or any kind of breaking news.

Tracking Rumors (a la the OpenSSH Exploit)

By now, I'm sure you've all heard the OpenSSH Exploit rumor.  The short and sweet points are:

• The rumored exploit doesn't work on the current version (5.2/5.2p1 as of writing)
• The rumored exploit does work against older versions (but we don't know how old or when it got fixed)
• It's not a bad idea to upgrade your OpenSSH (and derivative) services to OpenSSH 5.2.

What really concerns me are forks from OpenSSH that are likely to be ubiquitous in the enterprise. There are many, but the following two seem like A Pretty Big Deal to me:

• Red Hat Enterprise Linux ships with OpenSSH 4.x, but patches it in-house and releases these updates to RHEL users to fix certain bugs as they're fixed in the 5.x series. 
  
• Sun Solaris 10 ships with "SunSSH 1.1" which is basically a mash-up based on OpenSSH 3.5p1.

You see why I'm more than a little concerned, right?   Without having the exploit code to test with, we don't know if the exploit will work against these bastardizations of the OpenSSH code-base.

Without some solid proof, I'm not going to go to my boss and scream that the sky is falling. I just want to stay in touch with the OpenSSH / 0pwn0wn exploit drama. Google Alerts to the rescue!

Google Alerts allows you to get rapid-fire email or RSS feed updates when new items show up in Google's index for given search terms. You can use this for vanity searching and a host of other things... or, as I do, to keep an eye on breaking news for more obscure stuff.

With that, I set up alerts for OpenSSH (News and Blogs) and 0pen0wn (Comprehensive search) - If an exploit is released publicly, I want to know about it so that I can test it and make recommendations on how to fix it.

Also, it's not a bad idea to set up google alerts for other mission-critical products or services you rely on, if for nothing else, to keep your fingers on their pulse.

Asmodian's Work Bench: Hacking The WD MyBook NAS Drive

The Project
The WD World Book II is a network attached storage device meant for a typical home or professional user. It varies in storage capacity. At the time of writing the sizes vary between 250GB and 2TB. The device itself is the size of an external USB hard-drive, the larger capacities being wider than the base end models. Inside there is a controller board connected to one or two 3.5 inch hard drives. The Controller is running an ARM processor on 32 mb of ram.

The group at mybookworld.wikidot.com collected some scripts to trick the upgrade feature on the drives web interface to run arbitrary commands as root.

Out of the box, the underlying OS is not available. The goal of this project is to enable the ssh server and/or telnet.

The Process
Side notes on recent firmware revisions. Recent revisions will display an error message, the payload will have still have been run though.

1. Login to the world book web interface. (the default login is admin : 123456)


2. Make a user with the web interface


3. Type the following into your browser : http://(DEVICE IP)/auth/firmware_upgrade.pl?fwserver=martin.hinner.info/mybook/firmware.php or http://(DEVICE IP)/auth/firmware_upgrade.pl?fwserver=mybook1.110mb.com/firmware.php


4. You will now see a screen telling you that there is a new firmware release available. Click on the "Download and install" button.


5. The process will take about 5 minutes or less. Then login via ssh as the user you made on step two then su to the root user with a blank password. If you use the alternate firmware site the password is 'root'

Now I would like to say, at this point, with the firmware release I have this did not work for me. If you still cannot login via ssh, try these steps:

1. Type in http://(DEVICE IP)/auth/firmware_upgrade.pl?fwserver=www.geekoh.com/mybook/telnet


2. Telnet to the device and login as your user. Then su to root with the password being root or "" depending on which scripts you ran.

Once you have root edit the /etc/inittab file and add "::sysinit:/usr/sbin/sshd".

There is more than enough storage space to work with so there isn't any reason you couldn't run a full LAMP environment off of it along with whatever services you want.

Resources:

The Hacks:

mybookworld.wikidot.com. "hacks and howto" Accessed January, 2009.

Hinner, Martin. "Hacking Western Digital MyBook World Edition" (September 14, 2008) Accessed January, 2009.


Physical Repair:
Pascucci, Mario. "How to Revive Western Digital Mybook World Edition" (July 25, 2007) Accessed January, 2009.

The Device:
Western Digital Corp. "My Book® World Edition™ II" Accessed January, 2009.


Related HiR Articles:
Ax0n. "La fonera lab: Fon unbricking howto", (October 10, 2008) Accessed January, 2009.

Ax0n. "Jasagar Lives Muahahaha" (October 9,2008) Accessed January, 2009.

Reverse Tunnel with SSH

Inspired by a thread on the Hak5 Forums...

Sometimes, you might find yourself wishing you could poke arbitrary holes through a NAT or firewall. The potential uses (both good and evil) are nearly limitless. Perhaps you want to be able to log in to the computer in your dorm room while you're a thousand miles away on vacation with family.

You will need a few things before we start:

1. Physical access to a computer on the "inside" of the network
2. Access and permission to install software on that computer
3. A system on the "outside" that can accept SSH connections from the Internet

It helps if the 2 computers involved are running Linux, Mac OS X, BSD or some UNIX variant, but you could probably use PuTTy on Windows for the "inside" computer in a pinch.

I'll be using NetBSD on my SparcStation 20 on the inside of the HiR Lab, and one of my web servers with a public IP address on the outside.

SSH Tunneling is the process of encapsulating some other protocol within an SSH session. There are many advantages to this. Essentially, if you can get out of a network with SSH, you can get to almost any TCP service on the outside world, even if it's blocked.

Reverse SSH Tunneling is a similar process. If you can get out of the network with SSH, you can use that SSH tunnel to spawn a listening process on the outside of the network, from which you can get to anything internal.

Step One: Deploy SSH keys without a password
This has to be done from the "inside" computer. You can use your personal desktop system or a server where you have an account. It must be able to SSH out to the Internet.

I don't like to use public keys without passwords, but they do come in handy, particularly for tunneling. The un-protected public key should only be placed into the authorized keys list for UN-TRUSTED accounts that aren't in the wheel group or sudoers file.

I wrote about using passwordless SSH keys before. If you're using a UNIX variant, just follow the first few steps on generating and distributing the ssh keys. If you're using Windows, install PuTTy and follow the steps that Steve Friedl put together.

Step Two: Create and test your tunnel
I wrote a little about tunneling (forward and reverse) back in February. In this case, I'm going to tunnel SSH over a reverse SSH connection -- that is, I'm going to SSH from my inside box (NetBSD) to the web server (betaweb.h-i-r.net), and tell SSH to open a reverse tunnel port (2222) on the web server that connects to the SSH port on the NetBSD box (localhost:22). The -g option allows any remote host to use the tunnel. Otherwise, it will bind only to the localhost interface.

[axon@NetBSD]$ ssh axon@betaweb.h-i-r.net -g -R 2222:localhost:22
Last login: Sun Nov 30 14:13:08 2008 from netbsd.labs.h-i-r.net
[axon@betaweb.h-i-r.net]$

Now, from anywhere in the world that can access port 2222 on betaweb.h-i-r.net, I can SSH through my home NAT to get to the internal NetBSD box:

Chimera:~ axon$ ssh -oPort=2222 axon@betaweb.h-i-r.net
Password: [my password]
Last login: Tue Oct 14 19:01:57 2008 from localhost
NetBSD 4.0.1 (GENERIC) #0: Wed Oct 8 01:06:02 PDT 2008
Welcome to NetBSD!

[axon@NetBSD]$

You don't need to reverse tunnel to localhost, and you could just as easily use this trick to reverse-tunnel your e-mail (POP/IMAP), VNC to a Windows/Mac desktop or even to an internal web proxy server.

Step 3: Automate!
If you just leave the SSH tunnel up and running, it won't likely last too long because of session timeouts. You'll need some process to start the tunnel. Here's where you can get creative. Obviously, a process involving the use of cron or at would be one way of going about it. A script running in the background (with nohup) or in a screen session could also take care of things nicely. One idea I had was to set up fetchmail to check a dummy e-mail account every 5 minutes or so. If there's new mail, it initiates the tunnel. Send mail. Wait. Tunnel. Another way would be to upload a file to a web or ftp site to trigger it. You could even make it read information from that file to create the tunnel for different protocols. I opted to create a configuration file on the web server that would be downloaded and used.

On the "inside" box, I created this script and called it tunnel.sh:

#!/bin/sh
ssh betaweb.h-i-r.net rm .tunnel
args=`cat .tunnel`
rm .tunnel
ssh $args

Then, I added this to my crontab:

*/5 * * * * scp axon@betaweb.h-i-r.net:.tunnel ~/.tunnel && ~/scripts/tunnel.sh

Every five minutes, it will try to scp a file called ".tunnel" from the "outside" box. If it was successful, it runs the tunnel.sh script.

The tunnel.sh script removes the .tunnel file from the remote box (so that it doesn't try to spawn multiple tunnels every 5 minutes) and then uses the contents of .tunnel as the arguments for ssh.

I create a file on my laptop named ".tunnel" and add the following:

axon@betaweb.h-i-r.net -g -R 2222:localhost:22 sleep 600

Then, I upload it to the location and wait about 5 minutes. Then, I ssh to port 2222 of betaweb.h-i-r.net. The "sleep 600" is executed on the remote end, which means it will only stay connected for 10 minutes (600 seconds).

Tada!

chimaera:~ axon$ scp .tunnel axon@betaweb.h-i-r.net:
axon@betaweb.h-i-r.net's password:
stdin: is not a tty
.tunnel 100% 58 0.1KB/s 00:00

[ wait 5 minutes ... cue Jeopardy theme song ]

chimaera:~ axon$ ssh -oPort=2222 axon@betaweb.h-i-r.net
Password: [my password]
Last login: Tue Oct 14 23:24:10 2008 from localhost
NetBSD 4.0.1 (GENERIC) #0: Wed Oct 8 01:06:02 PDT 2008
Welcome to NetBSD!

[axon@NetBSD]$

Considerations and Risks:
I've already stated that using public keys without a password is risky business.

The other shady business is directly executing anything using arguments pulled from some file on a remote server. All it would take is "; [insert evil command here]" added to the .tunnel file and havoc can potentially be wreaked on that internal box.

If you can get away with it, BY ALL MEANS use separate accounts that are not used for anything other than this tunneling process. Lock them down as best you can and be mindful of local privilege escalation attacks.

See? This is why admins put firewalls up in the first place. And don't go pointing your finger at me if you get in trouble for unauthorized tunneling.

Sysadmin Sunday: Quick & Dirty SSH Tunneling

Occasionally, you might need to tunnel some other traffic over SSH. This could be to get access to an external web proxy, to get a remote X display up, or to get around a firewall-blocked port that you must access.

The syntax (on the command-line OpenSSH client) for a Local forward is like this:

# ssh remote-example.h-i-r.net -L 3128:localhost:3128

This tells my SSH client to tunnel traffic to port 3128 on my workstation to port 3128 on my DMZ box. Port 3128 isn't accessible because of a firewall, but tunnelled over SSH it works fine. In this case, I'm running squid on the remote example host. Telling Firefox to use http://localhost:3128 as the proxy will now tunnel all of my web traffic over the SSH tunnel to the squid proxy behind the firewall. The reason it's called a local forward because it forwards a local port over the SSH connection.

A remote forward will open up a port on the remote machine and connect it to a port on the client's network. The syntax is similar:

# ssh remote-example.h-i-r.net -R 3306:dbserver:3306

This would open up port 3306 (the MySQL server port) on the remote host and tunnel it to the MySQL service on the host named dbserver on my local network.

While running forwarding of either type, you can enter the hotkey sequence "~#" to see all the open connections through the forwarded ports.

Sysadmin Sunday: Automate remote tasks with SSH

The SSH Suite can be leveraged to perform some pretty amazing tasks. I maintain hundreds upon hundreds of open systems as part of my job, but a good deal of my experience with SSH for batch processing came from a brief stint at a startup company that had employed a few hundred OpenBSD systems in an application-clustered environment. Keeping them in sync wasn't too hard once I came up with a few creative ways to get everything straight.

Currently, the most popular (and arguably most secure) SSH suite is OpenSSH which ships with almost all Open-Source operating systems and some commercial ones. Packages exist for most others, or there's always the source code.

For the un-initiated, SSH itself replaces RSH and Telnet as remote-access protocols for interactive logins. SFTP replaces the insecure FTP protocol with an encrypted, SSH-tunneled file transfer protocol. SCP replaces RCP with a secure remote copy protocol. At its core, SSH can simply authenticate these tools against the host's existing password database. The real power of SSH comes from its ability to use public keys to authenticate accounts. This kind of authentication is fragile if implemented improperly. If one account is compromised and the account has weakly-protected keys distributed to other systems, the impact of that one account can be much further-reaching. It is for this reason that I recommend using un-privileged accounts when using passwordless SSH keys.

Generating SSH Keys
To generate SSH keys for your account, you login to the account on the system that you'll be running automated or batch tasks from, then run ssh-keygen and follow the prompts, entering a blank password for the key.

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/axon/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/axon/.ssh/id_rsa.
Your public key has been saved in /home/axon/.ssh/id_rsa.pub.
The key fingerprint is:
5c:f4:64:3c:51:a3:a7:80:fd:fa:55:7a:06:f7:4d:84 axon@chimerabsd.labs.h-i-r.net

Distributing SSH keys
From here, you have to push the key out to each remote host by adding the contents of the id_rsa.pub file to the end of "authorized_keys" in the .ssh directory of the account you're going to be connecting to. My usual method goes something like this:

ssh user@remote.host "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub

If this returns an error such as ".ssh/authorized_keys: No such file or directory", then you need to go to the remote host and create the .ssh directory manually.  This ssh'es to the remote account, and runs "cat >> .ssh/authorized_keys", which takes any input and appends it to the authorized_keys file. At the same time, it's sending the contents of of id_rsa.pub to be appended.  

The above command will prompt you for a password for the remote user, then exit immediately. From that point on "ssh user@remote.host" should no longer ask for a password, and then the account can be used for remote automation. Note: You can also enter a password during key generation, and this password will then be prompted for when ssh'ing to remote hosts. This hinders automation, however.

I chose, instead, to make a script to push the keys out to multiple systems, since I'll be demonstrating batch processing of remote commands.  First, come up with a list of all the remote systems you wish to effect at the same time.  I'm going to choose from a subset of HiR Lab boxes that I've got running right now:

---

File: labs.h-i-r.net.list

---

backtrack.labs.h-i-r.net
bouncer.labs.h-i-r.net
chimera.labs.h-i-r.net
chimerabsd.labs.h-i-r.net
lampdev.labs.h-i-r.net

---

The key push script:

---

File: key.pu.sh

---

#!/bin/sh
# mass ssh key push script by ax0n@h-i-r.net 2008-01-06
# Pushes ssh keys out to an entire list of hosts from a file
if [ -z $1 ]
then
echo "Syntax: $0 server-list "
exit 1
fi
boxlist=$1
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub done

---

Then run key.pu.sh [server-list-file].  It will ask for passwords for each machine.  That's the breaks, kid.

-bash-3.2$ ./key.pu.sh labs.h-i-r.net.list
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
axon@backtrack.labs.h-i-r.net's password:
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
axon@bouncer.labs.h-i-r.net's password:
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
Password:
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
axon@chimerabsd.labs.h-i-r.net's password:
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
axon@lampdev.labs.h-i-r.net's password:

Testing the SSH keys

Now that the keys are pushed, remote work will be simple.  Try to ssh to any of the systems in the list.  It should just work without a password:

-bash-3.2$ ssh axon@backtrack.labs.h-i-r.net
Last login: Sun Jan 6 22:34:20 2008 from 192.168.0.107
Linux 2.6.21.5.
backtrack ~ $ logout
Connection to backtrack.labs.h-i-r.net closed.

If you wish to run a single command remotely, then exit, you can also add the command to the end of the command line like this:

-bash-3.2$ ssh bouncer.labs.h-i-r.net uname -a
OpenBSD bouncer.labs.h-i-r.net 4.2 GENERIC#851 sparc

Similar to my key push script is one that I wrote to run a command across all of the systems in a list file:

---

File: mass-ssh.sh

---

#!/bin/sh
# mass ssh script by ax0n@h-i-r.net 2008-01-06
# Runs the same command across an entire list of hosts from a file

if [ -z $2 ]
then
echo "Syntax: $0 server-list command-string"
exit 1
fi
boxlist=$1
shift
commandstring=$*
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "$*"
done

---

Let's see it in action by running the "uptime" command across my lab environment:

-bash-3.2$ ./mass-ssh.sh labs.h-i-r.net.list uptime
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
22:53:40 up 5 days, 5:21, 2 users, load average: 0.24, 0.05, 0.02
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
10:07PM up 2:31, 0 users, load averages: 0.28, 0.15, 0.10
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
16:53 up 1 day, 6:02, 3 users, load averages: 1.44 1.32 1.15
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
10:54AM up 2:32, 1 user, load averages: 0.32, 0.14, 0.10
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
16:53:19 up 2:08, 0 users, load average: 0.08, 0.02, 0.01

Notice how I don't get prompted for any passwords.  It's important to note that my account on these systems is a non-privileged account without access to sudo or membership in any special system groups such as root, wheel, adm, or bin.  You could use this script with a privileged account for adding or removing users, but I'd keep very tight security on any system with passwordless ssh keys.  It's mostly useful for seeing who is logged in, checking for high system loads, and whatnot.

Transferring files

sftp works much like ftp.  The syntax is "sftp user@host.name" and then it either prompts for a password, or if SSH keys are present, uses those.  After that, most of your usual FTP commands work as usual. In the following example, I do not have SSH keys from chimera (my laptop) to itself (localhost):

chimera$ sftp axon@localhost
Connecting to localhost...
Password:
sftp> ls
Desktop Documents Downloads Library Movies Music Photos
Pictures Public Sites myphotos scp.sh
sftp> cd Photos
sftp> ls
06-06-07_1911.jpg 06-25-07_0600.jpg PowerLinkChain.jpg
sftp> get PowerLinkChain.jpg
Fetching /Users/axon/Photos/PowerLinkChain.jpg to PowerLinkChain.jpg
/Users/axon/Photos/PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

Better yet, in my opinion, is scp, the secure replacement for rcp. The syntax is simply the same as "cp" with the exception of it accepting a user and host name and a colon in front of the filename. This allows you to copy files over the network via an encrypted channel from the command-line. I'll do the same thing as before, from chimerabsd (which has ssh keys to chimera) to my home directory locally:

-bash-3.2$ scp axon@chimera.labs.h-i-r.net:Photos/PowerLinkChain.jpg ~
PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

And as you guessed, I also have a program for pushing a file to a list of hosts as well.

---

File: pu.sh

---

#!/bin/sh
# mass scp push script by ax0n@h-i-r.net 2008-01-06
# uses scp to push one file out to an entire list of hosts from a file
if [ -z $2 ]
then
echo "Syntax: $0 file server-list [remote-path]"
exit 1
fi
boxlist=$2
pushfile=$1
rempath=$3
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
scp $file $box:$rempath
done

---

Troubleshooting
If you have trouble getting SSH keys to work, make sure the "PubkeyAuthentication" option in the sshd configuration file (usually /etc/ssh/sshd_config) is set to "yes" and un-commented.

In Closing

My scripts were quick and dirty examples -- use the scripts as a template for more complex tasks, such as uploading and unpacking a tarball nightly, running a database backup via mysqldump, or any other thing you can imagine.  You may wish to add an option for an alternate username, or hard-code a username into your own scripts.  Once SSH keys are in place, you can automate many non-interactive tasks via Cron or At as well.

Sysadmin Sunday: A somewhat secure quick and dirty network backup with ssh, rsync, tar and cron

Greetings and salutations.

In this article I will cover a method of backing up a small number of servers to a central storage server for daily, weekly and monthly archiving. This is good for under 10 machines needing a basic backup solution. It is not meant for large installations due to limitations of rsync and space considerations.

-=- Index -=-

1...... Summary
2...... Description of the process
3...... Server Setup Details
4...... Client setup details
5...... Sripts
6...... Security Concerns
7...... Informative resources

1. Summary
This article is intended for system administrators with intermediate experience with Un*x environments. You will require a server with a large stable amount of space. Bonus points for raid and other storage redundancy features. You will also need an up to date version of sshd installed on the server as well as tar, gzip and rsync. You also will need up to date versions of ssh and rsync on each of the machines to be backed up. I say somewhat secure because it uses ssh to shuttle the files to the server and thus is granted a level of security from eavesdropping. Its a simple way of backing up a server with out the hassle of implementing a full blown backup server software on a multi-unix-platform environment. SSH, rsync, gzip, tar and split are all usually available in a unix or unix-like installation and for the most part compatible with each other. Thus, very little compiling is needed to implement this.

2. Description of the process
Each client will be configured with a time slot and at the given time it will open a ssh connection to the backup server to an un-privileged account. It will then start rsync and synchronize a list of folders with a corresponding set of folders on the remote server. It will then disconnect and continue on its business.

The server, during off-peak hours will perform weekly differential tar backups of the
rsync archive. The server will roll the whole archive into a compressed tar archive and move it to an archive directory either on the server or using some other attached storage.

3. Server Setup Details
SSH needs to be available to the machines needing to backed up. I'm leaving the security details fuzzy here because there are so many ways to secure this setup, setting the ssh server to only use trusted host keys, certain usernames, groups ...etc In this case we are using public key authentication so this option needs to be enabled on the sshd server.

Our backup user is a regular user (no special groups, just the basic usergroup). For this example our backup user is "backupuser" who belongs to group "backupuser".

Our backup user's quota setup and home directory need to have lots of storage space available.

Inside their home directory we need a directory structure similar to this:

-SERVERBACKUPS
- SERVER1
+ Daily
+ Weekly
- SERVER2
+ Daily
+ Weekly
- SERVER3
+ Daily
+ Weekly

NOTE: Security precautions you should look at taking is switching on the no-execute feature on the file system (if the folder resides some where that wont need scripts being executed from). The backup user account needs to have a restricted shell (ex. /bin/rbash). Security is beyond the scope of this article so use your best judgement.

Then configure cron to run your roll scripts. (see section 5)

You will want to run crontab on the backup server as root or some other user with enough permissions to manipulate the backup data owned by our backup user.
Run :

#crontab -e

(It will then run vi or the other default editor)
... add the following lines

# 12:30pm on Saturdays
30 12 * * 6 /root/scripts/weekly_diff.sh
# 12:30pm on Sunday once every month
30 12 * 1-12 0 /root/scripts/monthly_roll.sh

(Save the file and exit the editor)

We will now need to make the ssh private key.

#ssh-keygen -t dsa

***You will not want to put a password on this key (just press enter).

Then place the public key into the backup users ~/.ssh/authorized_keys and chmod 700 on the .ssh directory and chmod 600 on the key itself. (this would be a good time to verify that sshd is configured for public key logins)

You can also put multiple public keys into the authorized keys list (one for each client).

4...... Client setup details

Each client needs to have a copy of the private key we made in section 3. You will want to run chmod 600 on the key to prevent other system users access to the key. Then put the backup.sh script and the private key into a folder accessible only to the root account (like /root/scripts) and run chmod 700 on the script so only the owner (root) can execute it.

make sure the file is where the script expects to to find it

You will then want to put in a cron job for the root account.

#crontab -e

(it will then launch the system default editor like vi)
insert the following commands:

# Backup the fs at 11:30pm every day of the week
30 23 * * * /root/scripts/backup.sh

Save the file and exit. Now it will now execute the /root/backup.sh script at 11:30pm every day.

5...... Scripts

#!/bin/sh
#--------SERVER WEEKLY ARCHIVING SCRIPT (weekly_diff.sh) -----------
#!/bin/bash
DATE=`date +%V%g`
cd /data/serverbackups
for file in *; do
tar --create \
-z \
--file=$file/weekly/$file$DATE.tar.gz \
-g $file/weekly/weekly-diff.snar \
$file/daily
done

#end weekly script

#--------SERVER MONTHLY ARCHIVING SCRIPT (monthly_roll.sh)-----------
#!/bin/bash
DATE=`date +%b%g`
cd /path/to/serverbackups
mkdir /path/to/archive/$DATE
for file in *; do
rm $file/weekly/*
FILENAME=$DATE.tgz
tar -zcvf $file/weekly/$FILENAME -g $file/weekly/weekly-diff.snar $file/daily/.
cp $file/weekly/*.tgz /path/to/archive/$DATE
done

#end monthly script

#--------SERVER OFF SITE ARCHIVING SCRIPT -----------
MOUNT_CMD="/path/to/mount"
MOUNT_DEV="/path/to/external/storage/device"
MOUNT_POINT="/path/to/mount/point"
MOUNT_FS="filesystem name -t "
MKNOD_CMD="/path/to/mknod /tmp/tar_pipe p"
SPLIT_CMD="/path/to/split -b 512000000"
ARC_PATH="/path/to/monthly/archive"
TAR="/path/to/tar -cvf"
GZ="/path/to/gzip"
DATE=`/path/to/date +%m%Y`

$MOUNT_CMD $MOUNT_FS $MOUNT_DEV $MOUNT_POINT
mkdir $MOUNT_POINT/$DATE
$TAR $MOUNT_POINT/$DATE/archive_$DATE.tgz /tmp/tar_pipe &
$SPLIT_CMD /tmp/tar_pipe $MOUNT_POINT/$DATE/archive_backup_$DATE.tar.
#end of archive script

#------------------CLIENT BACKUP.SH-------------------
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/path/to/key
RUSER=backupuser
#SERVER IP
RHOST="11.22.33.44"
RPATH="/path/to/serverbackups/servername/daily"
LPATH=/

$RSYNC -az --links --safe-links --exclude /dev --exclude /proc --exclude /mnt $LPATH -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH


#end of backup.sh

6...... Security Concerns
Obviously, having an account which contains all of your network data available to any one who has the secure key is a problem. Having an rsynced archive of everything also has other file related issues. You could use different backup server local users, keys or tack on some countermeasures which chmod the files to something less offensive.

Then we have concerns about ssh itself. Internet sites are vulnerable to scripted ssh probes using dictionary attacks ..etc
You could move ssh to a different port and avoid some of the scripted attacks. Once again this is all outside the scope of this article.

If these concerns are a bit too much for your environment consider using a backup system like Baccula or Amanda or one of the commercial backup solutions.

7...... Informative resources

Johnson, Troy. "Using Rsync and SSH." Accessed December 2007
http://troy.jdmz.net/rsync/index.html

Linux-Backup.net "Examples." Accessed December 2007
http://www.linux-backup.net/Example

OpenSSH.org "Manuals." Accessed December 2007
http://openssh.org/manual.html