Reverse SSH Tunnel Watchdog

We've covered tunneling before on HiR. I even wrote a little about reverse tunneling in my quick-and-dirty tunneling howto. This time, I'm building a setup to make an always-on reverse tunnel with a cron-powered watchdog script. I've even coded a cron watchdog before, but this way is less hackish, in my opinion.

Here's what's needed to make it work:

• A Linux/BSD/Unix system on the inside of your target network that's capable of SSH-ing out to the Internet (even if via strange ports)
  
• An SSH server you control on the Internet. This can be at home, or elsewhere

The reverse tunnel can handle pretty much any protocol. To a squid proxy, for example. I'll be using SSH to reverse-tunnel SSH, though, to allow SSH access to a server behind a firewall that I do not control. Here's how it'll work:

Cron will run a script on the server on the inside. This script will check to see if the SSH tunnel is working properly. If it's not (or if it hasn't been started yet), it will start the reverse tunnel.

Here's the script, which I put in /usr/local/bin/rtunnel.sh. Obviously, you need to edit the first 4 variables to reflect your environment.

#!/bin/sh
USERHOST=axon@somewhere.labs.h-i-r.net # Login and External system
RPORT=22 # SSH Listener port on your external system
FPORT=1337 # Port that will be opened locally to tunnel SSH
CONN=localhost:22   # SSH Listener on the system behind the firewall

COMMAND="ssh -q -N -R $FPORT:$CONN $USERHOST -p $RPORT"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
ssh $USERHOST -p $RPORT netstat -an | egrep \
"tcp.*:$FPORT.*LISTEN">/dev/null 2>&1
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi

The above essentially runs this as the command line:

ssh -q -N -R 1337:localhost:22 axon@somewhere.labs.h-i-r.net -p 22

Then uses pgrep (ps command with grep functionality) to see if it's up and running, and tries to ssh to the outside system, using netstat to check the status on that end. If either of those fail, the process is killed with pkill (pgrep, with kill functionality) and restarted.

And then I put the entry in root's crontab, to run every 5 minutes:

*/5     *       *       *       *       /usr/local/bin/rtunnel.sh

In my implementation, the firewalled host (An OpenBSD box) is connecting to a Ubuntu desktop system at my home. I can just log into it, then use the tunnel on port 1337, using the -p [port] option.

axon@somewhere:~$ ssh localhost -p 1337
axon@localhost's password:
Last login: Sat Nov 14 00:01:04 2009 from localhost.labs.h-i-r.net
OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

-bash-3.2$

If the connection times out or fails for any other reason, the remote end should re-spawn the connection in the next 5 minutes. If you've waited, and don't get a response, something else might be amiss. DNS rules, a network admin that's blocked you, etc... You can try switching the port SSH runs on

The one problem I've had is that occasionally the session will be alive, the port will be forwarded, but I can't get it to log in. It just hangs then times out. If this happens, I use lsof on my workstation to find and kill off the process that's listening on the TCP port I am using for forwarding.

axon@somewhere:~$ sudo lsof -n | grep TCP.*:1337
sshd   20386   axon    9u   IPv6    2862057  TCP [::1]:1337 (LISTEN)
sshd   20386   axon   10u   IPv4    2862058  TCP 127.0.0.1:1337 (LISTEN)
axon@somewhere:~$ kill 20386

Of course, you can also tunnel stuff over this reverse tunnel. The possibilities are endless!

Gustav, the hackerspace twitter-bot

Early on in Cowtown Computer Congress' progress, Gustav became our official mascot. When Jestin bought Gustav at a garage-sale, he was a "butler" statue that held a tray. This was probably for halloween candy. Gustav's primitive electronics could sense people nearby with a photocell and do simple actions like breathe and move his eyes. Mostly, though, he just looked kind of cool. We formally adopted Gustav as Professor Emeritus of our hackerspace.

Over the last year or so, we've changed his clothes, added accessories, "facial hair", a remote speaker with a voice changer and swapped out all his circuitry with some homebrew stuff. Usually, Gustav just sits there with a smirk on his face, watching over the hackerspace. Sometimes, he finds his way over to the door and startles people when they first enter the space too. Such an ornery guy.

As CCCKC's official mascot, he has a twitter account. That's in addition to the CCCKC twitter account that's used more for CCCKC-related news.

Since Gustav watches over our hackerspace, I thought it appropriate to empower him to let others know when people are hanging out with him. This is done with a motion sensor. The first time Gustav sees activity, he will tweet about it. As long as people keep moving around, he observes stoically. When the hackerspace remains idle (currently, I'm using 30 minutes as the time-out), he tweets again to notify others that things have gone quiet. I didn't want to clutter the CCCKC twitter feed with such minutiae as the comings-and-goings of hackers on a daily basis, so Gustav chronicles their activity dutifully in his own feed.

This solves the problem of passively letting others know when there's something going on, but keeps privacy at a maximum. Without calling or visiting, there's no way to tell who is doing what at CCCKC, just that there's something going on. This solution avoids the problems posed by public-access webcams and other solutions that might give away too much information for some peoples' comfort.

Hardware

• An old 1U Rackmount x86 system I had laying around
• X10 TM751 Transceiver
• X10 CM11A Bi-Directional Serial Interface
• X10 MS13A "Hawkeye" motion sensor

The Hawkeye motion sensors are pretty weak. I eventually want to buy six more ($60 total at the evil website that sells them) so that full coverage can be had for all the rooms at CCCKC. They simply send an RF signal to the X10 Transceiver.

The signal is then placed on the electrical system for any peripherals to detect. In this case, the only peripheral for now is the CM11A serial interface. I could have the motion detector turn lights on and off, sound a chime, or perform several other actions if I wanted. For now, I'm interested in getting the motion detector input to the computer.

The CM11A can sense X10 network data on the electrical system, and can also send X10 network data as well.

Software

• OpenBSD 4.5
• Heyu 2.8.0
• curl 7.19.3

OpenBSD is a spartan operating system that works well on slow systems. It has a minimal installation footprint but maintains a rich developer environment for compiling software. It was chosen because it was already installed on the 1U system I am using for the project, and because I've already used Heyu on OpenBSD in the past without any problems.

Curl is a command line tool for transferring files with URL syntax. It's lightweight and works well. Its only job will be to update Twitter from within a shell script.

Heyu is a software package with the ability to make sense of the X10 data and act on it. It's quite extensible, but I'm only using it to trigger a shell script.

Configuration

Compiling and installing heyu on OpenBSD is straight-forward. Unpack the tarball, run "make" and then as root, run "make install"

This is the heyu configuration file I put together. There isn't a default configuration file installed, so heyu isn't "install and go" by any means. This file is /etc/heyu/x10config

TTY             /dev/tty00
HOUSECODE       C
SCRIPT_MODE  HEYUHELPER

The HEYUHELPER Script Mode just tells heyu to look for a script in the path called "heyuhelper" and run it. It passes some X10 parameters in the arguments, but for the time being, I am not using them. The above configuration is almost the simplest one you can put together and have a working Heyu install.

To make heyu start automatically at boot, I placed the following in /etc/rc.local:

/usr/local/bin/heyu -c /etc/heyu/x10config start

The "heyuhelper" script mentioned above, is just a quick line of shell in /usr/local/bin/heyuhelper that appends an epoch timestamp to a log file. This can be extended quite a bit to address individual sensors for determining which rooms are in use. For now, I'm keeping it simple. As configured, any X10 trigger on the house-code Heyu is monitoring will append a timestamp to the log file.

#!/bin/sh
date +%s >> /var/log/motion.log

The final piece of the puzzle is a script: /usr/local/bin/cavecheck.sh, that runs from cron.

#!/bin/sh
curdtme=`date +%s`
lastdtme=`tail -1 /var/log/motion.log`
dif=`expr $curdtme - $lastdtme`

if [ "$dif" -lt 1800 ]
then
if [ ! -e "/var/log/caveactive" ]
then
/usr/local/bin/curl --basic --user "username:somepass" \
-d status="#ccckc: Hackers are in the cave!" \
http://twitter.com/statuses/update.xml
touch /var/log/caveactive
fi
fi

if [ "$dif" -gt 1800 ]
then
if [ -e "/var/log/caveactive" ]
then
/usr/local/bin/curl --basic --user "username:somepass" \
-d status="#ccckc is kinda quiet..." \
http://twitter.com/statuses/update.xml
rm /var/log/caveactive
fi
fi

The cron entry itself is pretty easy. I added this to /var/cron/tabs/root so that it runs once every minute.

*       *       *       *       *       /usr/local/bin/cavecheck.sh

Once configured, I rebooted the system to make sure that everything came up automatically the way it should. If you're not down for that, simply sending a HUP signal to cron and starting heyu manually should work fine.

Once I get more motion sensors and all of the rooms are being monitored, I'll probably turn the timeout down to 15 minutes or less.