Reverse SSH Tunnel Watchdog

We've covered tunneling before on HiR. I even wrote a little about reverse tunneling in my quick-and-dirty tunneling howto. This time, I'm building a setup to make an always-on reverse tunnel with a cron-powered watchdog script. I've even coded a cron watchdog before, but this way is less hackish, in my opinion.

Here's what's needed to make it work:

• A Linux/BSD/Unix system on the inside of your target network that's capable of SSH-ing out to the Internet (even if via strange ports)
  
• An SSH server you control on the Internet. This can be at home, or elsewhere

The reverse tunnel can handle pretty much any protocol. To a squid proxy, for example. I'll be using SSH to reverse-tunnel SSH, though, to allow SSH access to a server behind a firewall that I do not control. Here's how it'll work:

Cron will run a script on the server on the inside. This script will check to see if the SSH tunnel is working properly. If it's not (or if it hasn't been started yet), it will start the reverse tunnel.

Here's the script, which I put in /usr/local/bin/rtunnel.sh. Obviously, you need to edit the first 4 variables to reflect your environment.

#!/bin/sh
USERHOST=axon@somewhere.labs.h-i-r.net # Login and External system
RPORT=22 # SSH Listener port on your external system
FPORT=1337 # Port that will be opened locally to tunnel SSH
CONN=localhost:22   # SSH Listener on the system behind the firewall

COMMAND="ssh -q -N -R $FPORT:$CONN $USERHOST -p $RPORT"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
ssh $USERHOST -p $RPORT netstat -an | egrep \
"tcp.*:$FPORT.*LISTEN">/dev/null 2>&1
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi

The above essentially runs this as the command line:

ssh -q -N -R 1337:localhost:22 axon@somewhere.labs.h-i-r.net -p 22

Then uses pgrep (ps command with grep functionality) to see if it's up and running, and tries to ssh to the outside system, using netstat to check the status on that end. If either of those fail, the process is killed with pkill (pgrep, with kill functionality) and restarted.

And then I put the entry in root's crontab, to run every 5 minutes:

*/5     *       *       *       *       /usr/local/bin/rtunnel.sh

In my implementation, the firewalled host (An OpenBSD box) is connecting to a Ubuntu desktop system at my home. I can just log into it, then use the tunnel on port 1337, using the -p [port] option.

axon@somewhere:~$ ssh localhost -p 1337
axon@localhost's password:
Last login: Sat Nov 14 00:01:04 2009 from localhost.labs.h-i-r.net
OpenBSD 4.5 (GENERIC) #1749: Sat Feb 28 14:51:18 MST 2009

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

-bash-3.2$

If the connection times out or fails for any other reason, the remote end should re-spawn the connection in the next 5 minutes. If you've waited, and don't get a response, something else might be amiss. DNS rules, a network admin that's blocked you, etc... You can try switching the port SSH runs on

The one problem I've had is that occasionally the session will be alive, the port will be forwarded, but I can't get it to log in. It just hangs then times out. If this happens, I use lsof on my workstation to find and kill off the process that's listening on the TCP port I am using for forwarding.

axon@somewhere:~$ sudo lsof -n | grep TCP.*:1337
sshd   20386   axon    9u   IPv6    2862057  TCP [::1]:1337 (LISTEN)
sshd   20386   axon   10u   IPv4    2862058  TCP 127.0.0.1:1337 (LISTEN)
axon@somewhere:~$ kill 20386

Of course, you can also tunnel stuff over this reverse tunnel. The possibilities are endless!