Showing posts with label script. Show all posts
Showing posts with label script. Show all posts

2020-02-02

02/02/2020! The first palindrome date since 11/11/1111

It's been almost a decade since I've seen an "only happens every several hundred years" thing floating around that seemed too out-there to be real.

So let's debunk the "first palindrome date since 11/11/1111" shall we? Surely these happen more frequently than that. And for shame not using ISO-8601 format. Fortunately, today also works as an ISO-8601 palindrome: 2020-02-02.

I whipped up a quick bash script to scan for dates n-days before and after today's date looking for palindromes in ISO-8601 format as well as the two other formats that are commonly (ab)used here in the United States, DD-MM-YY and DD-MM-YYYY. It's inefficient, relying on a lot of calls to date(1) and rev(1) but it is what it is.


#!/bin/sh
export i=1
while true
do
# ISO-8601 or GTFO.
idtb=`date -v-${i}d +%Y%m%d`
idtf=`date -v+${i}d +%Y%m%d`

# MM-DD-YYYY format
ydtb=`date -v-${i}d +%m%d%Y`
ydtf=`date -v+${i}d +%m%d%Y`

# MM-DD-YY format
dtb=`date -v-${i}d +%m%d%y`
dtf=`date -v+${i}d +%m%d%y`

if [ "$idtb" -eq "`echo $idtb | rev`" ]
then
echo YYYYMMDD $idtb was a palindrome.
fi

if [ "$idtf" -eq "`echo $idtf | rev`" ]
then
echo YYYYMMDD $idtf will be a palindrome.
fi

if [ "$ydtb" -eq "`echo $ydtb | rev`" ]
then
echo MMDDYYYY $ydtb was a palindrome.
fi

if [ "$ydtf" -eq "`echo $ydtf | rev`" ]
then
echo MMDDYYYY $ydtf will be a palindrome.
fi

if [ "$dtb" -eq "`echo $dtb | rev`" ]
then
echo MMDDYY $dtb was a palindrome.
fi

if [ "$dtf" -eq "`echo $dtf | rev`" ]
then
echo MMDDYY $dtf will be a palindrome.
fi
export i=`expr $i + 1`
done

A quick run for a minute or so shows a lot of palindromes past and future.
MMDDYY 021120 will be a palindrome.
MMDDYY 022220 will be a palindrome.
YYYYMMDD 20211202 will be a palindrome.
MMDDYYYY 12022021 will be a palindrome.
MMDDYY 121121 will be a palindrome.
MMDDYY 122221 will be a palindrome.
MMDDYY 112211 was a palindrome.
MMDDYY 111111 was a palindrome.
YYYYMMDD 20111102 was a palindrome.
MMDDYYYY 11022011 was a palindrome.
MMDDYY 012210 was a palindrome.
MMDDYY 011110 was a palindrome.
YYYYMMDD 20300302 will be a palindrome.
MMDDYYYY 03022030 will be a palindrome.
YYYYMMDD 20100102 was a palindrome.
MMDDYYYY 01022010 was a palindrome.
MMDDYY 031130 will be a palindrome.
MMDDYY 032230 will be a palindrome.  

2011-07-17

Sysadmin Sunday: parse strings with spaces using shell script

I run into this once in a while: I'm trying to perform some operation on a bunch of files or a big line of text, and a space in the filename or text file janks everything up. Take for example all these recordings from a podcast that got batch-named with spaces in them.


Chimera:Recordings axon$ ls
(110) - .mp3 (12) - .mp3 (18) - .mp3 (39) - .mp3 (79) - .mp3
(111) - .mp3 (15) - .mp3 (3) - .mp3 (70) - .mp3

I really don't want spaces in the names. No problem, just use ls -1 (the number one) to list the files on their own line, and use sed or something for renaming them and changing every space to a null character, right?

Chimera:Recordings axon$ for file in `ls -1`
> do mv "$file" `echo $file | sed s/" "//g`
> done
mv: rename (110) to (110): No such file or directory
mv: rename - to -: No such file or directory
mv: rename .mp3 to .mp3: No such file or directory
mv: rename (111) to (111): No such file or directory
mv: rename - to -: No such file or directory
mv: rename .mp3 to .mp3: No such file or directory
[truncated]

That did not go as planned...

There are a few interesting ways to solve this one. The actual reason for this problem is your shell's internal field separator. When iterating over some input (here, the results of "ls -1"), the shell interprets any kind of whitespace as a field separator, including spaces, tabs and newline characters.

Although there are some other clever ways to get around this limitation when dealing with filenames specifically, my favorite solution to this problem works on any whole line of input regardless its source, whether reading a text file and operating on it one line at a time or taking filenames as input from another command such as ls or find. You simply have to use something that can accept spaces and requires a newline character in order to set a variable. Of course, I'm talking about a rather unsavory (but totally viable) use of the read command, which most unixy shell-script writers are familiar with when they require user input. Check it:

Chimera:Recordings axon$ ls -1 | while read file
> do mv "$file" `echo $file | sed s/" "//g`
> done

Chimera:Recordings axon$ ls -1
(110)-.mp3
(111)-.mp3
(12)-.mp3
(15)-.mp3
(18)-.mp3
(3)-.mp3
(39)-.mp3
(70)-.mp3
(79)-.mp3

You can also remap the $IFS variable to contain a newline, but be sure to unset it afterwards (if using BASH, this will set it back to default), or your shell will act differently than you likely expect when you're done. Messing with the internal field separator can be useful for other things (such as parsing /etc/passwd or handling CSV files) but honestly I'd probably be more inclined to use awk for those. If we remap IFS to a newline, our original script that errored out above works just fine.

Chimera:Recordings axon$ IFS=`echo -en "\n\b"`
Chimera:Recordings axon$ for file in `ls -1`
> do mv "$file" `echo $file | sed s/" "//g`
> done
Chimera:Recordings axon$ ls -1
(110)-.mp3
(111)-.mp3
(12)-.mp3
(15)-.mp3
(18)-.mp3
(3)-.mp3
(39)-.mp3
(70)-.mp3
(79)-.mp3
Chimera:Recordings axon$ unset IFS


2010-10-11

Nessus XML parsing with awk

At the office, I use Nessus for automated network scanning and patch auditing. With credentials and proper tuning of the scan policy, Nessus is a very powerful tool for more than skript kiddie network scanning. This leaves me with a whole bunch of data to wade through on a weekly basis.

Usually, I only concern myself with the high-severity issues for weekly reports, then as I have time, I dig deeper into the more trivial problems. Still, this required me to manually open the scan files, filter them by severity, and export the data. I got tired of that and made a quick and really dirty XML parser (.nessus files are XML) with shell and grep. It was horrendously slow.

Andy, a fellow KC2600-er helped me wrap my brain around some of the finer points of awk to make it more efficient. This is slightly modified from the one I use at work, which is part of a bigger script that does other things. I figure it's useful for others who use Nessus regularly. The script is here.

Basically, it stores the HostName tag when it encounters it, then iterates through the lines, storing them temporarily until it runs into a line indicating a high-severity plugin has been triggered (severity level 3), then it spits out the host name and the plugin that was triggered. I probably could write the whole thing in awk, but I wrapped it in a little bit of plain old shell script.

Output looks something like this:


Windows
----------------------------------------------------
x.x.x.19:MS10-062: Vulnerability in MPEG-4 Codec Could Allow Remote Code Execution (975558)
x.x.x.19:Adobe Reader <= 9.3.4 / 8.2.4 CoolType.dll SING Font 'uniqueName' Field Parsing Overflow (APSA10-02)
x.x.x.20:MS10-066: Vulnerability in Remote Procedure Call Could Allow Remote Code Execution (982802)

Mac
----------------------------------------------------
x.x.x.8:Mac OS X AFP Shared Folders Unauthenticated Access (Security Update 2010-006) (uncredentialed check)

Linux
----------------------------------------------------
x.x.x.40:PHP 5.2 < 5.2.14 Multiple Vulnerabilities

2010-10-07

It only happens once every 823 years!

- OR -
Shell Scripting for Pedantry's Sake.


Today's "That can't be true!" moment hit me when I started seeing this making the rounds (in various different paraphrased versions) on Teh Intarwebs:

"This month has 5 Fridays, 5 Saturdays and 5 sundays-Only happens every 823 years!"

Truth be known, I don't really care about how many weekends are in a month except for the fact that I get three paychecks this month. That happens about twice per year, and that's always welcome! Once in a while, though, I just can't help it. I have to disprove something. I figured the easiest way to disprove this particular claim would be to write a shell script that used the "cal" tool, found in every unix variant known to mankind.

For there to be 5 Fridays, Saturdays and Sundays in a single month, there is a basic requirement for a 31-day month that begins on a Friday, and only then will the 31st fall on a Sunday to complete 5 "whole weekends" in one month.

Initially, I was thinking of ways to see what months started on a Friday. That would get me close. It would give me months such as February 2013, which have only 28 days. Then it hit me: Look for any month with a 31st day that falls on Sunday. Using "cal," I can simply roll through the calendar year looking for a line that begins with "31" and guarantee that the month will satisfy the requirements of having five Fridays, Saturdays and Sundays.

So here we go!

#!/bin/sh
ye=2010
mo=1
while true
do
until [ $mo -gt 12 ]
do
cal=`cal $mo $ye | grep ^31`
if [ -z "$cal" ]
then
echo -n ""
else
echo
cal $mo $ye
fi
mo=`expr $mo + 1`
done
mo=1
ye=`expr $ye + 1`
done


Output:

January 2010
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

October 2010
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

July 2011
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

March 2013
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

August 2014
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

May 2015
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31


I don't know. It looks like these things happen more than every 823 years. You can rest easy knowing that it will actually happen a total of 825 times in the next 823 years. Yep, I counted them.

One of the derivatives mentioned October specifically, though. Perhaps this only happens once every 823 Octobers?

Slightly modified, we make the script check Octobers...

#!/bin/sh
ye=2010
mo=10
while true
do
cal=`cal $mo $ye | grep ^31`
if [ -z "$cal" ]
then
echo -n ""
else
echo
cal $mo $ye
fi
ye=`expr $ye + 1`
done

Output:

October 2010
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

October 2021
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

October 2027
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31

October 2032
Su Mo Tu We Th Fr Sa
1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31



Nope. More than a decade at times, but not 823 years.

2009-09-12

Gustav, the hackerspace twitter-bot



Early on in Cowtown Computer Congress' progress, Gustav became our official mascot. When Jestin bought Gustav at a garage-sale, he was a "butler" statue that held a tray. This was probably for halloween candy. Gustav's primitive electronics could sense people nearby with a photocell and do simple actions like breathe and move his eyes. Mostly, though, he just looked kind of cool. We formally adopted Gustav as Professor Emeritus of our hackerspace.

Over the last year or so, we've changed his clothes, added accessories, "facial hair", a remote speaker with a voice changer and swapped out all his circuitry with some homebrew stuff. Usually, Gustav just sits there with a smirk on his face, watching over the hackerspace. Sometimes, he finds his way over to the door and startles people when they first enter the space too. Such an ornery guy.

As CCCKC's official mascot, he has a twitter account. That's in addition to the CCCKC twitter account that's used more for CCCKC-related news.

Since Gustav watches over our hackerspace, I thought it appropriate to empower him to let others know when people are hanging out with him. This is done with a motion sensor. The first time Gustav sees activity, he will tweet about it. As long as people keep moving around, he observes stoically. When the hackerspace remains idle (currently, I'm using 30 minutes as the time-out), he tweets again to notify others that things have gone quiet. I didn't want to clutter the CCCKC twitter feed with such minutiae as the comings-and-goings of hackers on a daily basis, so Gustav chronicles their activity dutifully in his own feed.


This solves the problem of passively letting others know when there's something going on, but keeps privacy at a maximum. Without calling or visiting, there's no way to tell who is doing what at CCCKC, just that there's something going on. This solution avoids the problems posed by public-access webcams and other solutions that might give away too much information for some peoples' comfort.

Hardware
  • An old 1U Rackmount x86 system I had laying around
  • X10 TM751 Transceiver
  • X10 CM11A Bi-Directional Serial Interface
  • X10 MS13A "Hawkeye" motion sensor

The Hawkeye motion sensors are pretty weak. I eventually want to buy six more ($60 total at the evil website that sells them) so that full coverage can be had for all the rooms at CCCKC. They simply send an RF signal to the X10 Transceiver.

The signal is then placed on the electrical system for any peripherals to detect. In this case, the only peripheral for now is the CM11A serial interface. I could have the motion detector turn lights on and off, sound a chime, or perform several other actions if I wanted. For now, I'm interested in getting the motion detector input to the computer.

The CM11A can sense X10 network data on the electrical system, and can also send X10 network data as well.

Software
OpenBSD is a spartan operating system that works well on slow systems. It has a minimal installation footprint but maintains a rich developer environment for compiling software. It was chosen because it was already installed on the 1U system I am using for the project, and because I've already used Heyu on OpenBSD in the past without any problems.

Curl is a command line tool for transferring files with URL syntax. It's lightweight and works well. Its only job will be to update Twitter from within a shell script.

Heyu is a software package with the ability to make sense of the X10 data and act on it. It's quite extensible, but I'm only using it to trigger a shell script.

Configuration
Compiling and installing heyu on OpenBSD is straight-forward. Unpack the tarball, run "make" and then as root, run "make install"

This is the heyu configuration file I put together. There isn't a default configuration file installed, so heyu isn't "install and go" by any means. This file is /etc/heyu/x10config
TTY             /dev/tty00
HOUSECODE C
SCRIPT_MODE HEYUHELPER
The HEYUHELPER Script Mode just tells heyu to look for a script in the path called "heyuhelper" and run it. It passes some X10 parameters in the arguments, but for the time being, I am not using them. The above configuration is almost the simplest one you can put together and have a working Heyu install.

To make heyu start automatically at boot, I placed the following in /etc/rc.local:
/usr/local/bin/heyu -c /etc/heyu/x10config start
The "heyuhelper" script mentioned above, is just a quick line of shell in /usr/local/bin/heyuhelper that appends an epoch timestamp to a log file. This can be extended quite a bit to address individual sensors for determining which rooms are in use. For now, I'm keeping it simple. As configured, any X10 trigger on the house-code Heyu is monitoring will append a timestamp to the log file.
#!/bin/sh
date +%s >> /var/log/motion.log

The final piece of the puzzle is a script: /usr/local/bin/cavecheck.sh, that runs from cron.
#!/bin/sh
curdtme=`date +%s`
lastdtme=`tail -1 /var/log/motion.log`
dif=`expr $curdtme - $lastdtme`

if [ "$dif" -lt 1800 ]
then
if [ ! -e "/var/log/caveactive" ]
then
/usr/local/bin/curl --basic --user "username:somepass" \
-d status="#ccckc: Hackers are in the cave!" \
http://twitter.com/statuses/update.xml
touch /var/log/caveactive
fi
fi

if [ "$dif" -gt 1800 ]
then
if [ -e "/var/log/caveactive" ]
then
/usr/local/bin/curl --basic --user "username:somepass" \
-d status="#ccckc is kinda quiet..." \
http://twitter.com/statuses/update.xml
rm /var/log/caveactive
fi
fi
The cron entry itself is pretty easy. I added this to /var/cron/tabs/root so that it runs once every minute.

*       *       *       *       *       /usr/local/bin/cavecheck.sh
Once configured, I rebooted the system to make sure that everything came up automatically the way it should. If you're not down for that, simply sending a HUP signal to cron and starting heyu manually should work fine.

Once I get more motion sensors and all of the rooms are being monitored, I'll probably turn the timeout down to 15 minutes or less.

2008-12-23

Asmodian's Workbench: Suhosin Hardened PHP extension and patch.

Suhosin is a plug-in and patch for PHP. It places a white-list filter of actions which are allowed. It prevents a pile of PHP exploits from happening such as buffer overflows and certain kinds of injection attacks. You can find it at the Hardened PHP project web-page. It has a number of default items it blocks, one of which is the number of variables it allows to be posted and received.

You can configure it to either block potential attacks a and to log the results in unix syslog. you can also configure it to allow issues to occur and to only log events too.

You can control the Suhosin default values in your php.ini file.

Some php applications use an enormous amount of post variables so the default value (200) is probably too low. As I have explained to my co-worker, getting rid of the plug-in because your script uses too many post variables is probably not the best solution.

The solution in the that event is to modify the maximum request and post vars.

You can also tell suhosin to in the event of encountering a possible attack to run a different script or a http redirect instead. Like perhaps something like this:
(php.ini entry)


[suhosin]
suhosin.filter.action=[302,]http://www.youtube.com/watch?v=Yu_moia-oVI

As you can see this has a number of interesting possibilities.

If you are interested in PHP and AMP (oamp,lamp ...etc) technologies See also:

The hardened PHP project:
http://www.hardened-php.net/suhosin/

Ax0n's OAMP (Apache, Mysql, PHP on OpenBSD) Article:
http://www.h-i-r.net/2008/12/sysadmin-sunday-amp-on-openbsd-44.html

Asmodian X's Name based hosting mini-howto:
http://www.h-i-r.net/2008/10/sysadmin-sunday-apache-name-based.html

The PHP main website:
http://www.php.net/

The Apache webserver website:
http://httpd.apache.org/

2008-12-20

Firefox plugins for security and geeky fun

I don't run too many Firefox plugins, but I really love the ones I do use.  Here's a run-down. The title of each section will link directly to the plugin on the mozilla site.

NoScript 

Even if you don't use Firefox plugins at all, I recommend giving NoScript a try. From the NoScript website:

When you install NoScript, JavaScript, Java, Flash Silverlight and possibly other executable contents are blocked by default. You will be able to allow JavaScript/Java/... execution (scripts from now on) selectively, on the sites you trust. You can allow a site to run scripts temporarily, if you're just surfing randomly, or permanently, when you visit it often and you really trust it. This means that NoScript learns from your own browser habits and tends to disappear in the background after a while, but it promptly comes back to save your day if you stumble upon a malicious web page.
NoScript is updated frequently as malware blocking methods are improved. It was one of the first products to offer protection against clickjacking



FoxyProxy  allows you to set up multiple proxy configurations. This comes in handy when SSH Tunneling to your own proxy or just using public proxies for web filter evasion or privacy reasons.  FoxyProxy is a little unwieldy at first glance, but it's quite flexible; more so than other proxy management plugins. If that's a little over the top for you, a more minimalist plugin is SwitchProxy Tool
Security Reality Check by ax0n: Switching between multiple public proxies every 30 seconds might seem like a good idea for making yourself harder to track, but it also dramatically increases the number of places your traffic goes. You leave more footprints in more places, which could actually make it easier to track something back to you, even if it's harder to figure out everything you did.

Leet Key lets you transform text with a number of popular encoding algorithms, for example, when @lithium posts stuff like this.  Grr. 


Select the text, right-click, then hit the text transformer tool within Leet Key. In this case, it was not only Base64, but rot13 as well. The bad news is that you have to be able to guess what it's encoded with in order to use Leet Key. After having played with many different encoders, I can usually tell what it is that I'm looking at. 


Leet Key also lets you easily encode editable forms, so you can type something into a web mail client or forum posting form, then encode it on the fly before sending it.


User Agent Switcher is for testing how certain sites react to different user-agent strings, but I originally installed it so that I could trick Starbucks' WiFi into thinking I was using an iPhone (and thus, get free WiFi). I've found it useful for other things, though: Particularly when testing heavy JavaScript pages.

Ubiquity

Ubiquity is a command line interface to Mozilla Firefox. This allows you to create small, re-usable custom functions and subscribe to third party functions. If, like me, you find yourself willing and ready to script-automate the repetitive things in life, You'll probably love Ubiquity. 


I'm paranoid by nature, and NoScript is the only plugin I leave enabled all the time. The rest of these I will only enable when I will need them. I'm leery about using a lot of Greasemonkey scripts, and don't really like loading my browser with dozens of add-ons. Do you have some must-have favorites that I'm really missing out on? 

2008-12-19

Asmodian's Workbench: The archive sorter

In the past when I have done backups for home computers, I used a cd-rom burner and dumped everything in a tarball. The problem is that the archive is full of stuff I don't need, want or remember anything about.

So to aid in figuring out whats what I turned to the wonderful unix command FILE.

The file command spits out what file format it thinks a given file is. Its does this through magic numbers.


#!/bin/bash
TEMP_DIR1=`mktemp -d -q /tmp/TMP1.XXXXXX`
TEMP_DIR2=`mktemp -d -q /tmp/TMP2.XXXXXX`
DATE=`date "+%m_%d_%y"`
ARC_FILE=$1
TF=`mktemp -q /tmp/TF.XXXXX`
cp $ARC_FILE $TF.tgz
tar -zx -C $TEMP_DIR1 -f $TF.tgz
NUM1=1
find=`find -X -P $TEMP_DIR1/. -type f`
for FILE in $find; do
TYPE=`file -b $FILE|tr [:space:][:cntrl:] \_`
FILTERED=`basename $FILE|tr [:space:][:cntrl:][:punct:] \.`
FILEFILTERED=`echo $NUM1$FILTERED`
mkdir -p $TEMP_DIR2/$TYPE
cp $FILE $TEMP_DIR2/$TYPE/$FILEFILTERED
echo $FILE >> $TEMP_DIR2/md5_file_list.txt
md5 $FILE >> $TEMP_DIR2/md5_file_list.txt
NUM1=`expr $NUM1 + 1`
done
mv $TEMP_DIR2 ~/recovered$DATE
rm -rf $TEMP_DIR1
rm $TF.tgz
rm $TF



This code has been tested on Mac OS X 10.5 . Linux handles the find and file commands differently. OS X either classifies it as a file (well gee now I know its a normal file... Thanks OS X, you were very helpful there...) or it responds with everything up to and including the picture size. Linux responds with some basics about the file or the whole mime-type, which comes in handy if you want to sub categorize. And I made use of the kick ass TR command (which Ax0n based a previous article about). The find command was useful too but once again there is a syntax difference between OSX and Linux.



Interesting Facts:
Wikipedia: Magic Numbers in files
Apple Man pages

2008-04-09

Shell script for Flickr/Blogger Goodness

Embed your Flickr images into blogger while linking to their page (as opposed to just the image itself). Like this (not my image, just an interesting one I found):



Just edit the flickrbase url in the script, and enjoy. Run the script with the image URL in the command-line, and it gives you the HTML to paste into blogger.

Note, this won't link to other peoples' flickr pages as-is, nor can it tell you the username for any given flickr image. If you want to embed someone else's flickr images, you'll need to edit the flickrbase url to match that of the person whose images you wish to use. Although I don't advise shameless ripping of other peoples' stuff.

Script is available here:
http://stuff.h-i-r.net/blogstuff/fr.sh

2007-12-09

Sysadmin Sunday: A somewhat secure quick and dirty network backup with ssh, rsync, tar and cron

Greetings and salutations.

In this article I will cover a method of backing up a small number of servers to a central storage server for daily, weekly and monthly archiving. This is good for under 10 machines needing a basic backup solution. It is not meant for large installations due to limitations of rsync and space considerations.

-=- Index -=-
1...... Summary
2...... Description of the process
3...... Server Setup Details
4...... Client setup details
5...... Sripts
6...... Security Concerns
7...... Informative resources

1. Summary
This article is intended for system administrators with intermediate experience with Un*x environments. You will require a server with a large stable amount of space. Bonus points for raid and other storage redundancy features. You will also need an up to date version of sshd installed on the server as well as tar, gzip and rsync. You also will need up to date versions of ssh and rsync on each of the machines to be backed up. I say somewhat secure because it uses ssh to shuttle the files to the server and thus is granted a level of security from eavesdropping. Its a simple way of backing up a server with out the hassle of implementing a full blown backup server software on a multi-unix-platform environment. SSH, rsync, gzip, tar and split are all usually available in a unix or unix-like installation and for the most part compatible with each other. Thus, very little compiling is needed to implement this.

2. Description of the process
Each client will be configured with a time slot and at the given time it will open a ssh connection to the backup server to an un-privileged account. It will then start rsync and synchronize a list of folders with a corresponding set of folders on the remote server. It will then disconnect and continue on its business.

The server, during off-peak hours will perform weekly differential tar backups of the
rsync archive. The server will roll the whole archive into a compressed tar archive and move it to an archive directory either on the server or using some other attached storage.

3. Server Setup Details
SSH needs to be available to the machines needing to backed up. I'm leaving the security details fuzzy here because there are so many ways to secure this setup, setting the ssh server to only use trusted host keys, certain usernames, groups ...etc In this case we are using public key authentication so this option needs to be enabled on the sshd server.

Our backup user is a regular user (no special groups, just the basic usergroup). For this example our backup user is "backupuser" who belongs to group "backupuser".

Our backup user's quota setup and home directory need to have lots of storage space available.

Inside their home directory we need a directory structure similar to this:
-SERVERBACKUPS
- SERVER1
+ Daily
+ Weekly
- SERVER2
+ Daily
+ Weekly
- SERVER3
+ Daily
+ Weekly

NOTE: Security precautions you should look at taking is switching on the no-execute feature on the file system (if the folder resides some where that wont need scripts being executed from). The backup user account needs to have a restricted shell (ex. /bin/rbash). Security is beyond the scope of this article so use your best judgement.

Then configure cron to run your roll scripts. (see section 5)

You will want to run crontab on the backup server as root or some other user with enough permissions to manipulate the backup data owned by our backup user.
Run :

#crontab -e


(It will then run vi or the other default editor)
... add the following lines

# 12:30pm on Saturdays
30 12 * * 6 /root/scripts/weekly_diff.sh
# 12:30pm on Sunday once every month
30 12 * 1-12 0 /root/scripts/monthly_roll.sh


(Save the file and exit the editor)

We will now need to make the ssh private key.

#ssh-keygen -t dsa

***You will not want to put a password on this key (just press enter).

Then place the public key into the backup users ~/.ssh/authorized_keys and chmod 700 on the .ssh directory and chmod 600 on the key itself. (this would be a good time to verify that sshd is configured for public key logins)

You can also put multiple public keys into the authorized keys list (one for each client).

4...... Client setup details

Each client needs to have a copy of the private key we made in section 3. You will want to run chmod 600 on the key to prevent other system users access to the key. Then put the backup.sh script and the private key into a folder accessible only to the root account (like /root/scripts) and run chmod 700 on the script so only the owner (root) can execute it.

make sure the file is where the script expects to to find it

You will then want to put in a cron job for the root account.

#crontab -e

(it will then launch the system default editor like vi)
insert the following commands:

# Backup the fs at 11:30pm every day of the week
30 23 * * * /root/scripts/backup.sh

Save the file and exit. Now it will now execute the /root/backup.sh script at 11:30pm every day.

5...... Scripts

#!/bin/sh
#--------SERVER WEEKLY ARCHIVING SCRIPT (weekly_diff.sh) -----------
#!/bin/bash
DATE=`date +%V%g`
cd /data/serverbackups
for file in *; do
tar --create \
-z \
--file=$file/weekly/$file$DATE.tar.gz \
-g $file/weekly/weekly-diff.snar \
$file/daily
done

#end weekly script

#--------SERVER MONTHLY ARCHIVING SCRIPT (monthly_roll.sh)-----------
#!/bin/bash
DATE=`date +%b%g`
cd /path/to/serverbackups
mkdir /path/to/archive/$DATE
for file in *; do
rm $file/weekly/*
FILENAME=$DATE.tgz
tar -zcvf $file/weekly/$FILENAME -g $file/weekly/weekly-diff.snar $file/daily/.
cp $file/weekly/*.tgz /path/to/archive/$DATE
done

#end monthly script
#--------SERVER OFF SITE ARCHIVING SCRIPT -----------
MOUNT_CMD="/path/to/mount"
MOUNT_DEV="/path/to/external/storage/device"
MOUNT_POINT="/path/to/mount/point"
MOUNT_FS="filesystem name -t "
MKNOD_CMD="/path/to/mknod /tmp/tar_pipe p"
SPLIT_CMD="/path/to/split -b 512000000"
ARC_PATH="/path/to/monthly/archive"
TAR="/path/to/tar -cvf"
GZ="/path/to/gzip"
DATE=`/path/to/date +%m%Y`

$MOUNT_CMD $MOUNT_FS $MOUNT_DEV $MOUNT_POINT
mkdir $MOUNT_POINT/$DATE
$TAR $MOUNT_POINT/$DATE/archive_$DATE.tgz /tmp/tar_pipe &
$SPLIT_CMD /tmp/tar_pipe $MOUNT_POINT/$DATE/archive_backup_$DATE.tar.
#end of archive script



#------------------CLIENT BACKUP.SH-------------------
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/path/to/key
RUSER=backupuser
#SERVER IP
RHOST="11.22.33.44"
RPATH="/path/to/serverbackups/servername/daily"
LPATH=/

$RSYNC -az --links --safe-links --exclude /dev --exclude /proc --exclude /mnt $LPATH -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH


#end of backup.sh



6...... Security Concerns
Obviously, having an account which contains all of your network data available to any one who has the secure key is a problem. Having an rsynced archive of everything also has other file related issues. You could use different backup server local users, keys or tack on some countermeasures which chmod the files to something less offensive.

Then we have concerns about ssh itself. Internet sites are vulnerable to scripted ssh probes using dictionary attacks ..etc
You could move ssh to a different port and avoid some of the scripted attacks. Once again this is all outside the scope of this article.

If these concerns are a bit too much for your environment consider using a backup system like Baccula or Amanda or one of the commercial backup solutions.

7...... Informative resources

Johnson, Troy. "Using Rsync and SSH." Accessed December 2007
http://troy.jdmz.net/rsync/index.html

Linux-Backup.net "Examples." Accessed December 2007
http://www.linux-backup.net/Example

OpenSSH.org "Manuals." Accessed December 2007
http://openssh.org/manual.html

2007-10-05

Shell Scripting: friendly command-line arguments

Ah, the joys of shell scripting! If you've spent any time on UNIX-like operating systems, you've probably encountered or written shell scripts. The theory is simple. For the most part, shell scripts simply execute shell commands in order. You find them everywhere. Simply booting a Linux or BSD host might execute scores of shell scripts. Scheduled processes like those launched with cron or at are usually shell scripts. As a system administrator or a hacker, well-programmed scripts can make your life and the lives of your users a lot easier. On the other hand, scripts that are arcane and cryptic can be more trouble than they're worth.

One of the major hang-ups of complex shell scripts is the strict syntax of the command-line arguments. When calling the command-line arguments from within the script, the first argument is referenced as $1, the next as $2 and so on. $0 is the name of the script itself as it was entered on the command line (including the path, if typed). Also, $# is a numeric variable that contains the number of command line arguments passed. Using the exit command is a way to make sure the script stops where it is without processing any further commands. Using exit 0 creates a "clean" exit, whereas exit 1 (or any other integer) is a way to symbolize an error. This doesn't matter much unless other scripts rely on the ones you're making. It's good practice to specify a proper exit status for your scripts, but it's not mandatory.

Most shell scripts that accept arguments require the end-user to know exactly what arguments to pass or they will simply fail. Take, for example, this script I wrote to get my wireless adapter online in OpenBSD.

-------------------------------------------------------------------------------


#!/bin/sh
sudo ifconfig $1 nwid $2 nwkey $3 up
sudo dhclient $1

-------------------------------------------------------------------------------

It requires me to select the device name of my wireless ethernet adapter, the SSID, and the WEP password. The command line could look like this:

wifi.sh ural0 mywlan 0x31337e1ee7

If I just executed wifi.sh without any arguments, the ifconfig would fail miserably on syntax alone, and dhclient would not know what ethernet adapter to use to get an IP address. The script would not work.

Some more advanced scripts will determine if you entered enough arguments. If you did not, it may give you some brief explanation as to what it wants for arguments. The "apachectl" script for controlling the Apache Web Server is a good example of this. If you run it alone, you are shown a list of arguments that it accepts:

usage: apachectl [ start | startssl | stop | restart | graceful |
status | fullstatus | configtest | help ]
<... output truncated ...>

This style of script is fairly straight-forward. You code it to accept one command line argument, referenced as $1 using the case command as shown in this simple example:

-------------------------------------------------------------------------------

#!/bin/sh
if [ -z "$1" ]
then
echo "usage: $0 [ start | stop ]"
exit 1
fi
case $1 in
start)
echo "You chose start!"
;;
stop)
echo "you chose stop!"
;;
*)
echo "I'm sorry, you didn't choose stop or start."
;;
esac

-------------------------------------------------------------------------------

You can fill in the echo commands with whatever you find useful. This is fairly mundane, and doesn't allow you to pass parameters or multiple flags to your script. For those unfamiliar with the "case" command, it's quite simple to use. If the contents of the variable referenced in the "case" line match the expression before the parenthesis, it executes the code on the following lines, and stops processing when it encounters two semicolons.

Let's face it, with just a single "case" structure and some error checking, you won't be writing any truly powerful shell scripts.

Enter "shift". Within a shell script, shift destroys $1 and shifts all the other arguments down by one, and decrements the value in $# by one as well in order to reflect the new (lower) number of command-line arguments left. The contents of $2 become $1, $3 becomes $2, etc. While you might not think that sounds too exciting, it will allow you to pull off some argument-processing trickery with a simple loop to read arguments. Check out this example:

-------------------------------------------------------------------------------

#!/bin/sh
until [ $# == 0 ]
do
case $1 in
foo)
echo "you have selected foo"
shift
;;
baz)
echo "you have selected baz"
shift
;;
bar)
echo "you have selected bar"
shift
;;
zot)
echo "you have selected zot"
shift
;;
*)
#ignore arguments we don't recognize, just shift them and move on
shift
;;
esac
done

-------------------------------------------------------------------------------

The until/do loop above simply keeps running the arguments through the case block until there are 0 arguments left, then exits. If you pass it an argument that is not in the case block, it simply does a shift and ignores it. It runs the arguments in the order we choose.

bash-3.1$ ./foo.sh foo bar
you have selected foo
you have selected bar

In the case of my made-up wireless configuration script, this isn't directly all that helpful. Another thing you can do, however, is run another shift within a case. This allows you to give your script many very flexible command flags, much like other UNIX commands. Using my wifi script as an example, I'll show you how it's done. For arguments that don't require a second parameter (such as -h and -d) just use one shift statement within the case. For arguments that do require a parameter (such as -d, -s, -k, or -p, use two shifts: one before you assign $1 to a variable, and again at the end of the case. Notice that the * catch-all case is simply there to shift un-recognized arguments. If we don't do this, our loop will hang forever because there will always be an argument that hasn't been processed.

This script puts it all together. Its arguments are just as flexible as most compiled programs. After the script has processed all of the arguments, I use a series of if statements to build the command line for ifconfig by appending to the $ifconfig_args variable, then run dhclient if desired.

There's a little extra scripting (more if statements) to make sure that a value follows the arguments that require another parameter. In the end, this is a pretty lengthy script, but it's almost bullet-proof and a lot friendlier than most shell scripts. People might not even know it's a script!

-------------------------------------------------------------------------------

#!/bin/sh
if [ -z $1 ]
then
echo "Try using '$0 -h' for help."
exit 1
fi
until [ $# == 0 ]
do
case $1 in
-h)
help=1
shift
;;
-d)
shift
if [ -z "$1" ]
then
echo "You must specify a device with -d"
exit 2
fi
device=$1
shift
;;
-s)
shift
if [ -z "$1" ]
then
echo "You must specify an SSID with -s"
exit 2
fi
ssid=$1
shift
;;
-p)
shift
if [ -z "$1" ]
then
echo "You must specify a password with -p"
exit 2
fi
password=$1
shift
;;
-k)
shift
if [ -z "$1" ]
then
echo "You must specify a key with -k"
exit 2
fi
wepkey=$1
shift
;;
-c)
dhcp=1
shift
;;
*)
shift
;;
esac
done

if [ "$help" ]
then
echo "Usage: $0 -d -s [-h] [-c] [-p | -k ]"
echo "-d Wireless Ethernet device (wi0, ural0, etc.)"
echo "-s The SSID of the network you wish to join"
echo "-h This help page"
echo "-c Start DHCP client"
echo "-p 5 or 13 character WEP password"
echo "-k 10 or 26 character hexadecimal WEP key"
exit 0
fi
if [ -z "$ssid" ]
then
echo "You must specify an SSID"
exit 1
fi

if [ -z "$device" ]
then
echo "You must specify a device"
exit 1
fi

ifconfig_args="$device nwid $ssid"
if [ "$password" ]
then
ifconfig_args="$ifconfig_args nwkey $password"
fi

if [ "$wepkey" ]
then
ifconfig_args="$ifconfig_args nwkey 0x$wepkey"
fi

ifconfig $ifconfig_args

if [ "$dhcp" ]
then
dhclient $device
fi

exit 0

-------------------------------------------------------------------------------

The ifconfig syntax I used in my examples is fairly platform specific to the BSD family, but you can change it to work on Linux, Solaris, or any other UNIX-like OS.

The UNIX userland contains hundreds of little utilities that can be strung together with scripts and pipes to create very powerful programs without having to spend a lot of time learning a new programming language. Hopefully you don't just learn how to make an ifconfig script out of this, but take what I've written as an example of how to improve your own scripts or inspire you to start creating your own scripts.