Shell script for Flickr/Blogger Goodness

Embed your Flickr images into blogger while linking to their page (as opposed to just the image itself). Like this (not my image, just an interesting one I found):



Just edit the flickrbase url in the script, and enjoy. Run the script with the image URL in the command-line, and it gives you the HTML to paste into blogger.

Note, this won't link to other peoples' flickr pages as-is, nor can it tell you the username for any given flickr image. If you want to embed someone else's flickr images, you'll need to edit the flickrbase url to match that of the person whose images you wish to use. Although I don't advise shameless ripping of other peoples' stuff.

Script is available here:
http://stuff.h-i-r.net/blogstuff/fr.sh

Sysadmin Sunday: A somewhat secure quick and dirty network backup with ssh, rsync, tar and cron

Greetings and salutations.

In this article I will cover a method of backing up a small number of servers to a central storage server for daily, weekly and monthly archiving. This is good for under 10 machines needing a basic backup solution. It is not meant for large installations due to limitations of rsync and space considerations.

-=- Index -=-

1...... Summary
2...... Description of the process
3...... Server Setup Details
4...... Client setup details
5...... Sripts
6...... Security Concerns
7...... Informative resources

1. Summary
This article is intended for system administrators with intermediate experience with Un*x environments. You will require a server with a large stable amount of space. Bonus points for raid and other storage redundancy features. You will also need an up to date version of sshd installed on the server as well as tar, gzip and rsync. You also will need up to date versions of ssh and rsync on each of the machines to be backed up. I say somewhat secure because it uses ssh to shuttle the files to the server and thus is granted a level of security from eavesdropping. Its a simple way of backing up a server with out the hassle of implementing a full blown backup server software on a multi-unix-platform environment. SSH, rsync, gzip, tar and split are all usually available in a unix or unix-like installation and for the most part compatible with each other. Thus, very little compiling is needed to implement this.

2. Description of the process
Each client will be configured with a time slot and at the given time it will open a ssh connection to the backup server to an un-privileged account. It will then start rsync and synchronize a list of folders with a corresponding set of folders on the remote server. It will then disconnect and continue on its business.

The server, during off-peak hours will perform weekly differential tar backups of the
rsync archive. The server will roll the whole archive into a compressed tar archive and move it to an archive directory either on the server or using some other attached storage.

3. Server Setup Details
SSH needs to be available to the machines needing to backed up. I'm leaving the security details fuzzy here because there are so many ways to secure this setup, setting the ssh server to only use trusted host keys, certain usernames, groups ...etc In this case we are using public key authentication so this option needs to be enabled on the sshd server.

Our backup user is a regular user (no special groups, just the basic usergroup). For this example our backup user is "backupuser" who belongs to group "backupuser".

Our backup user's quota setup and home directory need to have lots of storage space available.

Inside their home directory we need a directory structure similar to this:

-SERVERBACKUPS
- SERVER1
+ Daily
+ Weekly
- SERVER2
+ Daily
+ Weekly
- SERVER3
+ Daily
+ Weekly

NOTE: Security precautions you should look at taking is switching on the no-execute feature on the file system (if the folder resides some where that wont need scripts being executed from). The backup user account needs to have a restricted shell (ex. /bin/rbash). Security is beyond the scope of this article so use your best judgement.

Then configure cron to run your roll scripts. (see section 5)

You will want to run crontab on the backup server as root or some other user with enough permissions to manipulate the backup data owned by our backup user.
Run :

#crontab -e

(It will then run vi or the other default editor)
... add the following lines

# 12:30pm on Saturdays
30 12 * * 6 /root/scripts/weekly_diff.sh
# 12:30pm on Sunday once every month
30 12 * 1-12 0 /root/scripts/monthly_roll.sh

(Save the file and exit the editor)

We will now need to make the ssh private key.

#ssh-keygen -t dsa

***You will not want to put a password on this key (just press enter).

Then place the public key into the backup users ~/.ssh/authorized_keys and chmod 700 on the .ssh directory and chmod 600 on the key itself. (this would be a good time to verify that sshd is configured for public key logins)

You can also put multiple public keys into the authorized keys list (one for each client).

4...... Client setup details

Each client needs to have a copy of the private key we made in section 3. You will want to run chmod 600 on the key to prevent other system users access to the key. Then put the backup.sh script and the private key into a folder accessible only to the root account (like /root/scripts) and run chmod 700 on the script so only the owner (root) can execute it.

make sure the file is where the script expects to to find it

You will then want to put in a cron job for the root account.

#crontab -e

(it will then launch the system default editor like vi)
insert the following commands:

# Backup the fs at 11:30pm every day of the week
30 23 * * * /root/scripts/backup.sh

Save the file and exit. Now it will now execute the /root/backup.sh script at 11:30pm every day.

5...... Scripts

#!/bin/sh
#--------SERVER WEEKLY ARCHIVING SCRIPT (weekly_diff.sh) -----------
#!/bin/bash
DATE=`date +%V%g`
cd /data/serverbackups
for file in *; do
tar --create \
-z \
--file=$file/weekly/$file$DATE.tar.gz \
-g $file/weekly/weekly-diff.snar \
$file/daily
done

#end weekly script

#--------SERVER MONTHLY ARCHIVING SCRIPT (monthly_roll.sh)-----------
#!/bin/bash
DATE=`date +%b%g`
cd /path/to/serverbackups
mkdir /path/to/archive/$DATE
for file in *; do
rm $file/weekly/*
FILENAME=$DATE.tgz
tar -zcvf $file/weekly/$FILENAME -g $file/weekly/weekly-diff.snar $file/daily/.
cp $file/weekly/*.tgz /path/to/archive/$DATE
done

#end monthly script

#--------SERVER OFF SITE ARCHIVING SCRIPT -----------
MOUNT_CMD="/path/to/mount"
MOUNT_DEV="/path/to/external/storage/device"
MOUNT_POINT="/path/to/mount/point"
MOUNT_FS="filesystem name -t "
MKNOD_CMD="/path/to/mknod /tmp/tar_pipe p"
SPLIT_CMD="/path/to/split -b 512000000"
ARC_PATH="/path/to/monthly/archive"
TAR="/path/to/tar -cvf"
GZ="/path/to/gzip"
DATE=`/path/to/date +%m%Y`

$MOUNT_CMD $MOUNT_FS $MOUNT_DEV $MOUNT_POINT
mkdir $MOUNT_POINT/$DATE
$TAR $MOUNT_POINT/$DATE/archive_$DATE.tgz /tmp/tar_pipe &
$SPLIT_CMD /tmp/tar_pipe $MOUNT_POINT/$DATE/archive_backup_$DATE.tar.
#end of archive script

#------------------CLIENT BACKUP.SH-------------------
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/path/to/key
RUSER=backupuser
#SERVER IP
RHOST="11.22.33.44"
RPATH="/path/to/serverbackups/servername/daily"
LPATH=/

$RSYNC -az --links --safe-links --exclude /dev --exclude /proc --exclude /mnt $LPATH -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH


#end of backup.sh

6...... Security Concerns
Obviously, having an account which contains all of your network data available to any one who has the secure key is a problem. Having an rsynced archive of everything also has other file related issues. You could use different backup server local users, keys or tack on some countermeasures which chmod the files to something less offensive.

Then we have concerns about ssh itself. Internet sites are vulnerable to scripted ssh probes using dictionary attacks ..etc
You could move ssh to a different port and avoid some of the scripted attacks. Once again this is all outside the scope of this article.

If these concerns are a bit too much for your environment consider using a backup system like Baccula or Amanda or one of the commercial backup solutions.

7...... Informative resources

Johnson, Troy. "Using Rsync and SSH." Accessed December 2007
http://troy.jdmz.net/rsync/index.html

Linux-Backup.net "Examples." Accessed December 2007
http://www.linux-backup.net/Example

OpenSSH.org "Manuals." Accessed December 2007
http://openssh.org/manual.html

Shell Scripting: friendly command-line arguments

Ah, the joys of shell scripting! If you've spent any time on UNIX-like operating systems, you've probably encountered or written shell scripts. The theory is simple. For the most part, shell scripts simply execute shell commands in order. You find them everywhere. Simply booting a Linux or BSD host might execute scores of shell scripts. Scheduled processes like those launched with cron or at are usually shell scripts. As a system administrator or a hacker, well-programmed scripts can make your life and the lives of your users a lot easier. On the other hand, scripts that are arcane and cryptic can be more trouble than they're worth.

One of the major hang-ups of complex shell scripts is the strict syntax of the command-line arguments. When calling the command-line arguments from within the script, the first argument is referenced as $1, the next as $2 and so on. $0 is the name of the script itself as it was entered on the command line (including the path, if typed). Also, $# is a numeric variable that contains the number of command line arguments passed. Using the exit command is a way to make sure the script stops where it is without processing any further commands. Using exit 0 creates a "clean" exit, whereas exit 1 (or any other integer) is a way to symbolize an error. This doesn't matter much unless other scripts rely on the ones you're making. It's good practice to specify a proper exit status for your scripts, but it's not mandatory.

Most shell scripts that accept arguments require the end-user to know exactly what arguments to pass or they will simply fail. Take, for example, this script I wrote to get my wireless adapter online in OpenBSD.

-------------------------------------------------------------------------------

#!/bin/sh
sudo ifconfig $1 nwid $2 nwkey $3 up
sudo dhclient $1

-------------------------------------------------------------------------------

It requires me to select the device name of my wireless ethernet adapter, the SSID, and the WEP password. The command line could look like this:

wifi.sh ural0 mywlan 0x31337e1ee7

If I just executed wifi.sh without any arguments, the ifconfig would fail miserably on syntax alone, and dhclient would not know what ethernet adapter to use to get an IP address. The script would not work.

Some more advanced scripts will determine if you entered enough arguments. If you did not, it may give you some brief explanation as to what it wants for arguments. The "apachectl" script for controlling the Apache Web Server is a good example of this. If you run it alone, you are shown a list of arguments that it accepts:

usage: apachectl [ start | startssl | stop | restart | graceful |
status | fullstatus | configtest | help ]
<... output truncated ...>

This style of script is fairly straight-forward. You code it to accept one command line argument, referenced as $1 using the case command as shown in this simple example:

-------------------------------------------------------------------------------

#!/bin/sh
if [ -z "$1" ]
then
 echo "usage: $0 [ start | stop ]"
 exit 1
fi
case $1 in
start)
echo "You chose start!"
;;
stop)
echo "you chose stop!"
;;
*)
echo "I'm sorry, you didn't choose stop or start."
;;
esac

-------------------------------------------------------------------------------

You can fill in the echo commands with whatever you find useful. This is fairly mundane, and doesn't allow you to pass parameters or multiple flags to your script. For those unfamiliar with the "case" command, it's quite simple to use. If the contents of the variable referenced in the "case" line match the expression before the parenthesis, it executes the code on the following lines, and stops processing when it encounters two semicolons.

Let's face it, with just a single "case" structure and some error checking, you won't be writing any truly powerful shell scripts.

Enter "shift". Within a shell script, shift destroys $1 and shifts all the other arguments down by one, and decrements the value in $# by one as well in order to reflect the new (lower) number of command-line arguments left. The contents of $2 become $1, $3 becomes $2, etc. While you might not think that sounds too exciting, it will allow you to pull off some argument-processing trickery with a simple loop to read arguments. Check out this example:

-------------------------------------------------------------------------------

#!/bin/sh
until [ $# == 0 ]
do
 case $1 in
 foo)
       echo "you have selected foo"
shift
;;  
 baz)
       echo "you have selected baz"
shift
;;  
 bar)
       echo "you have selected bar"
shift
;;  
 zot)
       echo "you have selected zot"
shift
;;  
 *)
#ignore arguments we don't recognize, just shift them and move on
shift
;;
 esac
done

-------------------------------------------------------------------------------

The until/do loop above simply keeps running the arguments through the case block until there are 0 arguments left, then exits. If you pass it an argument that is not in the case block, it simply does a shift and ignores it. It runs the arguments in the order we choose.

bash-3.1$ ./foo.sh foo bar
you have selected foo
you have selected bar

In the case of my made-up wireless configuration script, this isn't directly all that helpful. Another thing you can do, however, is run another shift within a case. This allows you to give your script many very flexible command flags, much like other UNIX commands. Using my wifi script as an example, I'll show you how it's done. For arguments that don't require a second parameter (such as -h and -d) just use one shift statement within the case. For arguments that do require a parameter (such as -d, -s, -k, or -p, use two shifts: one before you assign $1 to a variable, and again at the end of the case. Notice that the * catch-all case is simply there to shift un-recognized arguments. If we don't do this, our loop will hang forever because there will always be an argument that hasn't been processed.

This script puts it all together. Its arguments are just as flexible as most compiled programs. After the script has processed all of the arguments, I use a series of if statements to build the command line for ifconfig by appending to the $ifconfig_args variable, then run dhclient if desired.

There's a little extra scripting (more if statements) to make sure that a value follows the arguments that require another parameter. In the end, this is a pretty lengthy script, but it's almost bullet-proof and a lot friendlier than most shell scripts. People might not even know it's a script!

-------------------------------------------------------------------------------

#!/bin/sh
if [ -z $1 ]
then
 echo "Try using '$0 -h' for help."
 exit 1
fi
until [ $# == 0 ]
do
 case $1 in
 -h)
   help=1
   shift
   ;;
 -d)
   shift
   if [ -z "$1" ]
   then
     echo "You must specify a device with -d"
     exit 2
   fi
   device=$1
   shift
   ;;
 -s)
   shift
   if [ -z "$1" ]
   then
     echo "You must specify an SSID with -s"
     exit 2
   fi
   ssid=$1
   shift
   ;;
 -p)
   shift
   if [ -z "$1" ]
   then
     echo "You must specify a password with -p"
     exit 2
   fi
   password=$1
   shift
   ;;
 -k)
   shift
   if [ -z "$1" ]
   then
     echo "You must specify a key with -k"
     exit 2
   fi
   wepkey=$1
   shift
   ;;
 -c)
   dhcp=1
   shift
   ;;
 *)
   shift
   ;;
 esac
done

 if [ "$help" ]
 then
   echo "Usage: $0 -d  -s  [-h] [-c] [-p  | -k ]"
   echo "-d       Wireless Ethernet device (wi0, ural0, etc.)"
   echo "-s      The SSID of the network you wish to join"
   echo "-h            This help page"
   echo "-c            Start DHCP client"
   echo "-p  5 or 13 character WEP password"
   echo "-k    10 or 26 character hexadecimal WEP key"
   exit 0
 fi
 if [ -z "$ssid" ]
 then
   echo "You must specify an SSID"
   exit 1
 fi

 if [ -z "$device" ]
 then
   echo "You must specify a device"
   exit 1
 fi

 ifconfig_args="$device nwid $ssid"
 if [ "$password" ]
 then
   ifconfig_args="$ifconfig_args nwkey $password"
 fi

 if [ "$wepkey" ]
   then
   ifconfig_args="$ifconfig_args nwkey 0x$wepkey"
 fi

ifconfig $ifconfig_args

 if [ "$dhcp" ]
   then
   dhclient $device
 fi

exit 0

-------------------------------------------------------------------------------

The ifconfig syntax I used in my examples is fairly platform specific to the BSD family, but you can change it to work on Linux, Solaris, or any other UNIX-like OS.

The UNIX userland contains hundreds of little utilities that can be strung together with scripts and pipes to create very powerful programs without having to spend a lot of time learning a new programming language. Hopefully you don't just learn how to make an ifconfig script out of this, but take what I've written as an example of how to improve your own scripts or inspire you to start creating your own scripts.