On cloud computing

It seems everyone is blaming a general failure of cloud computing for the massive data loss that hit Danger, Microsoft and T-Mobile over the weekend.

From what I've read, a failed storage upgrade occurred without a good, solid backup in place. That sounds a lot more like a failure in backup, planning and design than a failure of cloud computing to me. Had the storage folks at my office made the same mistakes, that's what would have been said -- right before the human resources folks came to "have a talk" with the team.

It just so happens that T-Mobile's sidekick phones rely on a lot of back-end storage, so there's the whole "cloud" element to things. I'm not familiar enough with the Danger platform to know how easy it is to back up your own data, but I'd hope it's possible.

I think it goes for any service where you've entrusted storage of your data to someone else: make sure you back it up yourself, if you think it's important. The difference with the Danger/T-Mo disaster, I think, is that it was a lot less obvious to end-users that the data wasn't all stored permanently on the phone. Clearly, "cloud computing" was collateral damage in the wake of a much more mundane failure. The fact that it was completely avoidable offers little comfort for those affected.

Shifting gears: Along comes this piece on how e-mail is becoming less and less relevant.
The thing that separates e-mail as we know it from other messaging platforms is the fact that e-mail is decentralized. Using information stored in DNS, all Internet-facing e-mail servers can properly send mail to the correct server for a given address. IRC is another decentralized communication protocol. The days of decentralized infrastructure are fading fast, though, being replaced by walled gardens that want your constant attention, and many of them requiring a separate account and password. These walled gardens are supposed to be "the new way" of communicating.

You can't easily backup everything you've received through Twitter or Facebook, and the people who communicate with you there have to have accounts. Sure, anyone can get an account. What about Google Wave? Very few of the people I REALLY want to collaborate with have an account. So, while I do see a lot of value in these services for certain things, I don't think that any of them are quite ready to fill the roll that e-mail currently provides. Chiefly: if I have a local e-mail client running on my system, I don't need to suckle at the teat of the Interwebs in order to rifle through my data. It's right there, on my computer. Web mail has indeed blurred the line, but the good web-mail providers still offer mechanisms to back-up your data or use an offline mail client such as Thunderbird.

OpenID somewhat fixes the need to have multiple accounts and passwords scattered all over the web, but shifting authentication "into the cloud" just means that each OpenID account we have will be more catastrophic if compromised. OpenID is tantamount to using the same username and password everywhere, and we know how well that works for security.

How do you backup your cloud data? Well, for starters, you can try a native-client RSS aggregator such as Liferea. One thing that "Cloud" is doing is making syndication possible through ubiquitous RSS feeds. Backups won't work perfectly on every site, for example: you won't actually download all of the photos from Flickr with RSS, you'll only get links to them. It will nicely archive text content, though. This is good for things such as blog posts, twitter conversations and the like.

Friday Musings: Redundancy

Sorry for the lack of content lately. Work has been crazy, and I had a lazy five-day weekend that I decided to use for vegging out more than geeking out. Whilst camping over the Memorial Day weekend, I did come to appreciate the redundancy built in to Mag Instrument's AA Mini-Mag (and most other lights of theirs), a spare bulb hiding in the tail cap right below the battery spring:


I got this flashlight almost a decade ago as schwag (and seriously nice schwag at that) from Check Point while working at a Check Point Value-Added Reseller as a support technician and penetration tester. The bulb that burned out over this last weekend was the original. I can't even imagine how many hours of burn-time it had, how many drops to the pavement it's taken, or how many times it saved my ass.



I only wish that this kind of redundancy was more common in everyday life from household appliances to consumer-grade electronics. Parts that are most prone to failure should be included. Some old vacuum cleaners even came with a spare drive belt and a nice place to store it.

Now, it seems you only see redundancy addressed when you're dealing with large-scale Enterprise IT and industrial- or military-grade machinery. Otherwise, it's:


Somewhere in history, we've lost our way...

Personal backups for the win!

I might be preaching to the choir here a little bit, but perhaps you need a quick reminder to back up your personal workstations as well as your enterprise data.

After two and a half years of what I can only describe as abuse, my MacBook's hard drive gave up the ghost last night. This thing has traveled over 5,000 miles with me on my bicycle in my panniers, been slung hither to yon on buses, subways, planes and trains. Until recently, it was even my primary workstation at home.

It came with little surprise: my hard drive died mostly due to shock. Not one brute-force blow, by any means. It was just its time. As I sit here letting Time Machine restore my old stuff, I can say I'm thankful that I was at least somewhat diligent about backups. I probably only lost a few photographs that were on my hard drive for the past few days, and some of those got uploaded to Flickr anyway. The only lump in my throat was from having to explain to my wife why I'll need to lay out a Benjamin for a 320GB SATA drive this coming paycheck. It's amazing how much laptop storage you can get for $100. It's too bad I have to do this now, because it looks like the 500GB drives are going to be there soon.

You can read all kinds of reviews about Time Machine as a bare-metal restore tool as well as recovering from accidental data removal. I won't dwell on that here. All I can say is that when the time came, I wasn't sweating any bullets about my lost data. I can simply fork over some cash next week, and everything will be back the way I left it.

However you get your backups done, consider this a reminder to remain diligent. You never know when you'll need to restore, and the backup you made 8 months ago, while better than nothing, probably will leave you with a bit of regret.

OS X: Deleting unwanted files from Time Machine

I was a goofball when I installed VirtualBox. I forgot to disable backups of my VM's. Some might want their Virtual Machines backed up, but as they're mostly just test environments, I consider them disposable. Anything important in the VM gets replicated and synced through Subversion.

The problem is when my virtual machines change, Time Machine suddenly tries to back up the entire virtual drive, gobbling as many as a few gigs of data as one file, and seriously cramping my old backups. Once I told Time Machine not to back up the VirtualBox directory, the old backups (hogging many gigs of backup space) remained on my external hard drive. Trying to get rid of them from the command-line wasn't working, as OS X puts some kind of restrictions on the file system.

The filesystem layout is pretty nifty, as shown below:

Chimera:chimera axon$ cd /Volumes/Time\ Machine\ Backups/Backups.backupdb/chimera/
Chimera:chimera axon$ ls -la
total 8
drwxr-xr-x@ 10 root  staff  374 Sep  9 22:03 .
drwxr-xr-x+  3 root  staff  102 Jul 29 19:53 ..
drwxr-xr-x@  3 root  staff  204 Jul  5 18:52 2008-07-05-185224
drwxr-xr-x@  3 root  staff  204 Jul 29 22:00 2008-07-29-220053
drwxr-xr-x@  3 root  staff  204 Aug  5 21:50 2008-08-05-215007
drwxr-xr-x@  3 root  staff  204 Aug 30 16:53 2008-08-30-165311
drwxr-xr-x@  3 root  staff  204 Sep  5 23:26 2008-09-05-232659
drwxr-xr-x@  3 root  staff  204 Sep  6 11:42 2008-09-06-114242
drwxr-xr-x@  3 root  staff  204 Sep  6 21:38 2008-09-06-213829
drwxr-xr-x@  3 root  staff  204 Sep  9 22:03 2008-09-09-220316
lrwxr-xr-x   1 root  staff   17 Sep  9 22:03 Latest -> 2008-09-09-220316

We won't get too deep into that, though, because it doesn't matter. The answer, oddly, was staring me right in the face... in the graphical Interface... The "Delete All Backups" option shows up under the gear menu, but only when you're browsing your time machine backups.

Obviously, this applies to questionable content and anything else you may have inadvertently backed up, which you no longer wish to be visible through Time Machine.  Keep in mind that forensic eyes can probably see it anyway, and that erasing evidence might be just as good as admitting guilt.

I'm sure there's some way to delete the files manually via the command-line. I'm sure the problems I encountered using "sudo rm [file]" could have been resolved (for example, via xattr), but in the interest of NOT corrupting all of my backups, I guess the GUI will have to suffice for now.

Sysadmin Sunday: A somewhat secure quick and dirty network backup with ssh, rsync, tar and cron

Greetings and salutations.

In this article I will cover a method of backing up a small number of servers to a central storage server for daily, weekly and monthly archiving. This is good for under 10 machines needing a basic backup solution. It is not meant for large installations due to limitations of rsync and space considerations.

-=- Index -=-

1...... Summary
2...... Description of the process
3...... Server Setup Details
4...... Client setup details
5...... Sripts
6...... Security Concerns
7...... Informative resources

1. Summary
This article is intended for system administrators with intermediate experience with Un*x environments. You will require a server with a large stable amount of space. Bonus points for raid and other storage redundancy features. You will also need an up to date version of sshd installed on the server as well as tar, gzip and rsync. You also will need up to date versions of ssh and rsync on each of the machines to be backed up. I say somewhat secure because it uses ssh to shuttle the files to the server and thus is granted a level of security from eavesdropping. Its a simple way of backing up a server with out the hassle of implementing a full blown backup server software on a multi-unix-platform environment. SSH, rsync, gzip, tar and split are all usually available in a unix or unix-like installation and for the most part compatible with each other. Thus, very little compiling is needed to implement this.

2. Description of the process
Each client will be configured with a time slot and at the given time it will open a ssh connection to the backup server to an un-privileged account. It will then start rsync and synchronize a list of folders with a corresponding set of folders on the remote server. It will then disconnect and continue on its business.

The server, during off-peak hours will perform weekly differential tar backups of the
rsync archive. The server will roll the whole archive into a compressed tar archive and move it to an archive directory either on the server or using some other attached storage.

3. Server Setup Details
SSH needs to be available to the machines needing to backed up. I'm leaving the security details fuzzy here because there are so many ways to secure this setup, setting the ssh server to only use trusted host keys, certain usernames, groups ...etc In this case we are using public key authentication so this option needs to be enabled on the sshd server.

Our backup user is a regular user (no special groups, just the basic usergroup). For this example our backup user is "backupuser" who belongs to group "backupuser".

Our backup user's quota setup and home directory need to have lots of storage space available.

Inside their home directory we need a directory structure similar to this:

-SERVERBACKUPS
- SERVER1
+ Daily
+ Weekly
- SERVER2
+ Daily
+ Weekly
- SERVER3
+ Daily
+ Weekly

NOTE: Security precautions you should look at taking is switching on the no-execute feature on the file system (if the folder resides some where that wont need scripts being executed from). The backup user account needs to have a restricted shell (ex. /bin/rbash). Security is beyond the scope of this article so use your best judgement.

Then configure cron to run your roll scripts. (see section 5)

You will want to run crontab on the backup server as root or some other user with enough permissions to manipulate the backup data owned by our backup user.
Run :

#crontab -e

(It will then run vi or the other default editor)
... add the following lines

# 12:30pm on Saturdays
30 12 * * 6 /root/scripts/weekly_diff.sh
# 12:30pm on Sunday once every month
30 12 * 1-12 0 /root/scripts/monthly_roll.sh

(Save the file and exit the editor)

We will now need to make the ssh private key.

#ssh-keygen -t dsa

***You will not want to put a password on this key (just press enter).

Then place the public key into the backup users ~/.ssh/authorized_keys and chmod 700 on the .ssh directory and chmod 600 on the key itself. (this would be a good time to verify that sshd is configured for public key logins)

You can also put multiple public keys into the authorized keys list (one for each client).

4...... Client setup details

Each client needs to have a copy of the private key we made in section 3. You will want to run chmod 600 on the key to prevent other system users access to the key. Then put the backup.sh script and the private key into a folder accessible only to the root account (like /root/scripts) and run chmod 700 on the script so only the owner (root) can execute it.

make sure the file is where the script expects to to find it

You will then want to put in a cron job for the root account.

#crontab -e

(it will then launch the system default editor like vi)
insert the following commands:

# Backup the fs at 11:30pm every day of the week
30 23 * * * /root/scripts/backup.sh

Save the file and exit. Now it will now execute the /root/backup.sh script at 11:30pm every day.

5...... Scripts

#!/bin/sh
#--------SERVER WEEKLY ARCHIVING SCRIPT (weekly_diff.sh) -----------
#!/bin/bash
DATE=`date +%V%g`
cd /data/serverbackups
for file in *; do
tar --create \
-z \
--file=$file/weekly/$file$DATE.tar.gz \
-g $file/weekly/weekly-diff.snar \
$file/daily
done

#end weekly script

#--------SERVER MONTHLY ARCHIVING SCRIPT (monthly_roll.sh)-----------
#!/bin/bash
DATE=`date +%b%g`
cd /path/to/serverbackups
mkdir /path/to/archive/$DATE
for file in *; do
rm $file/weekly/*
FILENAME=$DATE.tgz
tar -zcvf $file/weekly/$FILENAME -g $file/weekly/weekly-diff.snar $file/daily/.
cp $file/weekly/*.tgz /path/to/archive/$DATE
done

#end monthly script

#--------SERVER OFF SITE ARCHIVING SCRIPT -----------
MOUNT_CMD="/path/to/mount"
MOUNT_DEV="/path/to/external/storage/device"
MOUNT_POINT="/path/to/mount/point"
MOUNT_FS="filesystem name -t "
MKNOD_CMD="/path/to/mknod /tmp/tar_pipe p"
SPLIT_CMD="/path/to/split -b 512000000"
ARC_PATH="/path/to/monthly/archive"
TAR="/path/to/tar -cvf"
GZ="/path/to/gzip"
DATE=`/path/to/date +%m%Y`

$MOUNT_CMD $MOUNT_FS $MOUNT_DEV $MOUNT_POINT
mkdir $MOUNT_POINT/$DATE
$TAR $MOUNT_POINT/$DATE/archive_$DATE.tgz /tmp/tar_pipe &
$SPLIT_CMD /tmp/tar_pipe $MOUNT_POINT/$DATE/archive_backup_$DATE.tar.
#end of archive script

#------------------CLIENT BACKUP.SH-------------------
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/path/to/key
RUSER=backupuser
#SERVER IP
RHOST="11.22.33.44"
RPATH="/path/to/serverbackups/servername/daily"
LPATH=/

$RSYNC -az --links --safe-links --exclude /dev --exclude /proc --exclude /mnt $LPATH -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH


#end of backup.sh

6...... Security Concerns
Obviously, having an account which contains all of your network data available to any one who has the secure key is a problem. Having an rsynced archive of everything also has other file related issues. You could use different backup server local users, keys or tack on some countermeasures which chmod the files to something less offensive.

Then we have concerns about ssh itself. Internet sites are vulnerable to scripted ssh probes using dictionary attacks ..etc
You could move ssh to a different port and avoid some of the scripted attacks. Once again this is all outside the scope of this article.

If these concerns are a bit too much for your environment consider using a backup system like Baccula or Amanda or one of the commercial backup solutions.

7...... Informative resources

Johnson, Troy. "Using Rsync and SSH." Accessed December 2007
http://troy.jdmz.net/rsync/index.html

Linux-Backup.net "Examples." Accessed December 2007
http://www.linux-backup.net/Example

OpenSSH.org "Manuals." Accessed December 2007
http://openssh.org/manual.html