Sysadmin Sunday: Squid Cache

What is Squid?
Aside from something Bruce Schneier writes about every Friday, "Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on Unix and Windows and is licensed under the GNU GPL." -- taken from Squid-Cache.org, as I couldn't explain it any better myself.

On your network, you can set up Squid on a reasonably well-endowed server for the purposes of manipulating certain Internet traffic - specifically HTTP, usually for the purpose of increasing load times on frequently-visited content. There are other things you can do with Squid, such as blocking advertisements or turning all of the images upside down (or is that umop ap!sdn). For now, though, we'll set up a Squid server in our lab on our AMP (Apache/MySQL/PHP) fun box, running Gutsy server.

Getting and Installing Squid
Many operating systems' package installers have Squid available for installation. Squid has a list of binary packages for popular operating systems, but building Squid from source code is trivial at best (with a ./configure && make && sudo make install ) but for the sake of cleanliness, I'll just do a sudo apt-get install squid on Ubuntu.

Making Squid Work
Squid must have a cache directory that it can store the data in. If the package installation did not create this, then you must run (as root), squid -z, which will create the cache directory. You should also make sure that squid will start by default. If you reboot and squid isn't running, adding "squid" to your rc.local file is the easiest way, but there could be a start-up script somewhere that would do a more elegant job.


Proxy settings on your workstation(s)
Squid, by default, runs on port 3128. To enable proxy for your workstation, you will need to know the IP Address of your Squid server, and when asked for what port the proxy runs on, enter 3128. Here are some links for setting up Proxy in various browsers. Just remember to use your Squid IP and Port in place of any example proxy configuration used.

• MS Internet Explorer
• Mozilla Firefox
• Apple Safari
  

Now what?
At this point, you likely have a Squid server up and running with the basic default configuration enabled, and for the most part, it's completely useless. Shown below, I've pointed Safari's proxy settings to the Squid server then attempted to access HiR Information Report to no avail.



To fix this, find the ACL lines in squid.conf (on Gutsy, it's /etc/squid/squid.conf) and simply add a line define your local network. Obviously, you would want to replace this with your own IP subnetting information.

acl localnet src 192.168.0.0/255.255.255.0

Then, find the http_access lines and add a line (preferably above all of the other http_access lines) allowing machines on the local network to get out to the rest of the Internet:

http_access allow localnet all

After re-starting squid (killall squid, pkill squid, /etc/init.d/squid restart, etc...) I can access anything through squid, and the pages will now be cached. This will reduce the bandwidth used for day-to-day web browsing and make pages seem to load faster. For a small business running on typical consumer broadband connections, this is a quick and easy way to increase browsing performance. Also, operating system patches will only be downloaded once and then cached. When the time comes to do the updates on all of the workstations, this trick will make life a little easier for you.

If you wish to force all HTTP traffic through Squid (for instance, if you yourself are planning on implementing a Web filter with Squid) and you have a somewhat flexible router or firewall, you can block all outgoing traffic destined for port 80 and 443 except for traffic coming from the Squid server. That, however, is beyond the scope of this article.

Tomorrow, you'll learn how to tweak a few options for Squid so that you can use it to evade web filters in the fourth article in my ongoing web filter evasion series.

Sysadmin Sunday: Linux and BSD Filesystem attributes

Most sysadmins are familiar with the concept of file ownership and permissions. Read, Write, and Execute are ingrained in our memories from the start. Sometimes, though, that's not enough. Here are some extended attributes for files in Linux and *BSD:

BSD:
To set file flags in BSD, use chflags. To view them, use ls -lO. Flags are as follows (from the chflags(1) man page):

  arch, archived
          set the archived flag (super-user only)

  opaque  set the opaque flag (owner or super-user only).  [Directory
          is opaque when viewed through a union mount]

  nodump  set the nodump flag (owner or super-user only)

  sappnd, sappend
          set the system append-only flag (super-user only)

  schg, schange, simmutable
          set the system immutable flag (super-user only)

  uappnd, uappend
          set the user append-only flag (owner or super-user only)

  uchg, uchange, uimmutable
          set the user immutable flag (owner or super-user only)

  hidden  set the hidden flag [Hide item from GUI]

    As discussed in chflags(2), the sappnd and schg flags may only be unset
    when the system is in single-user mode.

    Putting the letters ``no'' before or removing the letters ``no'' from a
    keyword causes the flag to be cleared.  For example:

  nouchg  clear the user immutable flag (owner or super-user only)
  dump    clear the nodump flag (owner or super-user only)

schg and sappnd are particularly useful. As described, they place files in an append-only or immutable state, where nothing, not even root can break the rules without first going into single-user mode and disabling the flag. uchg will keep you from accidentally clobbering an important file as well, but can still be un-set by you.

Example viewing, setting, and demonstrating the uchg flag:

Chimera:Documents axon$ ls -lO
total 32
-rw-r--r-- 1 axon staff - 8258 Apr 6 19:16 description.html
-rw-r--r-- 1 axon staff - 21 Apr 2 12:25 file.doc

Chimera:Documents axon$ chflags uchg description.html

Chimera:Documents axon$ ls -lO
total 32
-rw-r--r-- 1 axon staff uchg 8258 Apr 6 19:16 description.html
-rw-r--r-- 1 axon staff - 21 Apr 2 12:25 file.doc

Chimera:Documents axon$ rm description.html
override rw-r--r-- axon/staff uchg for description.html? y
rm: description.html: Operation not permitted

Chimera:Documents axon$ chflags nouchg description.html

Chimera:Documents axon$ rm description.html

Chimera:Documents axon$ ls -lO
total 8
-rw-r--r-- 1 axon staff - 21 Apr 2 12:25 file.doc


Linux:
Use chattr to set attributes and lsattr to view them.

From the chattr(1) man page:

The format of a symbolic mode is +-=[ASacDdIijsTtu].

The operator ‘+’ causes the selected attributes  to  be  added  to  the
existing  attributes  of  the files; ‘-’ causes them to be removed; and
‘=’ causes them to be the only attributes that the files have.

The letters ‘acdijsuADST’ select the  new  attributes  for  the  files:
append only (a), compressed (c), no dump (d), immutable (i), data jour‐
nalling (j), secure deletion (s), no tail-merging (t), undeletable (u),
no  atime  updates  (A), synchronous directory updates (D), synchronous
updates (S), and top of directory hierarchy (T).

As you can see, most of these attributes are similar to the BSD flags. We'll do the same thing on Linux with the immutable attribute.

axon@hosting:~/hir-test$ lsattr
------------------ ./internet-resume.doc
------------------ ./mail.sql

axon@hosting:~/hir-test$ chattr +i mail.sql

axon@hosting:~/hir-test$ lsattr
------------------ ./internet-resume.doc
----i------------- ./mail.sql

axon@hosting:~/hir-test$ rm mail.sql
rm: remove write-protected regular file `mail.sql'? y
rm: cannot remove `mail.sql': Operation not permitted

Sysadmin Sunday: Subversion

Sorry for the recent lack of Sysadmin Sunday posts. I don't always have good Sysadmin content thought up. Sometimes I do, but I don't have time to actually set up an environment to demonstrate it. Sometimes I don't even have time to get around to writing much of anything.

So, we're back this week with a tutorial on setting up Subversion, a revision control system that, in my opinion, is much better and user-friendly than other revision control systems (such as CVS). You'll see me (and many others) casually use "SVN" when referencing Subversion. You can use subversion to synchronize directories (folders) across different workstations, to collaborate changes on large documentation or programming projects, or to simply have a revision control and change rollback system for your files. It works with binary and text, but can take up a lot of hard drive space on the server for large or frequently-changing data sets.

Installation
Initially installing Subversion varies on the platform and package management system available. I am installing subversion on Ubuntu Gutsy Gibbon Server Edition. Here, it's as simple as:
$ sudo apt-get install subversion

On FreeBSD 6.3, I got it installed using pkg_add as well, with no problems.
$ sudo pkg_add -r subversion

Subversion packages are available for many platforms, but if you wish to build it from source, the documentation is quite good.

HiR Reading Room
Of particular note is the O'Reilly and Associates book: Version Control with Subversion. It is free and open source, so it's also available to read in its entirety on the web. I personally liked it enough that I bought the paperback book, as I dealt with subversion on a daily basis at my last full-time job. If you'll be doing a lot with Subversion, this is a great book to have around as a reference guide.

Please note that I'm not dealing with any encryption whatsoever, so you should probably make sure that your svn server isn't accessible from the Internet. Being behind a private firewall or cheap router will work for testing this at home. Read the book for information on securing SVN with ssh or using better authentication options. There's even a web-based SVN Server, which allows your users to browse the repository with a browser while using all of the flexibility and authentication modules at Apache web server's disposal. All of that and plenty more is covered in this book.

Starting svnserve
First, make a directory to keep the repositories in. A repository is simply a directory on the SVN server where data is stored. I chose /var/svn/repositories.
$ sudo mkdir -p /var/svn/repositories

Then, we need to start our svnserve daemon. As I mentioned before, there are several ways to run SVN. Just starting a vanilla svnserve daemon is the easiest way but not always the best way. For the purposes of this demonstration, I will keep it simple.

Place "svnserve -d -r /var/svn/repositories" (or whatever repository root you chose) in the startup scripts, usually in /etc/rc.local. Either reboot, or run "[sudo] svnserve -d -r /var/svn/repositories" from the command line to start it. This restricts svn to exporting only the contents of /var/svn/repositories.

Your first repository
Let's set up our first repository. On the SVN Server, run the following. You may choose whatever you wish for the repository name, but it has to be under the repository root you defined when you started svnserve.
$ sudo svnadmin create /var/svn/repositories/hir-test/

You should create a username and a password by editing the "svnserve" and "passwd" files in the "conf" directory under the repository you just created. Keep in mind that passwords are all in plaintext for this example. It's not really the best way to do things, but it is the simplest.

$ vi /var/svn/repositories/hir-test/conf/svnserve.conf
un-comment the following lines:
anon-access = read
auth-access = write
password-db = passwd


$ vi /var/svn/repositories/hir-test/conf/passwd
Add a line like the following to the end of the file to define your username and password:
axon = setecastronomy

Now, go to a client computer (or simply access it locally but through the svn:// url scheme) and use SVN to check out the repository:
[axon@floaty-fbsd ~]$ svn checkout svn://axon@192.168.0.108/hir-test/
[axon@floaty-fbsd ~]$ cd hir-test
[axon@floaty-fbsd ~/hir-test]$ ls -la

total 8
drwxr-xr-x   3 axon  axon   512 Mar 29 21:37 .
drwxr-xr-x  20 axon  axon  2560 Mar 29 21:38 ..
drwxr-xr-x   6 axon  axon   512 Mar 29 21:37 .svn

Copy a file into the repository, or make a new file. Your choice. Then use "svn stat" to see the status of the files in the directory.
[axon@floaty-fbsd ~/hir-test]$ cp ~/internet-resume.doc .
[axon@floaty-fbsd ~/hir-test]$ svn stat
? internet-resume.doc

SVN doesn't recognize the file, hence the "?" before the file name. We must first add the file with "svn add"
[axon@floaty-fbsd ~/hir-test]$ svn add internet-resume.doc
A (bin) internet-resume.doc
[axon@floaty-fbsd ~/hir-test]$ svn stat
A internet-resume.doc

At this point, SVN is aware of the file, but it still hasn't uploaded it to the svn server. If we go to another machine and check out the repository, it will still be empty. (sorry, I used the DNS name in this example but it's the same server as 192.168.0.108)
axon@hosting:~$ svn checkout svn://axon@lampdev.labs.h-i-r.net/hir-test
Checked out revision 0.
axon@hosting:~$ cd hir-test/
axon@hosting:~/hir-test$ ls -la

total 12
drwxr-xr-x 3 axon axon 4096 2008-03-29 16:52 .
drwxr-xr-x 5 axon axon 4096 2008-03-29 16:52 ..
drwxr-xr-x 6 axon axon 4096 2008-03-29 16:52 .svn

Back on the client machine where we added the file, though, use "svn commit" to update the central repository on the SVN server. You should always commit with a message saying what changes were made. use --message for that. It should then prompt for your password.

[axon@floaty-fbsd ~/hir-test]$ svn commit --message "added my resume"
Authentication realm: 2c608312-9cd3-44f0-b88e-356728a5cc35
Password for 'axon': setecastronomy (not shown on screen)
Adding (bin) internet-resume.doc
Transmitting file data .
Committed revision 1.

Now, whenever you check out the repository elsewhere, the files will be in sync. Use svn update to refresh your local repository to the latest version:
axon@hosting:~/hir-test$ svn update
A internet-resume.doc
Updated to revision 1.
axon@hosting:~/hir-test$ ls
internet-resume.doc

When you delete, copy or move files within a local copy of the repository, it's best (practically mandatory) to use svn delete (svn rm), svn copy (svn cp), and svn move (svn mv) for these tasks, respectively.

So far, the commands we've covered the following for client machines. This should be enough to get you up and running.
svn checkout svn://[user@]host/repo-name - Checks out a local copy of the repository
svn update - Refreshes the local repository
svn add - Adds version control to new files in the local copy of the repository
svn commit --message "test message" - Updates the central repository
svn delete filename - remove a file from the local repository
svn rm filename - same as above
svn copy filename1 filename2 - make a copy of a file within the repo.
svn cp filename1 filename2 - same as above
svn move filename1 filename2 - move a file from one place to another in the repo.
svn mv filename1 filename2 - same as above

There is also a windows shell extension available, called TortoiseSVN. This puts a TortoiseSVN menu in the list when you right-click while browsing files on Windows. From here, you can check out, update, add, commit and manipulate repositories right from within Windows.

Sysadmin Sunday: LogCheck

I've been using LogCheck in its various incarnations for quite a while. When I started using it (it feels like a decade ago but it may have been a little less) it was called LogSentry. While it's still pretty simple, it gets the job done. LogCheck's been passed around many times, and I'm relatively sure that the package I downloaded has remained for the most part unchanged for at least 5 years. For a small group of servers you wish to keep tabs on, this tool is great. For an enterprise, you might want something more heavy duty such as a database-driven centralized logging server.

Note that by itself, LogCheck only does two things:

1. Looks through all the log files shown for suspicious or "hackish" activity
2. E-mails the report

That's it.

Installation:
I'll be doing the install on FreeBSD. LogCheck is just a script, so it will run on pretty much anything that isn't Windows.

$ sudo pkg_add -r logcheck
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6.3-release/Latest/logcheck.tbz... Done.

Configuration:
To see what LogCheck thinks is "hackish", check out logcheck.hacking and logcheck.violations (in /usr/local/etc/ when installed from FreeBSD's packages) Those contain a list of regular expressions (one per line) that will be used to gather reporting information from the logs. Similarly, logcheck.ignore and logcheck.violations.ignore contain regular expressions that can filter certain results from the report. I recommend looking at these files but not editing them at first. Let LogCheck run for a week or so (daily) to get a feel for it. Then, start adding or removing expressions to the lists to fine tune your reports.

To change what e-mail address the report is sent to, find logcheck.sh (it's also in /usr/local/etc/ when installed from FreeBSD's packages) and modify this block of text as appropriate:

# Person to send log activity to.
SYSADMIN=root

Look at the other variables, too. By default, the log files and configuration will likely be workable, but it's a good idea to make sure.

Adding the following line to the system's crontab will launch LogCheck every hour. This isn't a big deal because it will only mail you if there is something suspicious. If you would rather have a daily log, read the man page on crontab to figure out how the schedule options.

00 * * * * root /bin/sh /usr/local/etc/logcheck.sh

Sysadmin Sunday: Encrypted Swap Space in FreeBSD

New research shows that cold boot attacks can compromise encryption by giving you access to the RAM's contents. This even applies to full-disk encryption that encrypts swap space and whatnot. We'll probably talk more about that in our podcast.

Physical access almost always means that a total compromise of security is somehow possible. Sensitive data could be at risk long after the data in RAM decays into oblivion. Encryption keys, large documents, and other files containing data you don't want falling into the wrong hands will almost certainly see their fair share of time in virtual memory. That means pages of RAM will be written, for better or worse, to the hard drive.

Encrypting just your swap space can be easier than switching to full-disk encryption and makes it more difficult for an attacker to obtain sensitive data as it gets shuffled in and out of RAM.

OpenBSD encrypts swap by default without any further action. To the best of my knowledge it's alone in that fact. I can't outline how to encrypt swap space in every operating system, but since FreeBSD's the flavor-du-jour, I'll show you how easy it can be.

First, disable swap:

[axon@floaty-fbsd ~]$ sudo swapoff -a
swapoff: /dev/ad0s1b: Cannot allocate memory

Whoops! If there's too much stuff running to fit in RAM without swap, you'll need to cut back. Close your applications, stop the X Window system or whatever you have to do. Let's try that again.

[axon@floaty-fbsd ~]$ sudo swapoff -a

Now, find the swap partition and over-write it using dd(1) to stream /dev/urandom to it. It could take a while depending on the size of the swap partition. Yes, this is a paranoia countermeasure and nothing more. Keep in mind that only freshly-used parts of swap will be encrypted once we're finished. Right now, there could be (ab)usable data out there. Not for long!

[axon@floaty-fbsd ~]$ grep swap /etc/fstab
/dev/ad0s1b none swap sw 0 0
[axon@floaty-fbsd ~]$ sudo dd if=/dev/urandom of=/dev/ad0s1b
dd: /dev/ad0s1b: end of device
464657+0 records in
464656+0 records out
237903872 bytes transferred in 88.835558 secs (2678025 bytes/sec)

To enable GEOM ELI on your swap partition, edit /etc/fstab and add ".eli" to the end of the swap device.

[axon@floaty-fbsd ~]$ sudo vi /etc/fstab
change this line:
/dev/ad0s1b none swap sw 0 0
to this:
/dev/ad0s1b.eli none swap sw 0 0

You have to reboot for this to take effect. Once it comes back online, use kldstat and swapinfo to verify that the geom_eli.ko module is loaded and that the new swap device is being used.

[axon@floaty-fbsd ~]$ kldstat
Id Refs Address Size Name
1 10 0xc0400000 7b2d2c kernel
2 1 0xc0bb3000 6974 snd_ich.ko
3 2 0xc0bba000 239a8 sound.ko
4 1 0xc0bde000 5c304 acpi.ko
5 1 0xc1f19000 e000 geom_eli.ko
6 1 0xc1f27000 19000 crypto.ko
7 1 0xc1f40000 a000 zlib.ko
8 1 0xc21ec000 2000 warp_saver.ko
[axon@floaty-fbsd ~]$ swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ad0s1b.eli 232328 0 232328 0%

Voila!

Sysadmin Sunday: Quick & Dirty SSH Tunneling

Occasionally, you might need to tunnel some other traffic over SSH. This could be to get access to an external web proxy, to get a remote X display up, or to get around a firewall-blocked port that you must access.

The syntax (on the command-line OpenSSH client) for a Local forward is like this:

# ssh remote-example.h-i-r.net -L 3128:localhost:3128

This tells my SSH client to tunnel traffic to port 3128 on my workstation to port 3128 on my DMZ box. Port 3128 isn't accessible because of a firewall, but tunnelled over SSH it works fine. In this case, I'm running squid on the remote example host. Telling Firefox to use http://localhost:3128 as the proxy will now tunnel all of my web traffic over the SSH tunnel to the squid proxy behind the firewall. The reason it's called a local forward because it forwards a local port over the SSH connection.

A remote forward will open up a port on the remote machine and connect it to a port on the client's network. The syntax is similar:

# ssh remote-example.h-i-r.net -R 3306:dbserver:3306

This would open up port 3306 (the MySQL server port) on the remote host and tunnel it to the MySQL service on the host named dbserver on my local network.

While running forwarding of either type, you can enter the hotkey sequence "~#" to see all the open connections through the forwarded ports.

Asmodian's Workbench

Whats on Asmodian's workbench?
============================================
Google mini
============================================
The Google-mini is an Internet search appliance. It is essentially a 1u Intel server with Googles search engine with a simple web based interface to specify what sites it should index. The mini lacks the capability to access anything but web or samba based resources. To this end they have a feature called a Onebox.

A Onebox module is an XML profile describing what to do if a certain set or format of keywords are encountered during a query. The definition then tells Google to access a certain collection or an external script which then is sent the query and any applicable authentication information and expects back an XML response. The response is pared with an XSLT style sheet and displayed with the search results.

The Onebox can search queries by key words, Perl regular expressions or on every search.
The example they made was an employee directory which searched by lastname and returned basic contact information in a formated box with a graphic.

Informative resources:
Google Inc. "One Box Guide". Accessed 2-18-2008
http://code.google.com/enterprise/documentation/oneboxguide.html
============================================
Pure-FTPd Follow up.
I am working on an adaptation of my previous article on Pure-FTP using Mysql as the user database and creating an automated user web space system, or adapting an existing one.

I am in a situation where I have user information sitting in LDAP (via Novell NDS) however I cannot implement my own schema so I must use a Mysql back end to store the user information.

So what I plan on doing is setting up the interface to poll LDAP for user info and import it into the authentication database.

============================================
iPOD Touch Hassle
It seems I am cursed to buy the one technological widget which is resistant to modification.
It appears that the new iPOD's purchased since December 2007 are resistant to Jail breaking.

Worst yet, information on this is buried amongst a deluge of you tube videos and un-readable Interweb BLAG's with a crap ton of advertisements so that searching for helpful information is an all night task. I'll tell you what doesn't work:

Upgrading to the 1.1.1 firmware via Itunes 7.5+ (Mac and PC)

Though it has been a learning experience:
The iPOD touch /iPhone uses a Arm processor some where around 500Mhz.

You can access the media folders on the iPOD via iPHUC. However on a standard iPOD this dumps you into a chrooted folder (/var/root/media)

The downgrade/jailbreak process goes like this 1.) Get the old firmware 2.) Put it into recovery mode(see above) 3.)Use iTunes to load the old firmware 4.)on your ipod goto the site with the giant tiff with the overflow info and payload. 5.) It will load the program installer.

Then theres some tricks to upgrade the iPOD back to the latest firmware while keeping the ability to load 3rd party software.

Like I said, this has not been working.

You can resurrect a "bricked" iPOD or iPhone by holding in the Sleep button and the home button then letting go of the sleep button when you see the apple logo. Then you should see the iTunes logo and the iPOD cable symbols. Attach it to iTunes and then restore the firmware. (This wipes out EVERYTHING).

I understand that if the boot loader is corrupted or overwritten with an incompatible version there is a way to reload it too but I haven't seen any articles with sufficient details on this.

Informative Resources:
ARM web page. "ARM Powered Products". Accessed 2-18-2008 http://www.arm.com/markets/mobile_solutions/armpp/18665.html

Johnstone, Jeremy. "Howto: Run custom apps on iPhone (Part #1)" Accessed 2-18-2008
http://www.jeremyjohnstone.com/blog/archives/2007/08/05/howto-run-custom-apps-on-iphone-part-1/
*This describes a utility called iPhuc and has little to do with the iPOD touch

ilounge.com. "white screen lockup", Accessed 2-18-2008
http://forums.ilounge.com/archive/index.php/t-209541.html
*This references a couple of Youtube videos on updating the firmware.

iPHUC Homepage, "iPHUC" Accessed 2-18-2008
http://code.google.com/p/iphuc/
*iPhuc gives you an interactive (albet chrooted) shell to your iPhone or iPOD.

True, Nathan. "ibrickr" Accessed 2-18-2008
http://cre.ations.net/creation/ibrickr
*The iBrickr application is very handy in even though it's meant for the iPhone.
============================================

Sysadmin Sunday: Pure-ftpd configuration on Ubuntu Server Edition

0. Introduction

There are tons of ftp servers out there. Some are leftovers from the stone age others are fairly up to date with SSL capability and virtual user support. In this case I have chosen Ubuntu server and Pure-ftpd.

Why: Ubuntu comes out of the box unencumbered by unnecessary bloat that many server editions are forced to install out of a spider web of software dependencies. It at the same time has a very mature package management system which allows for easy software updates.

I'll stop there because I don't intend this to be a flame war on what Linux is best. An important difference that the Ubuntu installation has versus its native configuration is that it uses a configuration wrapper instead of modifying a start-up script. Which is why I am writing this article.

This article assumes that you have an intermediate or advanced knowledge of command line based UNIX operating systems.

-=-=-=-=-=-=-=-
Table of contents:
-=-=-=-=-=-=-=-
0.........Intro
1..........System setup and overview on what needs to be done
2..........Package installation
3..........Wrapper configuration with virtual users and SSL
4..........Access control via virtual users and iptables and or pure-ftp
5..........Informative resources
-=-=-=-=-=-=-=-


1.System setup and overview on what needs to be done
-=-=-=-=-=-=-=-
We will be needing an Ubuntu server edition installation. Preferably with a firewall in place.
In this case I have a pile of users who need to use dreamweaver to edit their web sites. Preferably they need to have their own logins, those logins need to be chrooted to their directory. Secondly, the users have no earthly reason to have a system user account.
For security sake I want the option for them to use encryption (once we can get licenses for the newer version of Dreamweaver which supports it). Lastly having these virtual users have disk space quotas so some idiot doesn't up load a ton of the family pictures that were taken with a 10 Megapixel camera and are like 20 Mb a piece causing the storage array to !#*& itself.

Pure-Ftpd can use arbitrary file paths for virtual user home directories. You can assign a local system user account and group for each virtual user or group of virtual users.

You can make a system user and group called webadmin who owns the "sites" folder under "/data/sites". All of the virtual users from a system standpoint are doing business as "webadmin". Pureftp does the job of doing access control and permissions on its end and keeping people in their home folders.
-=-=-=-=-=-=-=-
2.Package installation
-=-=-=-=-=-=-=-
Run the command:

apt-get install pure-ftpd

There is a GUI configuration tool but by default Ubuntu does not have a GUI so I leave that to you.

Once installed the wrapper configuration folder is on "/etc/pure-ftpd". The folder contains a structure like so:

root@stage:/etc/pure-ftpd# ls -al
total 32
drwxr-xr-x 5 root root 4096 2008-02-20 01:32 .
drwxr-xr-x 133 root root 12288 2008-02-20 01:32 ..
drwxr-xr-x 2 root root 4096 2008-02-20 01:32 auth
drwxr-xr-x 2 root root 4096 2008-02-20 01:32 conf
drwxr-xr-x 2 root root 4096 2007-06-21 19:01 db
-rw-r--r-- 1 root root 230 2007-06-21 19:01 pureftpd-dir-aliases

Under Auth you will find the following symlinks:

root@stage:/etc/pure-ftpd/auth# ls -alF
total 8
drwxr-xr-x 2 root root 4096 2008-02-20 01:32 ./
drwxr-xr-x 5 root root 4096 2008-02-20 01:32 ../
lrwxrwxrwx 1 root root 26 2008-02-20 01:32 65unix -> ../conf/UnixAuthentication
lrwxrwxrwx 1 root root 25 2008-02-20 01:32 70pam -> ../conf/PAMAuthentication

Delete all of these and make a symlink to "../conf/PureDB":

root@stage:/etc/pure-ftpd/auth# ln -s ../conf/PureDB

Go to the conf directory and edit the "PAMAuthentication" file to say NO insted of YES.
Add a new file called "ChrootEveryone" and edit it and add the word "YES".

Now lets make a user!

pure-pw useradd test -u webadmin -g webadmin -d /data/sites/localhost/ -N 25
pure-pw mkdb

Where -N is a 25 Mb quota and -u and -g is the userid and groupid of the corresponding system user. -d is the folder that the user is chrooted in. The command "mkdb" creates the binary password database.

Now we have our chrooted ftpd environment finished we just need to configure the ssl option.
now it is possible to FORCE users to use ftp-SSL however thats out side of the scope of this article.
(from the pure-ftpd documentation)

mkdir -p /etc/ssl/private  openssl req -x509 -nodes -newkey rsa:1024 -keyout \
 /etc/ssl/private/pure-ftpd.pem \
-out /etc/ssl/private/pure-ftpd.pem

chmod 600 /etc/ssl/private/*.pem

Then go into "/etc/pure-ftpd/conf" and edit the file named "TLS" and add the number "1".
(0 disables encryption, 1 makes it optional and 2 makes it mandatory).

Now restart the service :

/etc/init.d/pure-ftpd restart

And login to your new ftp server!
-=-=-=-=-=-
4.Access control via virtual users and iptables and or pure-ftp
-=-=-=-=-=-
It pisses me off to read the logs and see all of the automated Interweb
exploit scripted attacks. So here are some suggestions to keep the
automated attacks down.

1. Hosts.deny doesn't work. Use Ip tables filtering.
2. Grab a blackhole ip list (BBL) from http://www.unixhub.com/block.html.
3. Determine your scope of service, if your users work only within the continental United States then blocking AP-NIC and any other non-local IP ranges in their entirety would be a good idea.

If you want to tie it down on a per-user basis try this:

pure-pw usermod testuser \
-r IP-ADDRESS-RANGE\
-R IP-ADDRESS-RANGE

where -r is allowed IP ranged and -R is denied ranges (Example: -R 200.0.0.0/8 -r 192.168.0.0/16)

-=-=-=-=-=-
5.Informative resources
-=-=-=-=-=-
Barnes, Robert. "Bad IP addresses/Bob's Block List (BBL)", (Accessed Feb, 2008)
http://www.unixhub.com/block.html

Denis, Frank. "Pure ftpd", (Accessed Feb, 2008)
http://www.pureftpd.org
http://download.pureftpd.org/pub/pure-ftpd/doc/README.TLS

Hornburg, Stefan (Racke). "Debian pure-ftpd-wrapper man page", (Accessed Feb, 2008)
http://www.penguin-soft.com/penguin/man/8/pure-ftpd-wrapper.html

Sysadmin Sunday: Process Management

Managing processes in UNIX operating systems is a critical function. This goes beyond just killing runaway processes, though. This week in Sysadmin Sunday, we'll take a look at managing processes to make your life a little easier.

When you're in a shell, there are 3 kinds of processes: foreground, background, and stopped (sometimes called suspended). This is pretty straight-forward.

Shelling out of a running program
If you're in the middle of a really long-running program and don't want to or cannot switch consoles, you can normally shell out of that that process with the (Ctrl-Z) key combination. For example, encrypting a huge file with GNU Privacy Guard:

$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
(moments pass...)
^Z (Ctrl-Z)
[1]+  Stopped                 gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
$

Pressing (Ctrl-Z) gives you a shell but puts the running process in a stopped state. From there, you can run something else, such as vi. Just remember that the process that you escaped from is stopped and won't make any progress at this point. That also means it shouldn't display any output that might interrupt you if you decide to work on something else while that process is stopped. It's more "paused" than "stopped". Take note of the job ID [in brackets] of the stopped process. That will come in handy later on.

See what jobs your current shell has
I've taken the liberty of running and shelling out of vi as well for this example. Keep in mind that GPG is still sitting there doing nothing, too.

To see what's running in your current shell, use the jobs tool.

$ jobs
[1]-  Stopped                 gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
[2]+  Stopped                 vi /etc/resolv.conf

Here, you can see both gpg and vi are stopped. Again, the job ID for each process shows up in brackets. The + shows which of the jobs is the most recent one you have dealt with and will be used as the default job later on. Next, the status of the job is shown, followed by the actual command-line (or as much of it as can be displayed if it's a particularly long command line).

Switching between processes
We take a lot of things for granted, like being able to tab between 20 different windows on a GUI terminal. Not to be a curmudgeonly olde-school sysadmin, but back in the day, we didn't have that luxury. A lot of my early interactions with UNIX were over a serial terminal or a dial-up connection. Virtual consoles were not common, so switching between jobs was just part of working with UNIX on a daily basis for many people.

To bring a stopped process to the foreground again, use the fg command. If you just type fg, it will bring back the most recent job, the one marked with a + when you run the jobs command. If you wish to tell it to bring back a specific command, use jobs %2, or replace 2 with the job number you wish to bring to the foreground. You could have two different files open with two different instances of vi for example, and use ^Z, jobs, and fg to switch between them. This, of course, can get confusing, so be careful when multi-tasking this way.

$ fg %2
(vi opens back up)

Background processes
For programs that you don't expect much output from (such as gpg) you can simply background the job. Instead of fg, you use the bg command. Again bg %1 would work to background a specific job number. A background job is much like a stopped one, except it actually goes on about its business. If there is output, it will display the output onto your screen regardless what you're doing. Otherwise, it will silently chug away out of sight. When the process finishes, you'll be notified that it has completed. I'll background the gpg job to show how this works:

$ bg %1
[1]+ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
$
[1]+  Done                    gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
$

Alternatively, if you want to run something that you know will take a while and you're certain you won't have to actually interact with it, you can launch it in the background by placing an ampersand (&) at the end of the command line. You'll immediately get the job ID and process ID, then you'll get your shell back. When it's done running, you'll be notified.

$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
[1] 1567
$
[1]+  Done                    gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
$

What if you log off with stuff still running?
It really boils down to what's running and how it handles the various interrupt signals. As a general rule, any child processes under the shell get sent a HUP (HangUP) signal which is analogous to kill -1 or kill -HUP . This usually terminates the processes so that they're not out there hanging around forever. If you log off, there's a good chance that whatever you were running will stop.

If you really want the things to stay running, however, there are a few viable options. The first one good if you know you may need to log off while something is still running. It's the nohup command. Put it before your usual commnd and background it either with an ampersand at the end or by (Ctrl-Z) and bg. When you log off, the process should stay running because nohup doesn't pass the signal along to the child process. This is great for huge software compiles or other time-consuming, non-interactive tasks. Furthermore, nohup will re-direct all the output to a file (by default, nohup.out) so that you can go back and look at it later. This has the additional benefit of not barfing stuff all over your screen while you may be trying to get work done.

$ nohup nmap localhost&
[1] 1612
$ appending output to nohup.out
$ exit

At this point, you can log off the system and nmap will continue running. You can log in later to see the results of the nmap by checking the nohup.out file:

$ cat nohup.out

Starting Nmap 4.20 ( http://insecure.org ) at 2008-02-02 03:55 CST
Interesting ports on localhost (127.0.0.1):
Not shown: 1690 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
88/tcp   open  kerberos-sec
139/tcp  open  netbios-ssn
445/tcp  open  microsoft-ds
548/tcp  open  afpovertcp
631/tcp  open  ipp
3689/tcp open  rendezvous

Nmap finished: 1 IP address (1 host up) scanned in 15.157 seconds

But what if you're in the middle of something that you didn't expect to take so long, for example, compiling Mozilla Firefox from source? You have to log off, but want it to keep running. You don't want to stop it, then restart it with nohup. Not all hope is lost. The second method is to simply tell the shell not to send the HUP signal to any of the processes. Make sure only the things you want to remain running in the background are properly backgrounded, then on the command-line, type set NOHUP, then log off. Unlike the nohup command, output is not re-directed, so you won't be able to see any of the output that happens while you're logged off. For things like software compiling or moving large amounts of data, this isn't much of a problem.

$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
^Z
[1]+  Stopped                 gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
$ bg
[1]+ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
$ set NOHUP
$ exit

The process will finish normally.

There are other options that are more modern than using the jobs, bg, and fg commands. These are beyond the scope of this article, however my favorite among these options is the screen command, which gives you virtual consoles that persist even after you log out. Screen comes with many Linux distributions, but it's still a little-known utility except for the fairly hard-core command-line junkies. This allows you to leave things like text-mode IRC clients or other interactive programs running at all times, even if you're not actually logged in.

Stone-Age Sysadmin Sunday: Dealing with CPIO Archives

Some truly revolutionary things have been around in UNIX for a very long time because they're just that good. Some are around as-is: find, netstat or ifconfig for example. Others have been improved upon ad infinitum, like vim (a better, sleeker version of vi that somehow stays true to its vi roots).

Then, you have ugly things that just won't die. Case and point: cpio. Originally meant as a stream archiver for tape storage, any sysadmin worth their salt will eventually run across some silly software package or archive file that's in cpio format.  And inevitably, getting data out of that archive will be important for one reason or another to avoid some "crisis."  Cpio is legacy for many a good reason, but you will inevitably be forced to square off against cpio one of these days, and it's best not to enter the arena unarmed. This week will be a shorter, succinct edition of Sysadmin Sunday, as we delve into cpio.

Also, I started writing this article several weeks back.  I usually queue up several articles ahead of time, especially the weekly Sysadmin Sunday feature.  I had no intention of actually having to mess with cpio, but just like talking about what to do in the event of a catastrophic system failure often leads to having to deal with one, writing about cpio did not come without its consequences.

I was forced to deal with cpio last week at work.  It seems there are some old scripts that keep using cpio for archiving data. I found out that the version of cpio that we had running wouldn't acknowledge files owned by users or groups with a UID or GID higher than 65,535. This is the typical "32-bit UID/GID" problem that got fixed in the REST OF UNIX back in the 1990s sometime. While poking around, I found that the earliest version of cpio was actually written by a caveman named Ungh in abacus-native machine code. The dude had serious skills. Apparently, though, no one's bothered to update it in the last few millennia.

Create a cpio archive
(not recommended... haha!)

cpio expects standard input to bear a list of files to archive. The best way to do this is to pipe find output into it.  Keep in mind that cpio does NOT strip leading slashes.  If you restore a cpio archive that was from the root directory, it WILL over-write data starting at the root directory.  For this reason, I recommend you cd to the root directory and run a find with a relative path from root instead of running find with an absolute path.

$ cd /; find path/to/archive | cpio -o > file.cpio

Extract files from a cpio archive

(so you can put them on something more modern, like 5 1/4" floppies)

The simplest way to extract from cpio is like this:

$ cpio -id < file.cpio

Occasionally, someone will compress the file with the UNIX compress command, which will give you a file ending in .Z. Use zcat for this:

$ zcat file.cpio.Z | cpio -id

And similarly, if you run across the bizarre gzipped cpio file, you can do this:

$ gzcat file.cpio.gz | cpio -id

(Props to [jdoublep] for the hilarious post title)

Sysadmin Sunday: OpenBSD, Apache, MySQL, PHP (OAMP?)

Editor's note: I know, I know.  This is SATURDAY!  But I'm putting this out there for a good reason.  Today is also KC PHP User's Group's first meeting of 2008, and what better way to kick off the KCPUG new year than with some fresh AMP-Stack Content.  So, here we go.  

Last week, I discussed how OpenBSD sysadmins can make their lives a little easier with some software I wrote. This week on Sysadmin Sunday, I'll show you how easy it is to get a decent AMP (Apache, MySQL and PHP) environment up and running on OpenBSD 4.2, which is the current version as of writing.

It used to be that some stuff had to be compiled by hand to make everything work just right, but the package/port guys finally nailed it. With the help of obsd_pkgscripts, OAMP is a breeze. The benefits of running your AMP-driven website atop OpenBSD are many, not the least of which are the facts that it's a minimalistic, secure, and high-performance platform out of the box.

OpenBSD already ships with Apache. It's a patched and tweaked version of Apache, but it's Apache all the same. You're already halfway to OAMP land with OpenBSD and Apache installed out of the box.

First, install the MySQL Server. I obviously chose to use my pkg_get.sh tool for this, but it's not needed.

$ pkg_get.sh mysql-server
Attempting to fetch/install package...
mysql-client-5.0.45: complete
p5-DBD-mysql-3.0008:p5-Net-Daemon-0.39: complete
p5-DBD-mysql-3.0008:p5-PlRPC-0.2018p0: complete
p5-DBD-mysql-3.0008:p5-DBI-1.53: complete
p5-DBD-mysql-3.0008: complete
mysql-server-5.0.45: complete
--- mysql-server-5.0.45 -------------------
You can find detailed instructions on how to install a database
in /usr/local/share/doc/mysql/README.OpenBSD.

Next, that README contains some useful information for more advanced installations and performance, but summarized, follow these instructions. I use sudo, and rarely use the root account. I recommend doing the same.

Create the DB:

-bash-3.2$ sudo /usr/local/bin/mysql_install_db

Start MySQL:

-bash-3.2$ sudo /usr/local/share/mysql/mysql.server start

To make MySQL Start up at boot, add the above mysql startup line to /etc/rc.local without the "sudo" at the beginning (optional but highly recommended). There is a "theoretically correct" start-up snippit on the documentation, but I've found this works just as well.

Set a password for mysql's root user (choose a better one than this):

-bash-3.2$ sudo /usr/local/bin/mysqladmin -u root password 'new-password'

That's pretty much it as far as getting MySQL up and running is concerned. Now we just need to get PHP installed with the MySQL client library...

-bash-3.2$ sudo pkgscripts/pkg_get.sh php5-mysql-
Attempting to fetch/install package...
php5-core-5.2.3: complete
php5-mysql-5.2.3: complete
--- php5-core-5.2.3 -------------------
To finish the install, enable the php5 module with:
/usr/local/sbin/phpxs -s

To enable parsing of PHP scripts, add the following to
/var/www/conf/httpd.conf:

AddType application/x-httpd-php .php

Copy the config file below into /var/www/conf/php.ini
/usr/local/share/examples/php5/php.ini-recommended

Don't forget that the default OpenBSD httpd is chrooted
into /var/www by default, so you may need to create support
directories such as /var/www/tmp for PHP to work correctly.
--- php5-mysql-5.2.3 -------------------
Enable this module in php.ini using the following command:

/usr/local/sbin/phpxs -a mysql

Then you have to follow the post-install instructions:

-bash-3.2$ sudo /usr/local/sbin/phpxs -s
-bash-3.2$ sudo /usr/local/sbin/phpxs -a mysql

Edit /var/www/conf/httpd.conf and add the below line with the other AddTypes:
AddType application/x-httpd-php .php

I also recommend since we're going to be running PHP, adding index.php to the DirectoryIndex line:
DirectoryIndex index.html index.php

Save the httpd.conf file.

Now, copy the php.ini file over from the examples:

-bash-3.2-$ sudo cp \
/usr/local/share/examples/php5/php.ini-recommended \
/var/www/conf/php.ini

Now, make sure apache is set to start.
If you plan on keeping all your web-related content content in /var/www, find the httpd_flags line in /etc/rc.conf and change it to httpd_flags=""

If you intend on giving users traditional ~username (public_html) directories, start apache without being chrooted. Chroot is more secure, but it's a royal pain if you want user home-directories on a vanilla OAMP box, and beyond the "quick and dirty OAMP" scope of this article. Find the httpd_flags line in /etc/rc.conf and change it to httpd_flags="-u" to disable chroot if you must.

The above change to httpd_flags will make apache auto-start upon reboot. You can manually start it with apachectl, though:

-bash-3.2$ sudo apachectl start
/usr/sbin/apachectl start: httpd started

Make a phpinfo example php file and move it into the web root.

-bash-3.2$ echo "" > phpinfo.php
-bash-3.2$ sudo mv phpinfo.php /var/www/htdocs/

Then test it out by pointing a browser to your new phpinfo.php file (example: http://10.23.45.67/phpinfo.php ) and you should see output from PHP. If you see a blank screen or just the phpinfo line, something went wrong. Re-check your steps.


There you have it! As a test, I installed Joomla on this system (screen shot below). Joomla is a popular open-source CMS. I unpacked it into /var/www/htdocs/joomla and pointed my browser to it, following all of its instructions. This includes making a database, changing permissions on some of the Joomla files, and modifying some values in php.ini. The installation went off without any problems just like it would on any LAMP box. I'd imagine you'll see similar results with other *AMP packages such as phpMyAdmin, Xoops, and others.

Sysadmin Sunday: obsd_pkgscripts for OpenBSD

I'll admit it. As much as I adore OpenBSD, its management of binary packages really could use some work. I've been using OpenBSD since 1998 after meeting Theo in person when HiR went to Defcon 6. Heavy daily use of OpenBSD came in 1999 or early 2000. Back in 2004, I finally got fed up. 4 years of daily use without a really great package management system had finally worn me down to my last nerve. With almost every new release, OpenBSD's dev team says they've improved pkg_add functionality, but truth be known, I still find it left wanting to this very day.

This week on Sysadmin Sunday, I'll discuss some tools I wrote to make OpenBSD sysadmins' lives a little easier.

What are my complaints about OpenBSD's package management? Just to name a few:

1. No good way to search for packages by name
2. The need to memorize long path names to install a binary package from an FTP mirror
3. The need to guess at package file names (tricky with version numbers)

Enter: obsd_pkgscripts, the current incarnation of my work from 2004.

The premise was simple: Download a list of all the packages that are easily searchable, then make a script to search and install the packages via FTP, taking away the repetition, memorization, and guess work.

The package is available for download in the file archives of the HiR Google Group, but this download link may work as well:
obsd_pkgscripts-1.00.tar.gz (2.5 kB)

To install it, simply download it, uncompress it, and run the included pkg_update.sh script. I'm cheating and using lynx -source to download it. It comes by default with OpenBSD, shoot me.

$ lynx -source http://h-i-r.googlegroups.com/web/obsd_pkgscripts-1.00.tar.gz \
> obsd_pkgscripts-1.00.tar.gz
$ file obsd_pkgscripts-1.00.tar.gz
obsd_pkgscripts-1.00.tar.gz: gzip compressed data, from Unix
$ tar xzvf obsd_pkgscripts-1.00.tar.gz
pkgscripts
pkgscripts/README
pkgscripts/pkg_get.sh
pkgscripts/pkg_scan.sh
pkgscripts/pkg_update.sh
$ pkgscripts/pkg_update.sh
cat: /home/axon/.tmpkg/mirror: No such file or directory
Package index directory does not exist! Creating Directory...
Package index directory created.
Downloading package index from ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/sparc/ ...
Download complete!

As you can see, there's a README (I'd read it if I were you!), and three scripts: pkg_update.sh, pkg_scan.sh and pkg_get.sh. pkg_update, which we ran already, simply finds out what version and architecture of OpenBSD you're running and fetches the package index for that platform. It also creates the mirror configuration file if it doesn't already exist. This allows you to choose which OpenBSD mirror to use, if the default one isn't what you want.

pkg_scan.sh just tells you the names of packages that match your query. It simply uses grep on the package index. For instance, if we were looking for the xmms audio player, we'd find a lot of packages (xmms and all its plugin packages):

$ pkg_scan.sh xmms
faad-xmms-2.0p5.tgz
py-xmms-1.06p0.tgz
xmms-1.2.10p9.tgz
xmms-bonk-0.12p0.tgz
xmms-esd-1.2.10p5.tgz
xmms-flac-1.1.2p1.tgz
... output truncated ...

pkg_get.sh takes this a step further and actually installs the package. If you plan on running pkg_get as a normal user (recommended), you must have sudo access to run pkg_add. Otherwise, run pkg_get as root (not recommended). Optionally, you can add it to your path in the profile for your shell like I did. pkg_get.sh won't install a package until you have a totally unique package name. For example, autoconf is one of those bizarre packages where there are many versions available. If I try installing it with pkg_get.sh autoconf, I will get an error:

$ pkg_get.sh autoconf
autoconf-2.13p0.tgz
autoconf-2.52p1.tgz
autoconf-2.54p1.tgz
autoconf-2.56p0.tgz
autoconf-2.57p0.tgz
autoconf-2.58p1.tgz
autoconf-2.59p1.tgz
autoconf-2.60p1.tgz
autoconf-2.61p1.tgz
Ambiguous package name. Please be more specific.

But if I enter a unique-enough filename, it will install it. You don't need to type the whole filename, just enough of it to not match more than one package.

$ pkg_get.sh autoconf-2.61
Attempting to fetch/install package...
autoconf-2.61p1: complete

OpenBSD's pkg_add has come a long way. It will automatically download any necessary dependencies if it can find them in the same directory as the binary package you're installing. As such, my scripts don't have to do the exrta work as long as the dependencies are out there to be downloaded. This can be witnessed if we try something more complex, such as ImageMagick, a command-line image manipulation tool:

$ pkg_get.sh ImageMagick-
ImageMagick-6.3.4.1-no_x11.tgz
ImageMagick-6.3.4.1.tgz
Ambiguous package name. Please be more specific.

$ pkg_get.sh ImageMagick-6.3.4.1-no_x11
Attempting to fetch/install package...
bzip2-1.0.4: complete
jasper-1.900.1: complete
jbigkit-1.6p1: complete
lcms-1.15: complete
netpbm-10.26.42: complete
ghostscript-8.54p1-no_x11:ghostscript-fonts-8.11p0: complete
ghostscript-8.54p1-no_x11: complete
libxml-2.6.29: complete
ImageMagick-6.3.4.1-no_x11: complete
... post-install message truncated ...
--- ghostscript-fonts-8.11p0 -------------------
... post-install message truncated ...

As you can see, a few tiny scripts can make a world of difference when trying to install packages in OpenBSD.

See Also:
More HiR Information Report Sysadmin Sunday articles

Related OpenBSD Man pages:
pkg_add(1)
pkg_info(1)
pkg_delete(1)

Sysadmin Sunday: Automate remote tasks with SSH

The SSH Suite can be leveraged to perform some pretty amazing tasks. I maintain hundreds upon hundreds of open systems as part of my job, but a good deal of my experience with SSH for batch processing came from a brief stint at a startup company that had employed a few hundred OpenBSD systems in an application-clustered environment. Keeping them in sync wasn't too hard once I came up with a few creative ways to get everything straight.

Currently, the most popular (and arguably most secure) SSH suite is OpenSSH which ships with almost all Open-Source operating systems and some commercial ones. Packages exist for most others, or there's always the source code.

For the un-initiated, SSH itself replaces RSH and Telnet as remote-access protocols for interactive logins. SFTP replaces the insecure FTP protocol with an encrypted, SSH-tunneled file transfer protocol. SCP replaces RCP with a secure remote copy protocol. At its core, SSH can simply authenticate these tools against the host's existing password database. The real power of SSH comes from its ability to use public keys to authenticate accounts. This kind of authentication is fragile if implemented improperly. If one account is compromised and the account has weakly-protected keys distributed to other systems, the impact of that one account can be much further-reaching. It is for this reason that I recommend using un-privileged accounts when using passwordless SSH keys.

Generating SSH Keys
To generate SSH keys for your account, you login to the account on the system that you'll be running automated or batch tasks from, then run ssh-keygen and follow the prompts, entering a blank password for the key.

-bash-3.2$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/axon/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/axon/.ssh/id_rsa.
Your public key has been saved in /home/axon/.ssh/id_rsa.pub.
The key fingerprint is:
5c:f4:64:3c:51:a3:a7:80:fd:fa:55:7a:06:f7:4d:84 axon@chimerabsd.labs.h-i-r.net

Distributing SSH keys
From here, you have to push the key out to each remote host by adding the contents of the id_rsa.pub file to the end of "authorized_keys" in the .ssh directory of the account you're going to be connecting to. My usual method goes something like this:

ssh user@remote.host "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub

If this returns an error such as ".ssh/authorized_keys: No such file or directory", then you need to go to the remote host and create the .ssh directory manually.  This ssh'es to the remote account, and runs "cat >> .ssh/authorized_keys", which takes any input and appends it to the authorized_keys file. At the same time, it's sending the contents of of id_rsa.pub to be appended.  

The above command will prompt you for a password for the remote user, then exit immediately. From that point on "ssh user@remote.host" should no longer ask for a password, and then the account can be used for remote automation. Note: You can also enter a password during key generation, and this password will then be prompted for when ssh'ing to remote hosts. This hinders automation, however.

I chose, instead, to make a script to push the keys out to multiple systems, since I'll be demonstrating batch processing of remote commands.  First, come up with a list of all the remote systems you wish to effect at the same time.  I'm going to choose from a subset of HiR Lab boxes that I've got running right now:

---

File: labs.h-i-r.net.list

---

backtrack.labs.h-i-r.net
bouncer.labs.h-i-r.net
chimera.labs.h-i-r.net
chimerabsd.labs.h-i-r.net
lampdev.labs.h-i-r.net

---

The key push script:

---

File: key.pu.sh

---

#!/bin/sh
# mass ssh key push script by ax0n@h-i-r.net 2008-01-06
# Pushes ssh keys out to an entire list of hosts from a file
if [ -z $1 ]
then
echo "Syntax: $0 server-list "
exit 1
fi
boxlist=$1
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "cat >> .ssh/authorized_keys" < ~/.ssh/id_rsa.pub done

---

Then run key.pu.sh [server-list-file].  It will ask for passwords for each machine.  That's the breaks, kid.

-bash-3.2$ ./key.pu.sh labs.h-i-r.net.list
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
axon@backtrack.labs.h-i-r.net's password:
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
axon@bouncer.labs.h-i-r.net's password:
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
Password:
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
axon@chimerabsd.labs.h-i-r.net's password:
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
axon@lampdev.labs.h-i-r.net's password:

Testing the SSH keys

Now that the keys are pushed, remote work will be simple.  Try to ssh to any of the systems in the list.  It should just work without a password:

-bash-3.2$ ssh axon@backtrack.labs.h-i-r.net
Last login: Sun Jan 6 22:34:20 2008 from 192.168.0.107
Linux 2.6.21.5.
backtrack ~ $ logout
Connection to backtrack.labs.h-i-r.net closed.

If you wish to run a single command remotely, then exit, you can also add the command to the end of the command line like this:

-bash-3.2$ ssh bouncer.labs.h-i-r.net uname -a
OpenBSD bouncer.labs.h-i-r.net 4.2 GENERIC#851 sparc

Similar to my key push script is one that I wrote to run a command across all of the systems in a list file:

---

File: mass-ssh.sh

---

#!/bin/sh
# mass ssh script by ax0n@h-i-r.net 2008-01-06
# Runs the same command across an entire list of hosts from a file

if [ -z $2 ]
then
echo "Syntax: $0 server-list command-string"
exit 1
fi
boxlist=$1
shift
commandstring=$*
for box in `cat $boxlist`
do
echo "-=-=-=- $box -=-=-=-"
ssh $box "$*"
done

---

Let's see it in action by running the "uptime" command across my lab environment:

-bash-3.2$ ./mass-ssh.sh labs.h-i-r.net.list uptime
-=-=-=- backtrack.labs.h-i-r.net -=-=-=-
22:53:40 up 5 days, 5:21, 2 users, load average: 0.24, 0.05, 0.02
-=-=-=- bouncer.labs.h-i-r.net -=-=-=-
10:07PM up 2:31, 0 users, load averages: 0.28, 0.15, 0.10
-=-=-=- chimera.labs.h-i-r.net -=-=-=-
16:53 up 1 day, 6:02, 3 users, load averages: 1.44 1.32 1.15
-=-=-=- chimerabsd.labs.h-i-r.net -=-=-=-
10:54AM up 2:32, 1 user, load averages: 0.32, 0.14, 0.10
-=-=-=- lampdev.labs.h-i-r.net -=-=-=-
16:53:19 up 2:08, 0 users, load average: 0.08, 0.02, 0.01

Notice how I don't get prompted for any passwords.  It's important to note that my account on these systems is a non-privileged account without access to sudo or membership in any special system groups such as root, wheel, adm, or bin.  You could use this script with a privileged account for adding or removing users, but I'd keep very tight security on any system with passwordless ssh keys.  It's mostly useful for seeing who is logged in, checking for high system loads, and whatnot.

Transferring files

sftp works much like ftp.  The syntax is "sftp user@host.name" and then it either prompts for a password, or if SSH keys are present, uses those.  After that, most of your usual FTP commands work as usual. In the following example, I do not have SSH keys from chimera (my laptop) to itself (localhost):

chimera$ sftp axon@localhost
Connecting to localhost...
Password:
sftp> ls
Desktop Documents Downloads Library Movies Music Photos
Pictures Public Sites myphotos scp.sh
sftp> cd Photos
sftp> ls
06-06-07_1911.jpg 06-25-07_0600.jpg PowerLinkChain.jpg
sftp> get PowerLinkChain.jpg
Fetching /Users/axon/Photos/PowerLinkChain.jpg to PowerLinkChain.jpg
/Users/axon/Photos/PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

Better yet, in my opinion, is scp, the secure replacement for rcp. The syntax is simply the same as "cp" with the exception of it accepting a user and host name and a colon in front of the filename. This allows you to copy files over the network via an encrypted channel from the command-line. I'll do the same thing as before, from chimerabsd (which has ssh keys to chimera) to my home directory locally:

-bash-3.2$ scp axon@chimera.labs.h-i-r.net:Photos/PowerLinkChain.jpg ~
PowerLinkChain.jpg 100% 12KB 12.4KB/s 00:00

And as you guessed, I also have a program for pushing a file to a list of hosts