Building a honeypot army with Pi, EC2 and MHN

I've been thinking about honeypots a lot over the last year. I have a bunch of Raspberry Pi, Atheros-powered boards, old Linksys routers and other devices kicking around my home office that would make good honeypots, and my thoughts always wander back to wishing I could easily consolidate the logs from an army of diverse honeypots to a central location for reporting. That's what initially drew me to MHN -- despite development being pretty much stalled for the past 2 years save for minor fixes here and there.

It's kind of a shame, really, the state of blight attained by the once-vibrant intel-sharing honeypot-loving ecosystem. There are a couple of active projects (telnetlogger and cowrie among my favorites) but hpfriends vanished, and most of the popular honeypot projects haven't seen but a fistful of commits in years. The STIX/TAXII ecosystem always seems like it's in a state of flux for anyone outside of FS-ISAC. All the while, there seems to be this undertone of something big coming with regards to open sharing of threat information among the incident response crowd.

For all their warts, some honeypots and analysis tools still work pretty well. I started putting together a presentation for SecKC shortly after beginning my tinkering with MHN, and those slides can be found here. This post is more of a walk-through of setting up and managing MHN and honeypots than the presentation was.

Setting up MHN

The instructions on the MHN github page work well. I installed MHN using a vanilla EC2 Ubuntu AMI on a t2.micro (free tier) instance without any trouble, but as we added more honeypots to it, a bump up to t2.small was needed to keep it from running out of memory on a daily basis, and I'm still not sure how well that will scale. A few members of the SecKC community have set up almost 50 honeypots* and pointed them at this instance of MHN. It's logging, at times, more than 100,000 hits per day, and the UI is getting kind of sluggish. MongoDB runs out of memory and requires a restart a few times per week, but I just use a cron script to check it every 5 minutes, and nudge the services if needed. This is a trade-off I'm willing to make. On-demand pricing for t2.small costs about $17 per month, and moving up to t2.medium or r3.large (where it probably belongs, due to MongoDB being a pig) sets me up for a monthly Amazon bill of $35-124, which I don't feel like shelling out of my own pocket.

*( with MHN, each honeypot software package counts as a "sensor," even if you're running two, three or more on the same instance - it's more like 30 distinct VPSes, Raspberry Pi and cloud compute nodes)

When you add users to the web UI, they are all admins. Admins can disable other user accounts. This is kind of a pain. I recommend installing the sqlite3 package on your MHN instance if you end up adding more users. I'll cover MHN databases toward the end. Additionally, I think all users can edit deploy scripts and delete honeypots, even if they're not admins. Be careful who you give access to the MHN console.

I'd recommend disabling the feed of data back to ThreatStream/Anomali (as found in the MHN setup page) by running "sudo /opt/mhn/scripts/disable_collector.sh" after you get it installed. In fact, I also removed /etc/supervisor/conf.d/mhn-collector.conf entirely, then ran "sudo supervisorctl reload" to update the configuration.

Setting up TLS with Let's Encrypt

Since there's a login and password on MHN, it's probably best to set up HTTPS. Since the MHN server is Ubuntu 16.04 with nginx, I followed this DigitalOcean guide, though I left the firewall stuff alone, since we're relying on Amazon's AWS security groups for traffic control. Just make sure both :80 and :443 are open to everywhere. I added the static locations for the acme challenge, and a redirect to https at the beginning of the "Default" sites-enabled nginx config file, /etc/nginx/sites-enabled/default like so:

server {
    listen       80;
    server_name  mhn.h-i-r.net;

    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/www/letsencrypt;
    }
    location / {
        return 301 https://mhn.h-i-r.net$request_uri;
    }
}

Make sure you add the appropriate crontab entry for certbot, and include the renew-hook script as in the DigitalOcean howto guide. Mine looks like this:

20 3 * * * certbot renew --noninteractive --renew-hook /root/letsencrypt.sh

Mnemosyne WebAPI

I'd mentioned the Mnemosyne WebAPI earlier. It requires TLS in order to work properly, and runs on port 8181. If you don't have certs, it won't even try to start. You don't really need to worry about this unless you want to run some custom reports against Mnemosyne outside of MongoDB, such as with some Python scripts.You should probably use your Lets Encrypt certs for this. Symlinks will keep these duplicates up-to-date as certbot rotates your certificates.

sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/privkey.pem /opt/mnemosyne/server.key
sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/cert.pem /opt/mnemosyne/server.crt

If you're using this feature from outside of the Amazon VPC MHN resides in, make sure you adjust your Amazon security groups to allow access to this service, but I would not recommend you expose this port publicly. There is some good API documentation here.

Deploy scripts

Deploy scripts are an easy way to take a bare-bones OS (like a fresh Raspbian Jessie Lite installation, or an Ubuntu VM) and add a Honeypot to it. When you select a deploy script, you see a deploy command (usually in the form of wget ... -O deploy.sh ; sudo bash deploy.sh ...) that should perform all of the actions needed to install the honeypot in question onto the operating system noted. Most are Ubuntu but there are a few specific to Raspberry Pi and CentOS. You can edit the scripts, the names and the notes for each of these to make local tweaks. MHN ships with a number of useful honeypot deploy scripts. Note that I've adjusted and re-named a few in this screen shot, so they won't perfectly match a fresh MHN install:

This isn't what drew me to MHN, but it's a feature I've come to love, and being written in shell (and sometimes a bit of Python), it's stuff that I'm actually comfortable editing. Since some honeypot projects went dormant before MHN, some of the deploy scripts still work fine. A few just need tweaks for minor changes in the host OS (e.g. Ubuntu 16.04LTS switched to the systemd init). In cases like Cowrie, though, the honeypot has evolved far beyond what it looked like when the deploy scripts were created. 

I've been adjusting a few of these and submitting pull requests, but you could also use my MHN fork for the time being, or simply pick and choose some of the deploy scripts out of my fork, and manually update your MHN instances that way.

Here's a quick animation of a Conpot (SCADA honeypot) deployment on EC2. It ran in just a few minutes. Most of the scripts take a little longer on a Raspberry Pi -- particularly the old 600MHz single-core Model B.

HPFeeds

HPFeeds is the magic that ties the honeynet together. It's a lightweight, authenticated publish/subscribe protocol that runs on port 10000 by default. On the back-end, it's using MHN's fork of a framework called mnemosyne, which aims to normalize and store honeypot data while providing a web API for reporting. I'll cover the database part shortly. Each deploy script calls a registration routine (registration.sh) which gets a UUID and a secret key from MHN. This UUID is the honeypot's unique identifier, and the secret key is the authentication. Each honeypot publishes attack details to the hpfeeds broker that MHN installs. Each honeypot program has at least one channel it can publish to. For example, all conpot instances will publish to the "conpot.events" channel. Some honeypots have multiple channels, like one for events and one for captured malware. As the honeypot is installed, deploy.sh plugs these variables in to the configuration so the honeypot knows where to publish the details to.  Here's an example of the clause it added to conpot.cfg when I deployed it to a Pi in my lab:

[hpfriends]
enabled = True
host = mhn.h-i-r.net
port = 10000
ident = cb53aad4-7f08-1337-beef-0ad36352028b
secret = eteKJ6frn9bwQ1Hs
channels = ["conpot.events", ]

Just as a honeypot can publish to a channel, it's possible to set up a subscription to these channels. If you connect to HPFeeds with a tool like hpfeed-client from the python hpfeeds package, you can watch in near-real-time as attacks happen. Assuming you have network connectivity, a valid identifier, secret key and permission to subscribe to a channel, you can run the client from anywhere, and it'll pull the feed that MHN is getting. In the below example, I'm only querying dionaea.connections and cowrie.sessions. No dionaea connections showed up in the few seconds I had the client running, however.

To add a feed client user, I had to copy the "add_user.py" script from the hpfeeds git repository to /opt/mhn/env/bin on the MHN server, then run it inside the virtualenv. The syntax is [identifier] [secret] [publish feeds] [subscribe feeds] and since we don't need to publish with this tool, just subscribe, leave the publish empty (in quotes) and a list of subscriptions for your new account in quotes, comma separated like so:

$ cd /opt/mhn
$ source env/bin/activate
$ python env/bin/add_user.py test s3cr3t "" "wordpot.events,amun.events,cowrie.sessions,shockpot.events,dionaea.connections,snort.alerts"

Add as many feeds to the subscription as you like. These can be found in the hpfeeds fields of your deployed honeypots. This is the list of channels that the internal mnemosyne feed is subscribed to, which is likely all the feeds generated by all the honeypots MHN knows about:

• amun.events
• beeswarm.hive
• beeswarn.feeder
• conpot.events
• cowrie.sessions
• cuckoo.analysis
• dionaea.capture
• dionaea.connections
• elastichoney.events
• glastopf.events
• glastopf.files
• kippo.sessions
• mwbinary.dionaea.sensorunique
• p0f.events
• shockpot.events
• snort.alerts
• suricata.events
• thug.events
• thug.files
• wordpot.events

To access the hpfeeds stream, you can use an HPFeeds client library, or the reference client, written in python. An example command to pull the channels we set up earlier:

hpfeeds-client -i test -s s3cr3t -c wordpot.events -c amun.events -c cowrie.sessions -c shockpot.events -c dionaea.connections -c snort.alerts subscribe

The output should look like this, as sessions start rolling in (it can take a while if your honeypots sit idle a lot. 

[feedcli] connected to @hp2
[feedcli] publish to cowrie.sessions by 4fe5c378-7ec7-11e7-b58e-abc36352928a: {"peerIP": "171.243.14.183", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60332, "session": "abb74fda35ef", "startTime": "2017-09-14T12:13:30.060296Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:30.524580Z", "unknownCommands": []}
[feedcli] publish to cowrie.sessions by f9b8c9d8-7ede-11e7-b58e-abc36352928a: {"peerIP": "171.243.14.183", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60349, "session": "23ee29fca4f0", "startTime": "2017-09-14T12:13:32.161036Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:32.857748Z", "unknownCommands": []}

You can do quite a bit with just the HPFeeds data stream, for instance, if you funnel it into a database with some shell scripts, or run some post-processing on the HPFeeds log with awk.

Choosing a honeypot package

I have done a lot of testing over the past 2 months. Some of the deploy scripts needed some work (see above) but a lot of packages still run fine. Here are the ones I recommend. Note that Cowrie and Kippo deployments will move your real SSH server to port 2222, so be prepared for that.

For Raspberry Pi, installing the most recent image of Raspbian Jessie Lite from an image is probably your best bet, though the full Raspbian/PIXEL installation, from an image or from NOOBS also works fine on the Pi2 and Pi3. Most of my testing was done with the old-school Pi Model B and low-capacity SD cards (2 and 4GB). On the Pi, Cowrie, Dionaea and ShockPot are my favorites. WordPot also runs well, but you need to choose between it and ShockPot if you want something versatile on HTTP, and WordPot uses more resources. ConPot installs and runs properly, but it seemingly reports every single HTTP connection as an attack, which is noisy and counter-intuitive in my opinion.  Kippo works, but cowrie does it better. Amun works, but it's got a lot of overlap with Dionaea, which seems to provide more details about attacks in the logs.

On Ubuntu 16.04 LTS, my picks are Cowrie, Amun and WordPot. If you have a lot of RAM to spare, add Snort to the instance as well, because it will report all kinds of things that trigger an IDS signature from the EmergingThreats rule repository. Snort takes some manual configuration changes after install. ShockPot works, but as with the Pi, you can really only run one HTTP honeypot. ConPot works, but it's noisy. Kippo works, but Cowrie is better. Dionaea is broken for 16.04, hence my recommendation of Amun. I've spun up a bunch of Ubuntu honeypots with Cowrie, Amun, WordPot and Snort all running on the same instance in EC2 and Google Compute Engine.

Confirmed broken on Ubuntu 16.04: Suricata and Glastopf. I haven't invested much time in fixing these, but Glastopf seems to rely on ancient version of PHP that's (thankfully) missing from the package repository, and I'm not 100% sure what Suricata's deal is yet, but it looks to be something that can be fixed in the config file without much of a problem.

The SecKC crowd had questions about the safety of deploying honeypots. For the most part, the ones I discuss above are unlikely to lead to full shell access, but you can't be too sure. I had one attacker running tons of curl/wget commands from a Kippo honeypot last year, fetching ads, and essentially using my honeypot as a click fraud drone. Here was my advice from the presentation at SecKC:

Databases

I've found the MHN web front-end to be a bit lacking aside from the most basic of reports, and as I'd mentioned earlier, there's a problem when you add users through the web UI.

There are 2 database servers used by MHN, sqlite and mongodb. The majority of the MHN metadata is in sqlite. Use the ".tables" and ".schema [tablename]" commands to explore MHN's sqlite database, if you get curious. The tables user, roles_users are the most useful. All of the deploy scripts are stored in sqlite as well. They're imported from the shell scripts in the git repository upon installation and the scripts in /opt/mhn/scripts/deploy_* are never referenced again.

If you want to revoke admin privileges from all but the user account you set up when installing MHN, I recommend doing this:

$ sudo sqlite3 /opt/mhn/server/mhn.db
sqlite3>  update roles_users set role_id=2 where user_id > 1;
sqlite3> .exit

Most of the attack data, reporting and statistics are held in mongodb. You can run the "mongo" command line tool and start running queries directly. Mongo has "collections" instead of tables, and an interesting query syntax. There are a few databases, but the most useful are hpfeeds, where the subscription information is held, and the grand-daddy of them all, mnemosyne, which stores the normalized honeypot data. In reality, the Mnemosyne WebAPI might be easier for some to use, but I'm a fan of getting messy in the database. By default, the Mongo shell only displays 20 records at a time (type "it" to  continue) but I found out you can alter the default maximum results by creating a file called ".mongorc.js" in your home directory and adding this line to it:

DBQuery.shellBatchSize = 100

Here are a couple of useful example queries I've come up with.

See the most recent attacks:
> use mnemosyne
> db.session.find().sort( { timestamp : -1 } );

Find records for IP addresses that start with "192.168"
> use mnemosyne
> db.session.find({source_ip : /^192\.168.*/ })

Delete all records of a specific IP address (e.g. yours) from the records:
> use mnemosyne
> db.session.remove({source_ip : "192.168.1.94"})

Delete all records older than 7 days (in the event Mongo is getting sluggish. Change 7 * 24 to n * 24 for the number of days you want to save):
> use mnemosyne
> db.session.remove( { timestamp: { $lt: (new Date((new Date()).getTime() - ( 7 * 24 * 60 * 60 * 1000 ) ) ) } } )

Get a list of IP addresses based on the number of honeypots they've connected to in the last day (I've limited this to SSH)
> use mnemosyne > db.session.aggregate([
    { $match:
        { timestamp: { $gte: (new Date((new Date()).getTime() - ( 1 * 24 * 60 * 60 * 1000 ) ) ) },
        protocol: "ssh" }
    },
    { $group: { _id: "$source_ip", honeypots: { $addToSet: "$identifier"    } } },
    { $unwind: "$honeypots" },
    { $group: { _id: "$_id", honeypotcount: { $sum: 1 } } },
    { $sort : { honeypotcount : -1} },
    { $limit : 50 }
]);

Fun stuff

Brandon from SecKC and I collaborated on this fun dashboard, based on Rob Scanlon's Tron Legacy Encom Boardroom visualization. Brandon did all the code tweaks and wrote the middleware to get data into it. I just played project manager once the MHN firehose was ready. It really is a sight to behold. The "satellites" are approximate geo-locations for the deployed honeypots. Push-pins on the globe are attackers. The center pane is a live feed of attacks, with usernames and passwords where applicable. The right pane is a top attacker list and a map (if you click on the attacker) with more details. The chart at the bottom is the last 24 hours of attacks. Some of the panels won't display data unless you're logged in to avoid sharing the honeypot IPs publicly -- we're working on tweaking the back-end so that it can work well, safely, without authentication.

Sysadmin Sunday: parse strings with spaces using shell script

I run into this once in a while: I'm trying to perform some operation on a bunch of files or a big line of text, and a space in the filename or text file janks everything up. Take for example all these recordings from a podcast that got batch-named with spaces in them.

Chimera:Recordings axon$ ls

(110) - .mp3 (12) - .mp3 (18) - .mp3 (39) - .mp3 (79) - .mp3

(111) - .mp3 (15) - .mp3 (3) - .mp3 (70) - .mp3

I really don't want spaces in the names. No problem, just use ls -1 (the number one) to list the files on their own line, and use sed or something for renaming them and changing every space to a null character, right?

Chimera:Recordings axon$ for file in `ls -1`

> do mv "$file" `echo $file | sed s/" "//g`

> done

mv: rename (110) to (110): No such file or directory

mv: rename - to -: No such file or directory

mv: rename .mp3 to .mp3: No such file or directory

mv: rename (111) to (111): No such file or directory

mv: rename - to -: No such file or directory

mv: rename .mp3 to .mp3: No such file or directory

[truncated]

That did not go as planned...

There are a few interesting ways to solve this one. The actual reason for this problem is your shell's internal field separator. When iterating over some input (here, the results of "ls -1"), the shell interprets any kind of whitespace as a field separator, including spaces, tabs and newline characters.

Although there are some other clever ways to get around this limitation when dealing with filenames specifically, my favorite solution to this problem works on any whole line of input regardless its source, whether reading a text file and operating on it one line at a time or taking filenames as input from another command such as ls or find. You simply have to use something that can accept spaces and requires a newline character in order to set a variable. Of course, I'm talking about a rather unsavory (but totally viable) use of the read command, which most unixy shell-script writers are familiar with when they require user input. Check it:

Chimera:Recordings axon$ ls -1 | while read file

> do mv "$file" `echo $file | sed s/" "//g`

> done

Chimera:Recordings axon$ ls -1

(110)-.mp3

(111)-.mp3

(12)-.mp3

(15)-.mp3

(18)-.mp3

(3)-.mp3

(39)-.mp3

(70)-.mp3

(79)-.mp3

You can also remap the $IFS variable to contain a newline, but be sure to unset it afterwards (if using BASH, this will set it back to default), or your shell will act differently than you likely expect when you're done. Messing with the internal field separator can be useful for other things (such as parsing /etc/passwd or handling CSV files) but honestly I'd probably be more inclined to use awk for those. If we remap IFS to a newline, our original script that errored out above works just fine.

Chimera:Recordings axon$ IFS=`echo -en "\n\b"`

Chimera:Recordings axon$ for file in `ls -1`

> do mv "$file" `echo $file | sed s/" "//g`

> done

Chimera:Recordings axon$ ls -1

(110)-.mp3

(111)-.mp3

(12)-.mp3

(15)-.mp3

(18)-.mp3

(3)-.mp3

(39)-.mp3

(70)-.mp3

(79)-.mp3

Chimera:Recordings axon$ unset IFS

Links: 2010-12-08

From our Delicious feed today:

SANS - Computer Forensics and Incident Response with Rob Lee - Good to know, I suppose. I wonder if full erasure could be automated?
#tags: forensics linux infosec privacy

Police Hand Out The Club To Keep KC Residents Safe - Kansas City News Story - KCTV Kansas City - The fail is strong with them!
#tags: locks lockpicking physicalsecurity humor fail

What Car Thieves Think of the Club - NYTimes.com - This is why I laughed that cops handed out The Club to people.
#tags: locks security automotive lockpicking physicalsecurity

What's Your Favorite Old School Software? - While much of the software I still rely on daily could qualify as "old school" -- you know, awk, vi, and grep -- in the spirit of this post, I offer to you Telemate, Lotus WorksSuite and XTree Gold
#tags: dos oldschool software

makeit.png (PNG Image, 1124x777 pixels) - Scaled (80%) - Best. Poster. Ever.
#tags: cartoon humor

Nessus XML parsing with awk

At the office, I use Nessus for automated network scanning and patch auditing. With credentials and proper tuning of the scan policy, Nessus is a very powerful tool for more than skript kiddie network scanning. This leaves me with a whole bunch of data to wade through on a weekly basis.

Usually, I only concern myself with the high-severity issues for weekly reports, then as I have time, I dig deeper into the more trivial problems. Still, this required me to manually open the scan files, filter them by severity, and export the data. I got tired of that and made a quick and really dirty XML parser (.nessus files are XML) with shell and grep. It was horrendously slow.

Andy, a fellow KC2600-er helped me wrap my brain around some of the finer points of awk to make it more efficient. This is slightly modified from the one I use at work, which is part of a bigger script that does other things. I figure it's useful for others who use Nessus regularly. The script is here.

Basically, it stores the HostName tag when it encounters it, then iterates through the lines, storing them temporarily until it runs into a line indicating a high-severity plugin has been triggered (severity level 3), then it spits out the host name and the plugin that was triggered. I probably could write the whole thing in awk, but I wrapped it in a little bit of plain old shell script.

Output looks something like this:

Windows
----------------------------------------------------
x.x.x.19:MS10-062: Vulnerability in MPEG-4 Codec Could Allow Remote Code Execution (975558)
x.x.x.19:Adobe Reader <= 9.3.4 / 8.2.4 CoolType.dll SING Font 'uniqueName' Field Parsing Overflow (APSA10-02)
x.x.x.20:MS10-066: Vulnerability in Remote Procedure Call Could Allow Remote Code Execution (982802)

Mac
----------------------------------------------------
x.x.x.8:Mac OS X AFP Shared Folders Unauthenticated Access (Security Update 2010-006) (uncredentialed check)

Linux
----------------------------------------------------
x.x.x.40:PHP 5.2 < 5.2.14 Multiple Vulnerabilities

awk over non-interactive ssh sessions

I had trouble wrapping my brain around this one today. I was attempting to grab the second field of ps(1) output to display only the PID of a given process. Of course, if we had pgrep and pkill, this would be a no-brainer. But we don't.

Note, when I use grep on ps(1) I also pipe it through grep -v grep (which ignores any line containing the pattern "grep") so that grepping for some_process doesn't end up like this:

root 751 200 0 Jun 25 ? 3:11 /usr/sbin/some_process -d
axon 2429 222 0 4:54:59 pts/1 0:00 grep some_process

First attempt:

ssh somebox "ps -ef | grep some_process | grep -v grep | awk '{print $2}' "
The output, though, was the whole line out of ps.

root 751 200 0 Jun 25 ? 3:11 /usr/sbin/some_process -d

I attempted escaping the ticks, double ticks, escaped double ticks, double quotes, and all kinds of madness. Nothing was working. I was getting either a whole ps line (as if awk wasn't even there) or syntax errors from awk.

Finally, I ask a co-worker (the biggest shell geek I know) pointed out that it was being frobbed by two separate shells. He gave me a somewhat complicated line to use, but I figured out an easier way. The second shell was interpreting $2, thinking I was referencing a shell argument, and was passing nothing to awk. Solution? Escape $2.

ssh somebox "ps -ef | grep some_process | grep -v grep | awk '{print \$2}' "
751

Hooray! Hopefully this helps some poor sysadmin somewhere when the time comes to reference variables remotely in something like perl or awk.

Twitter Followers/Friends from the CLI

I started getting curious on twitter. I had two questions:

1. Who am I following that's not following me back? (i.e. can Martin Roesch hear me? The answer is no, he can't)
2. Who is following me that I'm not following back?

Already familiar enough with the Twitter API, I threw together some quick and ugly command-line foo, and @digitaljestin wanted to know how I did it. This is REALLY ugly, and could use a lot of refinement. That said:

I'll probably program a quick stand-alone newlisp or php tool for this over the weekend. Regardless, here's how I did it on the CLI.

First, Twitter will only hand you 100 friends and followers at once. If I were going to automate this, I would poll the followers_count and following_count attributes from http://twitter.com/users/show/username.xml to figure out how many "pages" I needed to fetch.

If you have 203 followers, you will have to do three requests for follower info. Same with friends (those whom you follow). I had over 200 (but less than 300) each. So I did 3 of each request.

I'm only interested in the screen_name attribute within the XML of each. Note that I'm doing a lot of cheap grep | awk crap here, so it just builds lists of screen names without any markup.

$ wget http://user:password@twitter.com/statuses/followers.xml \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' > followers.txt

$ wget http://user:password@twitter.com/statuses/followers.xml\?page=2 \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> followers.txt

$ wget http://user:password@twitter.com/statuses/followers.xml\?page=3 \

| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> followers.txt

$ wget http://user:password@twitter.com/statuses/friends.xml \
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' > friends.txt

$ wget http://user:password@twitter.com/statuses/friends.xml\?page=2 \
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> friends.txt

$ wget http://user:password@twitter.com/statuses/friends.xml\?page=3\
| grep "<screen_name>" | awk -F"[\<\>]" '{print $3}' >> friends.txt

Then, I just sorted them:
$ sort friends.txt > friends-sort.txt
$ sort followers.txt > followers-sort.txt

Using diff, it's easy to tell who is not following you, and who you aren't following.
The < shows lines that only appear only in the first file (ones you follow only). The > shows lines that only appear only in the second file (ones following you). Grepping for only lines that start with < and > avoids all the patch-file line offset stuff. Some diffs have varying syntax to do this, but letting grep filter it should work across more platforms.

$ diff friends.txt followers.txt | grep "[<>]" | sort
[excerpt]

< H_i_R
< Hak5
< KCWeather
< Scobleizer
< Veronica
< bacontwits
< beseKUre
< brightkite
< datalossdb
< hackadaydotcom
< ihacked
< ihackstuff
< kingpin_
< milw0rm
< mroesch
< obsessable
< om
< packetlife
< pauldotcom
< schneier
< textfiles
< wilw
< window
------------------ (split added by ax0n)
> BlackHatUSA
> Computersaurus
> HacClearwater
> HackersAlerts
> HackerspacesBot
> SOURCEBoston
> SecuritySatan
> quine
> reverz
> rsreese
> secureideas
> securitypro2009
> stopthemanga

UNIX Tip: the tr command, newlines and more

tr is used to translate characters. It can translate all uppercase letters to lowercase. It can delete all occurances of a single character, and much more.

To turn all uppercase letters lowercase (a useful thing when parsing user input from a shell script), use this:

bash-3.2$ echo This Is CrAzY! | tr [A-Z] [a-z]
this is crazy!

Using tr -d, you can remove any characters from a string as well. Here, I remove several punctuation marks:

bash-3.2$ echo "You're right. That *is* crazy." | tr -d \'\*\!\?\.
Youre right That is crazy

One of the best ways to use tr, though, is for converting newline characters to something useful.

When running shell commands or writing shell scripts, most people can quickly adapt to using commands such as cut or awk to manipulate tabular data. See also: more awk recipes. Let's say we have a text file with the following values, and we want this data to be represented in CSV format, all on one line. This is where newlines make things tricky.

0.00
128.00
49.00
7031
4218

tr can tackle it easily, using "\n" to represent the newline character. Just translate newlines to commas, and you're good to go:

bash-3.2$ cat file | tr "\n" ","
0.00,128.00,49.00,7031,4218,

Similarly, you can take large chunks of text and replace all newlines with a space if you wish to un-do old-school text-editor formatting before importing into a word processor.

tr is a somewhat obscure command, but it can save a lot of headache, particularly when trying to format text output. The same result can be achieved (albeit with much more complex commands) with awk or sed, so this is a good one to know about, and it's found on almost every posix-compliant platform (and in cygwin)

The Geek 100 Pt. 2: Sysadmin and Network Skills

See the whole series: The Geek 100

This is a list of 100 basic things and skills every geek should have. I've broken this series up into five parts. Let's face it: a list of 100 things would be tedious to wade through. Over the rest of the week, look for twenty more skills to show up daily. The skills assume you have done it in the past and can remember how to do it right now (or, like a good Geek, you've jotted it down in one of your notebooks). Having it in your personal notebook is okay. Scrambling to the Internet means you don't have the skill. Yet...

This one is for all your Information Technology geeks, although it's a bit UNIX biased.

Systems Administration. All geeks should be able to:

1. Compile a kernel
2. Set up a file/print server that works across most operating systems
3. Set up RAID on a server
   
4. Set up an AMP server
5. Set up an Internet mail server that won't get RBL'd
   
6. Set up an RDBMS
7. Use a Logical Volume Manager
8. Use find with xargs
9. Use sed and awk effectively
10. Use sysctl
    

Networking. No, not Social Networking. All geeks should be able to:

1. Explain the difference between PAT, NAT and a DMZ
2. Flash new firmware to routers and switches
3. Make an ethernet crossover cable
4. Run cable through walls without tearing them up
5. Set up port security on a network switch
6. Terminate a fiber optic cable by hand
7. Terminate a punch-down patch panel
8. Terminate category 5 cable with RJ-45 plugs
9. Trace cables
10. Use a network sniffer or protocol analyzer

See the whole series: The Geek 100

UNIX tip of the day: More awk recipes

Awk, as we have mentioned before, is a ridiculously handy utility that often goes under-appreciated by systems administrators and UNIX geeks alike.

A few days ago, a colleague of mine told me about some complex awk magic that he'd implemented in order to acquire not only the matching line of an input stream (in this case, a log file), but the two lines prior to that matching line which contained some useful information as to what was going on. It was an elaborate solution that worked well, but I swooped in with a much simpler recipe to do the same thing. This prints the two lines prior to the matching expression as well as the line containing the expression. Certain incarnations of grep can do the same thing, but this way you can also format the lines if you know your awk-fu.

$ awk '/some-regex/{print two "\n" one "\n" $0};{two=one};{one=$0}' /some/file.log

I actually keep a bunch of handy commands and UNIX tips written down. These are things that I know I won't need to use very frequently, but know I'll eventually need again.

I'll share some more awk magic with you from within its pages.

Get only the last field of a line that matches a regex:

$ awk '/some-regex/{print $NF}' /some/file

This works because NF contains the number of fields found in the line. $NF, then, contains the value of that last field. Just like $1 would contain the value of the first, if NF is 5, $NF would have the value of the 5th (and last) field. I love this one.

Example:
I'll set the field separator to a / and use awk to get only the last entry from the directory structure with find:

Raw find output:

$ find .
.
./.localized
./images
./images/apache_pb.gif
./images/gradient.jpg
./images/macosxlogo.png
./images/web_share.gif
./index.html

Now with awk:

$ find . | awk -F/ '/images/{print $NF}'
images
apache_pb.gif
gradient.jpg
macosxlogo.png
web_share.gif

If you're into AIX, a lot of the configuration files are in "Stanza" format. That is, a label, followed by a bunch of data and then a blank line between records. Awk can get just the one stanza you want from a stanza file. Example here is the /etc/security/user file on AIX, which tracks security profile information for every user on the system. The "default" stanza is an important one, as anything within it gets propagated to all users first, then any deviations from the default happen in the users' own stanzas:

# awk '/^default/,/^$/ {print}' /etc/security/user

Truth be told, stanza format and some of its variants are popular in other operating systems, but this particular awk recipe works best on AIX.

Have any awk-fu? Let's see some of your favorites. The comments are open!

UNIX Tip: Getting data out of wtmpx on Solaris

I recently had to do an audit and see who was logged in when some performance issues were being reported on one of our Solaris boxes. The problem? The performance issues had been talked about in an email thread for a long time before it fell onto my plate. The only way to find out who was logged in during that time frame was to pull data out of old backups of the wtmpx file.

On Solaris, wtmpx contains data such as what IP address a user connects from, when they logged in, and how long they stayed on. It's a binary, non-human-readable format. I did some research to figure out how to get usable data from it. The fwtmp command converts the wtmpx records to ascii, making it easy to parse them, for instance, with Awk. Unfortunately, this utility isn't in the path, and I don't know why. No matter, though. I'll tell you where it's at. It's a stream converter, so it takes the raw wtmpx file as input, and bars out plain text. This is how I usually run it:

$ /usr/lib/acct/fwtmp < /var/adm/wtmpx > wtmpx-`date +%Y-%m-%d`.txt

This creates an ascii wtmpx file with the current date as part of its filename.

See the man page for fwtmp for more information. It can also be used for importing records into wtmp. With some scripting, a nice log-rotation mechanism can be implemented.

UNIX tip of the day: cut columns from a text file

Most people who know their UNIX shells know of "cut", a utility which can grab only certain parts of each line within a text file. I'm kind of a weather geek, so I'll use some raw CSV METAR data from Weather Underground for this demo.

For instance, the following will grab the forth through tenth byte from each line of Data.csv:
$ cut -b 4-10 Data.csv
eCST,Te
53 AM,3
3 AM,37
3 AM,37
3 AM,37
3 AM,37
3 AM,37
3 AM,37
0 AM,37


If Data.csv happens to be a genuine comma-separated file, the following will grab the second and fifth fields (columns) from each line, using a comma as the delimiter between fields:
$ cut -f2,5 -d, Data.csv
TemperatureF,Sea Level PressureIn
37.0,29.79
37.0,29.77
37.0,29.75
37.0,29.69
37.9,29.66
37.9,29.65
37.0,29.65
37.4,29.61

This, as you can see, grabbed the Fahrenheit Temperature as well as the Sea Level Pressure (in Inches of Mercury), not like you cared. You're not here to check the weather. This is just a demo.

Unfortunately, cut always dumps the columns out in the order they're in in the file, regardless of which order you specify. For example switching the field order in the command line still nets you the identical output as before:
$ cut -f5,2 -d, Data.csv
TemperatureF,Sea Level PressureIn
37.0,29.79
37.0,29.77
37.0,29.75
37.0,29.69
37.9,29.66
37.9,29.65
37.0,29.65
37.4,29.61

Also, cut is really stupid about data that's organized in columns. Unless the formatting is very, very clean, it can be hard to get predictable results using cut.

For parsing data that's in columns (say, the output of the ls command, for example), it's better to take a crack at using awk. Awk is a somewhat intimidating beast for the unfamiliar user. For simply pulling data out of columns, though, it can't be beat.

Each column that awk identifies in a line is given an incremental variable, starting with $1. By default, awk uses white space as its field separator. Let's give this a try.

$ ls -la
total 16
drwx------+  2 axon  staff   136 Jan 14 13:50 .
drwxr-xr-x  14 axon  staff   646 Jan  4 17:38 ..
-rw-r--r--   1 axon  staff  2970 Jan 14 13:50 DailyHistory.csv
-rw-r--r--   1 axon  staff   672 Jan 14 13:50 Data.csv

Lets try to grab the owner and the file name only. That would be the third and ninth fields in the above listing.

$ ls -la | awk '{print $3 $9}'

axon.
axon..
axonDailyHistory.csv
axonData.csv

As you can see, there's no separation between the two fields now. You must separate your records within the print command. You can use a space, tabs, a comma, or whatever you wish. We'll just separate them with a space in quotes:

$ ls -la | awk '{print $3 " " $9}'

axon .
axon ..
axon DailyHistory.csv
axon Data.csv

Also, unlike cut, you can specify things in any order you wish. This is great for getting only the data you want from large CSV files or other tabular data right from the command line.

$ ls -la | awk '{print $9 "," $3 "," $1}'
,,total
.,axon,drwx------+
..,axon,drwxr-xr-x
DailyHistory.csv,axon,-rw-r--r--
Data.csv,axon,-rw-r--r--