Running pkgsrc on OpenBSD

After a discussion somewhere on teh webs, I decided to dig into the state of PHP 7.1 and PHP 7.2 on OpenBSD. The short version is "we're working on it" (in OpenBSD ports) but no ETA. However, being keen to NetBSD's pkgsrc distribution, I knew that they had been cooking up newer versions in their software tree. So I decided to kick the tires.

Pkgsrc is roughly NetBSD's equivalent to the OpenBSD/FreeBSD "Ports" repository, however, they've put significant effort into making it quite portable. It works in one way or another on other BSDs, Linux, OS X and even more esoteric platforms like Haiku and Illumos.

Initially, bootstrapping pkgsrc on OpenBSD 6.3-STABLE didn't work. Buried deep in my inbox from the pkgsrc mailing list in April, I found a hint from Sevan Janiyan about some patches that are needed to make it work. Partially, this is because OpenBSD uses both clang and gcc compilers in the base distribution on modern hardware.

Anyhow, on with the show.

First, check out the pkgsrc repository. You can do it with cvs:

env CVS_RSH=ssh cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P pkgsrc

It's going to churn for a few minutes while it downloads all the files.

When it's done downloading, you can move it to /usr (/usr/pkgsrc) if you want, but I usually just leave the pkgsrc tree in my home directory. Change into the pkgsrc directory:
cd pkgsrc

make a file called pkgsrc.patch with the following contents:

--- archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h
1 Aug 2017 22:21:17 -0000
+++ archivers/libarchive/files/libarchive/archive_openssl_hmac_private.h
5 Apr 2018 20:50:09 -0000
@@ -28,7 +28,8 @@

+#if OPENSSL_VERSION_NUMBER < 0x10100000L || \
 #include /* malloc, free */
 #include /* memset */
 static inline HMAC_CTX *HMAC_CTX_new(void)

Apply the patch:
patch -p0 < pkgsrc.patch

cd bootstrap

A "privileged" install requires root (via doas or su) but will store all of the binaries into /usr/pkg:
doas ./bootstrap --compiler clang

-- OR --

If you wish to build "unprivileged" without root, the binary packages will be installed in the "pkg" directory under your home dir.
./bootstrap --unprivileged --compiler clang

Then go have a coffee or something. It takes a while.

If it finishes up with out a screen full of errors, you're almost all the way there:

You'll need to edit your .profile (or .bashrc if you roll that way) to add the pkg/bin directory to your path. In a privileged install, add /usr/pkg/bin and /usr/pkg/sbin to your path. In an unprivileged install, add ~/pkg/bin and ~/pkg/sbin instead.

Adding these paths to the end of the PATH line in the default .profile should work for a privileged pkgsrc install:


You need that in your path, because pkgsrc portable requires you to use NetBSD's "bmake" in the pkgsrc tree, and bmake is compiled as part of the bootstrap. You can log out and log back in, or just run this command to get bmake into your working path.


Let's try building something. It works a lot like ports. It will recursively build any dependencies or libraries that are needed, and install them into the pkg directory before building the application you're trying to compile. I'll start with the latest version of PHP in pkgsrc, PHP 7.2, since that's how this whole journey started. I'm using doas since I did a privileged install.

cd pkgsrc/lang/php72
doas bmake

It compiled.

I'm going to run doas bmake install without running bmake test because pkgsrc isn't the boss of me.

And it all works.


Wi-Fi on OpenBSD just got a lot easier

... if you're running the latest OpenBSD-CURRENT snapshot, at any rate...

Last night, Reyk Floeter posted this teaser, hinting that phessler@'s work on this feature was inching toward completion.

And it's already in snapshots dated July 12, 2018 and newer. Some mirrors haven't gotten this snapshot yet, as of the time of publication. What this means is that you can load up all of your frequently-used wireless networks into your wifi adapter's /etc/hostname.if file, and it will attempt to auto-join them in the order they're listed. It'll be interesting to see if the installer for OpenBSD 6.4 uses this syntax if you set up a wireless network during the install process.

I like the simplicity of this, versus the complexity of configuring wpa_supplicant, NetworkManager, NetCtl and similar tools. I suspect if you've got only wpa-enabled networks in the list, it should be resistant to most evil twin attacks such as Karma, but I haven't tried that yet.

Documentation of this feature doesn't seem to be covered in the hostname.if or individual wireless driver man pages yet, but the above photo was obvious enough for me to create a working example configuration file for my daily-driver laptop running -CURRENT (on which I'm writing this article).

It looks like a lot of other good things are coming out of the g2k18 hackaton, including advances on unveil, a simple way to control filesystem-level access on a per-process basis, which Bob Beck presented at BSDCan 2018.


PHP/MySQL Articles updated for OpenBSD 6.3

No surprises, the existing instructions from OpenBSD 6.1 worked flawlessly for both the httpd and nginx web servers. I just made sure everything still works and updated some version numbers. Once again, I did all of the testing using OpenBSD's built-in vmm hypervisor on my personal laptop, and it did remind me of a few recent changes in vmm's network configuration that I had forgotten about.


OpenBSD 6.3 Early Release!

OpenBSD 6.3 was slated for release on April 15th, but it's already showing up on mirrors this morning. It looks like the full package tree is available only for the most popular platforms at the moment, but the install sets for all supported architectures are live on the two mirrors I use most frequently (sonic and ftp5). I haven't checked the rest of the mirrors. Elsewhere on the Internet, I saw a number of core developers confirm the release is underway almost two weeks ahead of schedule.

I'm looking forward to using the improved install script ("Please Listen Carefully as Our Menu Options Have Recently Changed") and taking advantage of some iterative improvements to the VMM hypervisor, such as snapshots and the ability to attach ISO images to VMs.


Bad idea? Let's put a Windows 2000 server on the Internet.

Today, I decided to install Windows 2000 Advanced Server onto my Dell Latitude D610. The laptop itself is a workhorse, if a bit dated. Mostly, I was just curious what would happen if I left it out on the Internet without any service packs or firewall rules* and I live-tweeted it as I did my research.

Here's my twitter thread with just a few additional notes added. pcap and IDS alerts are at the end:

Alright, so my ISP is giving me some firewall rules of their own, probably to stop the spread of EternalBlue exploit bots and WannaCry ransomware. Honestly, I appreciate it, but it's not helping me get pwned.

With that, here are the links to those:
Sanitized pcap (gzip): http://stuff.h-i-r.net/win2k.pcap.gz
Sanitized IDS log: http://stuff.h-i-r.net/win2k-ids-alerts.txt


Recovering passwords from a Casio graphing calculator

I bought this Casio CFX-9800G calculator in 1995. It's been through everything with me, even, apparently, my college electronics courses (per the "ELEC 120 Progs" thing I found on it):

I bought it because it was about 2/3 the price of the competing TI-81 calculator that was "required" for whatever math class I'd found myself sucked into that semester (I think statistics) and because it was the first calculator I'd ever seen with a color display, even if it was only 3 colors (Orange, green and blue). My math teachers hated anything that wasn't Texas Instruments, because they'd received formal training and supplies from TI. Of course, this extra annoyance was a bit of a pride point for me, and the one kid with his HP 48G.

Now, it sits on my desk at home for calculations that exceed the easy capacity of bc(1) and xcalc. Poking through the program menu, I ran into a few games and helper scripts I've written over the years. Some were password-protected.

It was about this time that I recalled buying the data cable and software. Somehow, I still had the CD on a spindle of old commercial software (among gems like Need For Speed SE, MechWarrior II, and CheckPoint Firewall-1 4.0) and my cable stash is organized enough that it was easy to find in the bucket labeled "Strange proprietary serial cables". One problem: Windows 10 doesn't like 25 year old Casio software. Rebooting my OpenBSD netbook into Windows 7 Starter Edition (ugh) did the trick, though. Add a USB/Serial dongle, and we're off!

I connect the cable, put the Casio software into "receive" mode, then do a data dump from the calculator itself. Things are looking up! Unfortunately, I can only see that these programs exist on the calculator, I can't do much with them aside from delete them from the archive.

I open the file in Notepad++ for giggles, and I'm pleasantly surprised. This whole catalog is ASCII. And I also found a password I haven't used since the late 90s.


Building a honeypot army with Pi, EC2 and MHN

I've been thinking about honeypots a lot over the last year. I have a bunch of Raspberry Pi, Atheros-powered boards, old Linksys routers and other devices kicking around my home office that would make good honeypots, and my thoughts always wander back to wishing I could easily consolidate the logs from an army of diverse honeypots to a central location for reporting. That's what initially drew me to MHN -- despite development being pretty much stalled for the past 2 years save for minor fixes here and there.

It's kind of a shame, really, the state of blight attained by the once-vibrant intel-sharing honeypot-loving ecosystem. There are a couple of active projects (telnetlogger and cowrie among my favorites) but hpfriends vanished, and most of the popular honeypot projects haven't seen but a fistful of commits in years. The STIX/TAXII ecosystem always seems like it's in a state of flux for anyone outside of FS-ISAC. All the while, there seems to be this undertone of something big coming with regards to open sharing of threat information among the incident response crowd.

For all their warts, some honeypots and analysis tools still work pretty well. I started putting together a presentation for SecKC shortly after beginning my tinkering with MHN, and those slides can be found here. This post is more of a walk-through of setting up and managing MHN and honeypots than the presentation was.

Setting up MHN

The instructions on the MHN github page work well. I installed MHN using a vanilla EC2 Ubuntu AMI on a t2.micro (free tier) instance without any trouble, but as we added more honeypots to it, a bump up to t2.small was needed to keep it from running out of memory on a daily basis, and I'm still not sure how well that will scale. A few members of the SecKC community have set up almost 50 honeypots* and pointed them at this instance of MHN. It's logging, at times, more than 100,000 hits per day, and the UI is getting kind of sluggish. MongoDB runs out of memory and requires a restart a few times per week, but I just use a cron script to check it every 5 minutes, and nudge the services if needed. This is a trade-off I'm willing to make. On-demand pricing for t2.small costs about $17 per month, and moving up to t2.medium or r3.large (where it probably belongs, due to MongoDB being a pig) sets me up for a monthly Amazon bill of $35-124, which I don't feel like shelling out of my own pocket.

*( with MHN, each honeypot software package counts as a "sensor," even if you're running two, three or more on the same instance - it's more like 30 distinct VPSes, Raspberry Pi and cloud compute nodes)

When you add users to the web UI, they are all admins. Admins can disable other user accounts. This is kind of a pain. I recommend installing the sqlite3 package on your MHN instance if you end up adding more users. I'll cover MHN databases toward the end. Additionally, I think all users can edit deploy scripts and delete honeypots, even if they're not admins. Be careful who you give access to the MHN console.

I'd recommend disabling the feed of data back to ThreatStream/Anomali (as found in the MHN setup page) by running "sudo /opt/mhn/scripts/disable_collector.sh" after you get it installed. In fact, I also removed /etc/supervisor/conf.d/mhn-collector.conf entirely, then ran "sudo supervisorctl reload" to update the configuration.

Setting up TLS with Let's Encrypt

Since there's a login and password on MHN, it's probably best to set up HTTPS. Since the MHN server is Ubuntu 16.04 with nginx, I followed this DigitalOcean guide, though I left the firewall stuff alone, since we're relying on Amazon's AWS security groups for traffic control. Just make sure both :80 and :443 are open to everywhere. I added the static locations for the acme challenge, and a redirect to https at the beginning of the "Default" sites-enabled nginx config file, /etc/nginx/sites-enabled/default like so:

server {
    listen       80;
    server_name  mhn.h-i-r.net;

    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/www/letsencrypt;
    location / {
        return 301 https://mhn.h-i-r.net$request_uri;

Make sure you add the appropriate crontab entry for certbot, and include the renew-hook script as in the DigitalOcean howto guide. Mine looks like this:

20 3 * * * certbot renew --noninteractive --renew-hook /root/letsencrypt.sh

Mnemosyne WebAPI

I'd mentioned the Mnemosyne WebAPI earlier. It requires TLS in order to work properly, and runs on port 8181. If you don't have certs, it won't even try to start. You don't really need to worry about this unless you want to run some custom reports against Mnemosyne outside of MongoDB, such as with some Python scripts.You should probably use your Lets Encrypt certs for this. Symlinks will keep these duplicates up-to-date as certbot rotates your certificates.

sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/privkey.pem /opt/mnemosyne/server.key
sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/cert.pem /opt/mnemosyne/server.crt

If you're using this feature from outside of the Amazon VPC MHN resides in, make sure you adjust your Amazon security groups to allow access to this service, but I would not recommend you expose this port publicly. There is some good API documentation here.

Deploy scripts

Deploy scripts are an easy way to take a bare-bones OS (like a fresh Raspbian Jessie Lite installation, or an Ubuntu VM) and add a Honeypot to it. When you select a deploy script, you see a deploy command (usually in the form of wget ... -O deploy.sh ; sudo bash deploy.sh ...) that should perform all of the actions needed to install the honeypot in question onto the operating system noted. Most are Ubuntu but there are a few specific to Raspberry Pi and CentOS. You can edit the scripts, the names and the notes for each of these to make local tweaks. MHN ships with a number of useful honeypot deploy scripts. Note that I've adjusted and re-named a few in this screen shot, so they won't perfectly match a fresh MHN install:

This isn't what drew me to MHN, but it's a feature I've come to love, and being written in shell (and sometimes a bit of Python), it's stuff that I'm actually comfortable editing. Since some honeypot projects went dormant before MHN, some of the deploy scripts still work fine. A few just need tweaks for minor changes in the host OS (e.g. Ubuntu 16.04LTS switched to the systemd init). In cases like Cowrie, though, the honeypot has evolved far beyond what it looked like when the deploy scripts were created. 

I've been adjusting a few of these and submitting pull requests, but you could also use my MHN fork for the time being, or simply pick and choose some of the deploy scripts out of my fork, and manually update your MHN instances that way.

Here's a quick animation of a Conpot (SCADA honeypot) deployment on EC2. It ran in just a few minutes. Most of the scripts take a little longer on a Raspberry Pi -- particularly the old 600MHz single-core Model B.


HPFeeds is the magic that ties the honeynet together. It's a lightweight, authenticated publish/subscribe protocol that runs on port 10000 by default. On the back-end, it's using MHN's fork of a framework called mnemosyne, which aims to normalize and store honeypot data while providing a web API for reporting. I'll cover the database part shortly. Each deploy script calls a registration routine (registration.sh) which gets a UUID and a secret key from MHN. This UUID is the honeypot's unique identifier, and the secret key is the authentication. Each honeypot publishes attack details to the hpfeeds broker that MHN installs. Each honeypot program has at least one channel it can publish to. For example, all conpot instances will publish to the "conpot.events" channel. Some honeypots have multiple channels, like one for events and one for captured malware. As the honeypot is installed, deploy.sh plugs these variables in to the configuration so the honeypot knows where to publish the details to.  Here's an example of the clause it added to conpot.cfg when I deployed it to a Pi in my lab:

enabled = True
host = mhn.h-i-r.net
port = 10000
ident = cb53aad4-7f08-1337-beef-0ad36352028b
secret = eteKJ6frn9bwQ1Hs
channels = ["conpot.events", ]

Just as a honeypot can publish to a channel, it's possible to set up a subscription to these channels. If you connect to HPFeeds with a tool like hpfeed-client from the python hpfeeds package, you can watch in near-real-time as attacks happen. Assuming you have network connectivity, a valid identifier, secret key and permission to subscribe to a channel, you can run the client from anywhere, and it'll pull the feed that MHN is getting. In the below example, I'm only querying dionaea.connections and cowrie.sessions. No dionaea connections showed up in the few seconds I had the client running, however.

To add a feed client user, I had to copy the "add_user.py" script from the hpfeeds git repository to /opt/mhn/env/bin on the MHN server, then run it inside the virtualenv. The syntax is [identifier] [secret] [publish feeds] [subscribe feeds] and since we don't need to publish with this tool, just subscribe, leave the publish empty (in quotes) and a list of subscriptions for your new account in quotes, comma separated like so:

$ cd /opt/mhn
$ source env/bin/activate
$ python env/bin/add_user.py test s3cr3t "" "wordpot.events,amun.events,cowrie.sessions,shockpot.events,dionaea.connections,snort.alerts"

Add as many feeds to the subscription as you like. These can be found in the hpfeeds fields of your deployed honeypots. This is the list of channels that the internal mnemosyne feed is subscribed to, which is likely all the feeds generated by all the honeypots MHN knows about:
  • amun.events
  • beeswarm.hive
  • beeswarn.feeder
  • conpot.events
  • cowrie.sessions
  • cuckoo.analysis
  • dionaea.capture
  • dionaea.connections
  • elastichoney.events
  • glastopf.events
  • glastopf.files
  • kippo.sessions
  • mwbinary.dionaea.sensorunique
  • p0f.events
  • shockpot.events
  • snort.alerts
  • suricata.events
  • thug.events
  • thug.files
  • wordpot.events
To access the hpfeeds stream, you can use an HPFeeds client library, or the reference client, written in python. An example command to pull the channels we set up earlier:

hpfeeds-client -i test -s s3cr3t -c wordpot.events -c amun.events -c cowrie.sessions -c shockpot.events -c dionaea.connections -c snort.alerts subscribe

The output should look like this, as sessions start rolling in (it can take a while if your honeypots sit idle a lot. 
[feedcli] connected to @hp2
[feedcli] publish to cowrie.sessions by 4fe5c378-7ec7-11e7-b58e-abc36352928a: {"peerIP": "", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60332, "session": "abb74fda35ef", "startTime": "2017-09-14T12:13:30.060296Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:30.524580Z", "unknownCommands": []}
[feedcli] publish to cowrie.sessions by f9b8c9d8-7ede-11e7-b58e-abc36352928a: {"peerIP": "", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60349, "session": "23ee29fca4f0", "startTime": "2017-09-14T12:13:32.161036Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:32.857748Z", "unknownCommands": []}

You can do quite a bit with just the HPFeeds data stream, for instance, if you funnel it into a database with some shell scripts, or run some post-processing on the HPFeeds log with awk.

Choosing a honeypot package

I have done a lot of testing over the past 2 months. Some of the deploy scripts needed some work (see above) but a lot of packages still run fine. Here are the ones I recommend. Note that Cowrie and Kippo deployments will move your real SSH server to port 2222, so be prepared for that.

For Raspberry Pi, installing the most recent image of Raspbian Jessie Lite from an image is probably your best bet, though the full Raspbian/PIXEL installation, from an image or from NOOBS also works fine on the Pi2 and Pi3. Most of my testing was done with the old-school Pi Model B and low-capacity SD cards (2 and 4GB). On the Pi, Cowrie, Dionaea and ShockPot are my favorites. WordPot also runs well, but you need to choose between it and ShockPot if you want something versatile on HTTP, and WordPot uses more resources. ConPot installs and runs properly, but it seemingly reports every single HTTP connection as an attack, which is noisy and counter-intuitive in my opinion.  Kippo works, but cowrie does it better. Amun works, but it's got a lot of overlap with Dionaea, which seems to provide more details about attacks in the logs.

On Ubuntu 16.04 LTS, my picks are Cowrie, Amun and WordPot. If you have a lot of RAM to spare, add Snort to the instance as well, because it will report all kinds of things that trigger an IDS signature from the EmergingThreats rule repository. Snort takes some manual configuration changes after install. ShockPot works, but as with the Pi, you can really only run one HTTP honeypot. ConPot works, but it's noisy. Kippo works, but Cowrie is better. Dionaea is broken for 16.04, hence my recommendation of Amun. I've spun up a bunch of Ubuntu honeypots with Cowrie, Amun, WordPot and Snort all running on the same instance in EC2 and Google Compute Engine.

Confirmed broken on Ubuntu 16.04: Suricata and Glastopf. I haven't invested much time in fixing these, but Glastopf seems to rely on ancient version of PHP that's (thankfully) missing from the package repository, and I'm not 100% sure what Suricata's deal is yet, but it looks to be something that can be fixed in the config file without much of a problem.

The SecKC crowd had questions about the safety of deploying honeypots. For the most part, the ones I discuss above are unlikely to lead to full shell access, but you can't be too sure. I had one attacker running tons of curl/wget commands from a Kippo honeypot last year, fetching ads, and essentially using my honeypot as a click fraud drone. Here was my advice from the presentation at SecKC:


I've found the MHN web front-end to be a bit lacking aside from the most basic of reports, and as I'd mentioned earlier, there's a problem when you add users through the web UI.

There are 2 database servers used by MHN, sqlite and mongodb. The majority of the MHN metadata is in sqlite. Use the ".tables" and ".schema [tablename]" commands to explore MHN's sqlite database, if you get curious. The tables user, roles_users are the most useful. All of the deploy scripts are stored in sqlite as well. They're imported from the shell scripts in the git repository upon installation and the scripts in /opt/mhn/scripts/deploy_* are never referenced again.

If you want to revoke admin privileges from all but the user account you set up when installing MHN, I recommend doing this:

$ sudo sqlite3 /opt/mhn/server/mhn.db
sqlite3>  update roles_users set role_id=2 where user_id > 1;
sqlite3> .exit

Most of the attack data, reporting and statistics are held in mongodb. You can run the "mongo" command line tool and start running queries directly. Mongo has "collections" instead of tables, and an interesting query syntax. There are a few databases, but the most useful are hpfeeds, where the subscription information is held, and the grand-daddy of them all, mnemosyne, which stores the normalized honeypot data. In reality, the Mnemosyne WebAPI might be easier for some to use, but I'm a fan of getting messy in the database. By default, the Mongo shell only displays 20 records at a time (type "it" to  continue) but I found out you can alter the default maximum results by creating a file called ".mongorc.js" in your home directory and adding this line to it:

DBQuery.shellBatchSize = 100

Here are a couple of useful example queries I've come up with.

See the most recent attacks:
> use mnemosyne
> db.session.find().sort( { timestamp : -1 } );

Find records for IP addresses that start with "192.168"
> use mnemosyne
> db.session.find({source_ip : /^192\.168.*/ })

Delete all records of a specific IP address (e.g. yours) from the records:
> use mnemosyne
> db.session.remove({source_ip : ""})

Delete all records older than 7 days (in the event Mongo is getting sluggish. Change 7 * 24 to n * 24 for the number of days you want to save):
> use mnemosyne
> db.session.remove( { timestamp: { $lt: (new Date((new Date()).getTime() - ( 7 * 24 * 60 * 60 * 1000 ) ) ) } } )

Get a list of IP addresses based on the number of honeypots they've connected to in the last day (I've limited this to SSH)
> use mnemosyne > db.session.aggregate([
    { $match:
        { timestamp: { $gte: (new Date((new Date()).getTime() - ( 1 * 24 * 60 * 60 * 1000 ) ) ) },
        protocol: "ssh" }
    { $group: { _id: "$source_ip", honeypots: { $addToSet: "$identifier"    } } },
    { $unwind: "$honeypots" },
    { $group: { _id: "$_id", honeypotcount: { $sum: 1 } } },
    { $sort : { honeypotcount : -1} },
    { $limit : 50 }

Fun stuff

Brandon from SecKC and I collaborated on this fun dashboard, based on Rob Scanlon's Tron Legacy Encom Boardroom visualization. Brandon did all the code tweaks and wrote the middleware to get data into it. I just played project manager once the MHN firehose was ready. It really is a sight to behold. The "satellites" are approximate geo-locations for the deployed honeypots. Push-pins on the globe are attackers. The center pane is a live feed of attacks, with usernames and passwords where applicable. The right pane is a top attacker list and a map (if you click on the attacker) with more details. The chart at the bottom is the last 24 hours of attacks. Some of the panels won't display data unless you're logged in to avoid sharing the honeypot IPs publicly -- we're working on tweaking the back-end so that it can work well, safely, without authentication.