2017-09-13

Building a honeypot army with Pi, EC2 and MHN

I've been thinking about honeypots a lot over the last year. I have a bunch of Raspberry Pi, Atheros-powered boards, old Linksys routers and other devices kicking around my home office that would make good honeypots, and my thoughts always wander back to wishing I could easily consolidate the logs from an army of diverse honeypots to a central location for reporting. That's what initially drew me to MHN -- despite development being pretty much stalled for the past 2 years save for minor fixes here and there.

It's kind of a shame, really, the state of blight attained by the once-vibrant intel-sharing honeypot-loving ecosystem. There are a couple of active projects (telnetlogger and cowrie among my favorites) but hpfriends vanished, and most of the popular honeypot projects haven't seen but a fistful of commits in years. The STIX/TAXII ecosystem always seems like it's in a state of flux for anyone outside of FS-ISAC. All the while, there seems to be this undertone of something big coming with regards to open sharing of threat information among the incident response crowd.

For all their warts, some honeypots and analysis tools still work pretty well. I started putting together a presentation for SecKC shortly after beginning my tinkering with MHN, and those slides can be found here. This post is more of a walk-through of setting up and managing MHN and honeypots than the presentation was.

Setting up MHN

The instructions on the MHN github page work well. I installed MHN using a vanilla EC2 Ubuntu AMI on a t2.micro (free tier) instance without any trouble, but as we added more honeypots to it, a bump up to t2.small was needed to keep it from running out of memory on a daily basis, and I'm still not sure how well that will scale. A few members of the SecKC community have set up almost 50 honeypots* and pointed them at this instance of MHN. It's logging, at times, more than 100,000 hits per day, and the UI is getting kind of sluggish. MongoDB runs out of memory and requires a restart a few times per week, but I just use a cron script to check it every 5 minutes, and nudge the services if needed. This is a trade-off I'm willing to make. On-demand pricing for t2.small costs about $17 per month, and moving up to t2.medium or r3.large (where it probably belongs, due to MongoDB being a pig) sets me up for a monthly Amazon bill of $35-124, which I don't feel like shelling out of my own pocket.

*( with MHN, each honeypot software package counts as a "sensor," even if you're running two, three or more on the same instance - it's more like 30 distinct VPSes, Raspberry Pi and cloud compute nodes)

When you add users to the web UI, they are all admins. Admins can disable other user accounts. This is kind of a pain. I recommend installing the sqlite3 package on your MHN instance if you end up adding more users. I'll cover MHN databases toward the end. Additionally, I think all users can edit deploy scripts and delete honeypots, even if they're not admins. Be careful who you give access to the MHN console.

I'd recommend disabling the feed of data back to ThreatStream/Anomali (as found in the MHN setup page) by running "sudo /opt/mhn/scripts/disable_collector.sh" after you get it installed. In fact, I also removed /etc/supervisor/conf.d/mhn-collector.conf entirely, then ran "sudo supervisorctl reload" to update the configuration.

Setting up TLS with Let's Encrypt

Since there's a login and password on MHN, it's probably best to set up HTTPS. Since the MHN server is Ubuntu 16.04 with nginx, I followed this DigitalOcean guide, though I left the firewall stuff alone, since we're relying on Amazon's AWS security groups for traffic control. Just make sure both :80 and :443 are open to everywhere. I added the static locations for the acme challenge, and a redirect to https at the beginning of the "Default" sites-enabled nginx config file, /etc/nginx/sites-enabled/default like so:

server {
    listen       80;
    server_name  mhn.h-i-r.net;

    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/www/letsencrypt;
    }
    location / {
        return 301 https://mhn.h-i-r.net$request_uri;
    }
}


Make sure you add the appropriate crontab entry for certbot, and include the renew-hook script as in the DigitalOcean howto guide. Mine looks like this:

20 3 * * * certbot renew --noninteractive --renew-hook /root/letsencrypt.sh

Mnemosyne WebAPI

I'd mentioned the Mnemosyne WebAPI earlier. It requires TLS in order to work properly, and runs on port 8181. If you don't have certs, it won't even try to start. You don't really need to worry about this unless you want to run some custom reports against Mnemosyne outside of MongoDB, such as with some Python scripts.You should probably use your Lets Encrypt certs for this. Symlinks will keep these duplicates up-to-date as certbot rotates your certificates.

sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/privkey.pem /opt/mnemosyne/server.key
sudo ln -s /etc/letsencrypt/live/mhn.h-i-r.net/cert.pem /opt/mnemosyne/server.crt

If you're using this feature from outside of the Amazon VPC MHN resides in, make sure you adjust your Amazon security groups to allow access to this service, but I would not recommend you expose this port publicly. There is some good API documentation here.

Deploy scripts

Deploy scripts are an easy way to take a bare-bones OS (like a fresh Raspbian Jessie Lite installation, or an Ubuntu VM) and add a Honeypot to it. When you select a deploy script, you see a deploy command (usually in the form of wget ... -O deploy.sh ; sudo bash deploy.sh ...) that should perform all of the actions needed to install the honeypot in question onto the operating system noted. Most are Ubuntu but there are a few specific to Raspberry Pi and CentOS. You can edit the scripts, the names and the notes for each of these to make local tweaks. MHN ships with a number of useful honeypot deploy scripts. Note that I've adjusted and re-named a few in this screen shot, so they won't perfectly match a fresh MHN install:


This isn't what drew me to MHN, but it's a feature I've come to love, and being written in shell (and sometimes a bit of Python), it's stuff that I'm actually comfortable editing. Since some honeypot projects went dormant before MHN, some of the deploy scripts still work fine. A few just need tweaks for minor changes in the host OS (e.g. Ubuntu 16.04LTS switched to the systemd init). In cases like Cowrie, though, the honeypot has evolved far beyond what it looked like when the deploy scripts were created. 

I've been adjusting a few of these and submitting pull requests, but you could also use my MHN fork for the time being, or simply pick and choose some of the deploy scripts out of my fork, and manually update your MHN instances that way.

Here's a quick animation of a Conpot (SCADA honeypot) deployment on EC2. It ran in just a few minutes. Most of the scripts take a little longer on a Raspberry Pi -- particularly the old 600MHz single-core Model B.

HPFeeds

HPFeeds is the magic that ties the honeynet together. It's a lightweight, authenticated publish/subscribe protocol that runs on port 10000 by default. On the back-end, it's using MHN's fork of a framework called mnemosyne, which aims to normalize and store honeypot data while providing a web API for reporting. I'll cover the database part shortly. Each deploy script calls a registration routine (registration.sh) which gets a UUID and a secret key from MHN. This UUID is the honeypot's unique identifier, and the secret key is the authentication. Each honeypot publishes attack details to the hpfeeds broker that MHN installs. Each honeypot program has at least one channel it can publish to. For example, all conpot instances will publish to the "conpot.events" channel. Some honeypots have multiple channels, like one for events and one for captured malware. As the honeypot is installed, deploy.sh plugs these variables in to the configuration so the honeypot knows where to publish the details to.  Here's an example of the clause it added to conpot.cfg when I deployed it to a Pi in my lab:

[hpfriends]
enabled = True
host = mhn.h-i-r.net
port = 10000
ident = cb53aad4-7f08-1337-beef-0ad36352028b
secret = eteKJ6frn9bwQ1Hs
channels = ["conpot.events", ]


Just as a honeypot can publish to a channel, it's possible to set up a subscription to these channels. If you connect to HPFeeds with a tool like hpfeed-client from the python hpfeeds package, you can watch in near-real-time as attacks happen. Assuming you have network connectivity, a valid identifier, secret key and permission to subscribe to a channel, you can run the client from anywhere, and it'll pull the feed that MHN is getting. In the below example, I'm only querying dionaea.connections and cowrie.sessions. No dionaea connections showed up in the few seconds I had the client running, however.



To add a feed client user, I had to copy the "add_user.py" script from the hpfeeds git repository to /opt/mhn/env/bin on the MHN server, then run it inside the virtualenv. The syntax is [identifier] [secret] [publish feeds] [subscribe feeds] and since we don't need to publish with this tool, just subscribe, leave the publish empty (in quotes) and a list of subscriptions for your new account in quotes, comma separated like so:

$ cd /opt/mhn
$ source env/bin/activate
$ python env/bin/add_user.py test s3cr3t "" "wordpot.events,amun.events,cowrie.sessions,shockpot.events,dionaea.connections,snort.alerts"

Add as many feeds to the subscription as you like. These can be found in the hpfeeds fields of your deployed honeypots. This is the list of channels that the internal mnemosyne feed is subscribed to, which is likely all the feeds generated by all the honeypots MHN knows about:
  • amun.events
  • beeswarm.hive
  • beeswarn.feeder
  • conpot.events
  • cowrie.sessions
  • cuckoo.analysis
  • dionaea.capture
  • dionaea.connections
  • elastichoney.events
  • glastopf.events
  • glastopf.files
  • kippo.sessions
  • mwbinary.dionaea.sensorunique
  • p0f.events
  • shockpot.events
  • snort.alerts
  • suricata.events
  • thug.events
  • thug.files
  • wordpot.events
To access the hpfeeds stream, you can use an HPFeeds client library, or the reference client, written in python. An example command to pull the channels we set up earlier:

hpfeeds-client -i test -s s3cr3t -c wordpot.events -c amun.events -c cowrie.sessions -c shockpot.events -c dionaea.connections -c snort.alerts subscribe

The output should look like this, as sessions start rolling in (it can take a while if your honeypots sit idle a lot. 
[feedcli] connected to @hp2
[feedcli] publish to cowrie.sessions by 4fe5c378-7ec7-11e7-b58e-abc36352928a: {"peerIP": "171.243.14.183", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60332, "session": "abb74fda35ef", "startTime": "2017-09-14T12:13:30.060296Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:30.524580Z", "unknownCommands": []}
[feedcli] publish to cowrie.sessions by f9b8c9d8-7ede-11e7-b58e-abc36352928a: {"peerIP": "171.243.14.183", "commands": [], "loggedin": null, "version": "SSH-2.0-Granados-1.0", "ttylog": null, "urls": [], "hostIP": "redacted", "peerPort": 60349, "session": "23ee29fca4f0", "startTime": "2017-09-14T12:13:32.161036Z", "hostPort": 22, "credentials": [], "endTime": "2017-09-14T12:13:32.857748Z", "unknownCommands": []}


You can do quite a bit with just the HPFeeds data stream, for instance, if you funnel it into a database with some shell scripts, or run some post-processing on the HPFeeds log with awk.

Choosing a honeypot package

I have done a lot of testing over the past 2 months. Some of the deploy scripts needed some work (see above) but a lot of packages still run fine. Here are the ones I recommend. Note that Cowrie and Kippo deployments will move your real SSH server to port 2222, so be prepared for that.

For Raspberry Pi, installing the most recent image of Raspbian Jessie Lite from an image is probably your best bet, though the full Raspbian/PIXEL installation, from an image or from NOOBS also works fine on the Pi2 and Pi3. Most of my testing was done with the old-school Pi Model B and low-capacity SD cards (2 and 4GB). On the Pi, Cowrie, Dionaea and ShockPot are my favorites. WordPot also runs well, but you need to choose between it and ShockPot if you want something versatile on HTTP, and WordPot uses more resources. ConPot installs and runs properly, but it seemingly reports every single HTTP connection as an attack, which is noisy and counter-intuitive in my opinion.  Kippo works, but cowrie does it better. Amun works, but it's got a lot of overlap with Dionaea, which seems to provide more details about attacks in the logs.

On Ubuntu 16.04 LTS, my picks are Cowrie, Amun and WordPot. If you have a lot of RAM to spare, add Snort to the instance as well, because it will report all kinds of things that trigger an IDS signature from the EmergingThreats rule repository. Snort takes some manual configuration changes after install. ShockPot works, but as with the Pi, you can really only run one HTTP honeypot. ConPot works, but it's noisy. Kippo works, but Cowrie is better. Dionaea is broken for 16.04, hence my recommendation of Amun. I've spun up a bunch of Ubuntu honeypots with Cowrie, Amun, WordPot and Snort all running on the same instance in EC2 and Google Compute Engine.

Confirmed broken on Ubuntu 16.04: Suricata and Glastopf. I haven't invested much time in fixing these, but Glastopf seems to rely on ancient version of PHP that's (thankfully) missing from the package repository, and I'm not 100% sure what Suricata's deal is yet, but it looks to be something that can be fixed in the config file without much of a problem.

The SecKC crowd had questions about the safety of deploying honeypots. For the most part, the ones I discuss above are unlikely to lead to full shell access, but you can't be too sure. I had one attacker running tons of curl/wget commands from a Kippo honeypot last year, fetching ads, and essentially using my honeypot as a click fraud drone. Here was my advice from the presentation at SecKC:



Databases

I've found the MHN web front-end to be a bit lacking aside from the most basic of reports, and as I'd mentioned earlier, there's a problem when you add users through the web UI.

There are 2 database servers used by MHN, sqlite and mongodb. The majority of the MHN metadata is in sqlite. Use the ".tables" and ".schema [tablename]" commands to explore MHN's sqlite database, if you get curious. The tables user, roles_users are the most useful. All of the deploy scripts are stored in sqlite as well. They're imported from the shell scripts in the git repository upon installation and the scripts in /opt/mhn/scripts/deploy_* are never referenced again.

If you want to revoke admin privileges from all but the user account you set up when installing MHN, I recommend doing this:

$ sudo sqlite3 /opt/mhn/server/mhn.db
sqlite3>  update roles_users set role_id=2 where user_id > 1;
sqlite3> .exit

Most of the attack data, reporting and statistics are held in mongodb. You can run the "mongo" command line tool and start running queries directly. Mongo has "collections" instead of tables, and an interesting query syntax. There are a few databases, but the most useful are hpfeeds, where the subscription information is held, and the grand-daddy of them all, mnemosyne, which stores the normalized honeypot data. In reality, the Mnemosyne WebAPI might be easier for some to use, but I'm a fan of getting messy in the database. By default, the Mongo shell only displays 20 records at a time (type "it" to  continue) but I found out you can alter the default maximum results by creating a file called ".mongorc.js" in your home directory and adding this line to it:

DBQuery.shellBatchSize = 100

Here are a couple of useful example queries I've come up with.

See the most recent attacks:
> use mnemosyne
> db.session.find().sort( { timestamp : -1 } );

Find records for IP addresses that start with "192.168"
> use mnemosyne
> db.session.find({source_ip : /^192\.168.*/ })

Delete all records of a specific IP address (e.g. yours) from the records:
> use mnemosyne
> db.session.remove({source_ip : "192.168.1.94"})

Delete all records older than 7 days (in the event Mongo is getting sluggish. Change 7 * 24 to n * 24 for the number of days you want to save):
> use mnemosyne
> db.session.remove( { timestamp: { $lt: (new Date((new Date()).getTime() - ( 7 * 24 * 60 * 60 * 1000 ) ) ) } } )

Get a list of IP addresses based on the number of honeypots they've connected to in the last day (I've limited this to SSH)
> use mnemosyne > db.session.aggregate([
    { $match:
        { timestamp: { $gte: (new Date((new Date()).getTime() - ( 1 * 24 * 60 * 60 * 1000 ) ) ) },
        protocol: "ssh" }
    },
    { $group: { _id: "$source_ip", honeypots: { $addToSet: "$identifier"    } } },
    { $unwind: "$honeypots" },
    { $group: { _id: "$_id", honeypotcount: { $sum: 1 } } },
    { $sort : { honeypotcount : -1} },
    { $limit : 50 }
]);

Fun stuff

Brandon from SecKC and I collaborated on this fun dashboard, based on Rob Scanlon's Tron Legacy Encom Boardroom visualization. Brandon did all the code tweaks and wrote the middleware to get data into it. I just played project manager once the MHN firehose was ready. It really is a sight to behold. The "satellites" are approximate geo-locations for the deployed honeypots. Push-pins on the globe are attackers. The center pane is a live feed of attacks, with usernames and passwords where applicable. The right pane is a top attacker list and a map (if you click on the attacker) with more details. The chart at the bottom is the last 24 hours of attacks. Some of the panels won't display data unless you're logged in to avoid sharing the honeypot IPs publicly -- we're working on tweaking the back-end so that it can work well, safely, without authentication.

2017-08-21

That one time the solar eclipse broke my phone

Okay, I broke my phone, but it's the moon's fault. I live about 30 miles south of the path of the 2017 solar eclipse's totality. I've been preparing for it for some time, planning to make a short drive to see it. Today was the big day.

Back in March, I started playing with building my own solar filters for my telescope, a Meade ETX-60AT.  My grandpa was a huge optics nerd, and I learned a lot from him. One trick he mentioned, but one I never really saw in action, was using stacked layers of aluminized mylar as a filter medium. I ended up going with 3 layers from a mylar emergency blanket. I cut the upper 1.5" of an old plastic cup, which happened to fit nicely over my telescope lens, and mounted the mylar to that.


I've had good luck with taking photos with my phone through the eyepiece of this scope in the past, but stabilizing the phone is kind of dicey, especially when you have to point the scope nearly straight up to catch the mid-day sun-and-moon dance of a total solar eclipse. They make jigs to hold phones to telescope eyepieces, but I'm a cheap bastard. This isn't sketchy at all.


Initial tests over the pre-eclipse weekend showed that it works well. I was concerned that the mylar was possibly passing dangerous IR spectrum, so I left it tracking the sun for 20 minutes with the mirror open (allowing the light to hit the back-end of the eyepiece housing without destroying anything terribly important), and it wasn't heating up. I engaged the mirror, and repeated the test, and the mirror itself was cooler than the outside of the scope. I used one of my old, un-unsed Android phones to capture the full disc of the noon-time sun. The aluminum of the mylar gives the sun a bluish-grey color. Check out those sunspots!


Today, my wife and I made the trek out to Excelsior Springs, MO, opting to get as far east as practical to avoid the clouds plaguing Kansas City. Long story short, my sketchy tape contraption didn't fare so well. My phone hit the pavement part-way through the eclipse festivities. Again, this is the moon's fault, not the fact that I used a single strip of masking tape. Honest.


Anyhow, here are some of the goods. Some whispy clouds flared up right at totality, which gave the Sun's corona an even eerier, dancing shimmer. Not bad for a phone with a cracked screen, but I've seen far better photos from some of my professional photographer friends.





After we got back home, I spent about 15 minutes swapping displays. Yeah, I had a box of Droid MAXX parts laying around.


All better now.

2017-05-18

Logical Domains on SunFire T2000 with OpenBSD/sparc64

I've been on a bit of a virtualization kick. This is about OpenBSD, but not vmm.

A couple of years ago, I picked up a Sun Fire T2000. This is a 2U rack mount server. Mine came with four 146GB SAS drives, a 32-core UltraSPARC T1 CPU and 32GB of RAM. Standard hardware includes redundant power supplies, network and serial Advanced Lights Out Management (ALOM), four gigabit ethernet interfaces and a number of USB ports. A number of PCIe and PCI-X slots are available for adding things like RAID controllers, additional network adapters or what-have-you. This system is a decade old, give or take a year or two.


Sun Microsystems incorporated Logical Domains (LDOMs) on this class of hardware. You don't often need 32 threads and 32GB of RAM in a single server. LDOMs are a kind of virtualization technology that's a bit closer to bare metal than vmm, Hyper-V, VirtualBox or even Xen. It works a bit like Xen, though. You can allocate processor, memory, storage and other resources to virtual servers on-board, with a blend of firmware that supports the hardware allocation, and some software in userland (on the so-called primary or control domain, similar to Xen DomU) to control it.

LDOMs are similar to what IBM calls Logical Partitions (LPARs) on its Mainframe and POWER series computers. My day job from 2006-2010 involved working with both of these virtualization technologies, and I've kind of missed it.

While upgrading OpenBSD to 6.1 on my T2000, I decided to delve into LDOM support under OpenBSD. This was pretty easy to do, but let's walk through it. Resources I used:
Once you get comfortable with the fact that there's a little-tiny computer (the ALOM) powered by VXWorks inside that's acting as the management system and console (there's no screen or keyboard/mouse input), Installing OpenBSD on the base server is pretty straightforward. The serial console is an RJ-45 jack, and, yes, the ubiquitous blue-colored serial console cables you find for certain kinds of popular routers will work fine. The networked part of ALOM, if enabled, will probably get a DHCP address and listen for an SSH connection. "admin/changeme" is the default. Resetting it isn't too hard, if you need to. The Internet can help you if you know how to search.

If you have seven minutes to spare to watch the ALOM and the T2000 itself boot up, watch the below video. Put on your earplugs (not your headphones), because this sucker is *loud*. Otherwise, just skip past it.



If there isn't an operating system on the box, the "ok" prompt should show up in the console eventually. If there is an operating system, you can use the "break" command to get to "ok". From there, cd booting is as easy as:
setenv boot-device cdrom
boot
OpenBSD installs quite easily, with the same installer you find on amd64 and i386. I chose to install to /dev/sd0, the first SAS drive only, leaving the others unused. It's possible to set them up in a hardware RAID configuration using tools available only under Solaris, or use softraid(4) on OpenBSD, but I didn't do this.

Enable ldomd:
doas rcctl enable ldomd

I set up the primary LDOM to use the first ethernet port, em0. I decided I wanted to bridge the logical domains to the second ethernet port. You could also use a bridge and vether interface, with pf and dhcpd to create a NAT environment, similar to how I networked the vmm(4) systems.

Cable the second ethernet port to the desired network segment. I'm assuming you got the first ethernet port configured by now. This second port doesn't need its own IP address from the primary LDOM. Configure it by putting this in /etc/hostname.em1:
-inet6
up
While you're at it, put those same two lines in /etc/hostname.vnet0. Create additional hostname.vnetN files for as many LDOMs as you plan to set up.

Create a bridge interface configuration with this chunk of text in /etc/hostname.bridge0. Add the rest of your vnet interfaces if you made more than one.
-inet6
up
add em1
add vnet0
Fetch the sparc64 minirootXX.fs file from an OpenBSD mirror. Make a blank 4GB disk image for the VM, then write the miniroot to the beginning of it. This will be the easiest way to boot to the OpenBSD installer inside the VM console.

mkdir ~/vm
dd if=/dev/zero of=~/vm/ldom1 bs=1m count=4000 
dd if=miniroot61.fs of=~/vm/ldom1 bs=64k conv=notrunc
 
Create an LDOM configuration file. You can put this anywhere that's convenient. All of this stuff was in a "vm" subdirectory of my home. I called it ldom.conf:
domain primary {
    vcpu 8
    memory 8G
}
domain puffy {
    vcpu 8
    memory 4G
    vdisk "/home/axon/vm/ldom1"
    vnet

Make as many disk images as you want, and make as many additional domain clauses as you wish. Be mindful of system resources. I couldn't actually allocate a full 32GB of RAM across all the LDOMs. I ended up allocating 31.

We're going to dump the factory default LDOM configuration (that is, everything assigned to the primary LDOM, with no actual VMs). We need to copy this as a template, so that we can modify it and send it back to the firmware, telling it to re-allocate hardware assets to an LDOM.

# dump the factory
mkdir ~/vm/default 
cd ~/vm/default 
doas ldomctl dump 

# copy the configuration to the "myldom" directory
cd .. 
cp -r default myldom
cd myldom

# Modify the configuration using the ldom.conf we created
doas ldomctl init-system ~/vm/ldom.conf

# Push ("download") the new "myldom" configuration to the system controller.
cd ..
doas ldomctl download myldom

At this point, you can list the configurations and see that your new one will boot next.

$ doas ldomctl list
factory-default [current]
myldom [next]

Halt the system.
doas halt

Exit the console with "#." and at the system controller prompt, run "reset -c"


Once the system boots up and you log in, check out the running LDOMS.

$ doas ldomctl status
primary           running           OpenBSD running                     0%
ldom1              running           OpenBSD running                     0%

Connect to the console of ldom1 (ttyV0). Additional LDOMs will have consoles at ttyV1, ttyV2, etc. Voila. OpenBSD installer! Assuming your network configuration and cabling is right, it will show up as a machine directly on the LAN.

$ doas cu -l ttyV0
Connected to /dev/ttyV0 (speed 9600)

(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? I
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN.  You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.

System hostname? (short form, e.g. 'foo')
...

I eventually provisioned seven LDOMs (in addition to the primary) on the T2000, each with 3GB of RAM and 4 vcpu cores. If you get creative with use of network interfaces, virtual ethernet, bridges and pf rules, you can run a pretty complex environment on a single chassis, with services that are only exposed to other VMs, a DMZ segment, and the internal LAN.


If, like me, you end up with an unbootable LDOM configuration, you can always revert back to the factory default, or switch between stored configurations from the system controller. At the sc> prompt, use:

bootmode config="factory-default"
reset -c

Any configuration that you've downloaded to the controller can be used. They will show up by name in the "ldomctl list" command.

2017-04-29

OpenBSD vmm Hypervisor Part 2: Installation and Networking

* Updated November 1, 2018, Thanks to some commenters below! 
This is going to be a (likely long-running, infrequently-appended) series of posts as I poke around in vmm.  A few months ago, I demonstrated some basic use of the vmm hypervisor as it existed in OpenBSD 6.0-CURRENT around late October, 2016. We'll call that video Part 1.

As good as the OpenBSD documentation is (and vmm/vmd/vmctl are no exception) I had to do a lot of fumbling around, and asking on the misc@ mailing list to really get to the point where I understand this stuff as well as I do. These are my notes so far.

Quite a bit of development was done on vmm before 6.1-RELEASE, and it's worth noting that some new features made their way in. Work continues, of course, and I can only imagine the hypervisor technology will mature plenty for the next release. As it stands, this is the first release of OpenBSD with a native hypervisor shipped in the base install, and that's exciting news in and of itself. In this article, we assume a few things:

  1. You are using OpenBSD 6.4-RELEASE or -STABLE  (or newer)
  2. You're running the amd64 architecture version of OpenBSD
  3. The system has an Intel CPU with a microarchitecture from the Nehalem family, or newer. (e.g. Westmere, Sandy Bridge, Ivy Bridge). Basically, if you've got an i5 or i7, you should be good to go.
  4. Your user-level account can perform actions as root via doas. See the man page for doas.conf(5) for details.
 Notes on formatting:
  1. Lines in blue are text inside configuration files.
  2. Lines in green are shell commands to be executed on the command line.

Basic network setup

To get our virtual machines onto the network, we have to spend some time setting up a virtual ethernet interface. We'll run a DHCP server on that, and it'll be the default route for our virtual machines. We'll keep all the VMs on a private network segment (via a bridge interface), and use NAT to allow them to get to the network. There is a way to directly bridge VMs to the network in some situations, but I won't be covering that today. (Hint: If you bridge to an interface with dhclient running in the host OS, no DHCP packets will reach the VMs, so it's best to static-assign your VM host's LAN port. or use a different interface entirely.)

Enable IP forwarding in sysctl. Add the following to /etc/sysctl.conf:

net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

You can reboot to enable these, or run these commands manually:

doas sysctl net.inet.ip.forwarding=1
doas sysctl net.inet6.ip6.forwarding=1


We're going to be using "vether0" as the VM interface, and in my current configuration, "athn0" as the external interface. The below block of pf configuration was taken partially from /etc/example/pf.conf and some of the PF NAT example from the OpenBSD FAQ. Change your ext_if as needed, and save this as /etc/pf.conf:

ext_if="athn0"
vmd_if="vether0"

set skip on lo

block return
pass           
 

match out on $ext_if inet from $vmd_if:network to any nat-to ($ext_if)

Next, you'll need to specify an IP and subnet for vether0. I am going with 10.13.37.1 (and a 255.255.255.0 netmask for a /24 network) in my example. Place something like this in /etc/hostname.vether0:

inet 10.13.37.1 255.255.255.0

We need to create a bridge interface for the guest VMs to attach to, and bridge vether0 to it. Place the following in /etc/hostname.bridge0:

add vether0

Bring vether0 and bridge0 online by running the following commands:

doas sh /etc/netstart vether0
doas sh /etc/netstart bridge0

Reload the pf configuration (h/t to voutilad for the heads up in the comments about moving this step down a few spots):

doas pfctl -f /etc/pf.conf

Create a basic DHCP server configuration file that matches the vether0 configuration. I've specified 10.13.37.32-127 as the default DHCP pool range, and set 10.13.37.1 as the default route. Save this in /etc/dhcpd.conf:

option  domain-name "vmm.openbsd.local";
option  domain-name-servers 8.8.8.8, 8.8.4.4;

subnet 10.13.37.0 netmask 255.255.255.0 {
        option routers 10.13.37.1;

        range 10.13.37.32 10.13.37.127;

}

Configure a switch for vmm, so the VMs have connectivity. Put the following in /etc/vm.conf:

switch "local" {
        interface bridge0
}


Enable and start the DHCP server. We also need to set the flags on dhcpd so that it only listens on vether0. Otherwise, you'll end up with a rogue DHCP server on your primary network. Your network administrators and other users on your network will not appreciate this because it could potentially keep others from getting on the network properly.

doas rcctl enable dhcpd
doas rcctl set dhcpd flags vether0
doas rcctl start dhcpd

Enable vmd, then start it as well. For good measure, re-run fw_update to make sure you have the proper vmm-bios package.

doas rcctl enable vmd
doas rcctl start vmd
doas fw_update

You should notice a new interface, likely named "bridge0" in ifconfig now.

VM Creation and installation

Create an empty disk image for your new VM. I'd recommend 1.5GB to play with at first. You can do this without doas or root if you want your user account to be able to start the VM later. I made a "vmm" directory inside my home directory to store VM disk images in. You might have a different partition you wish to store these large files in. Adjust as needed.

mkdir ~/vmm
vmctl create ~/vmm/test.img -s 1500M

Boot up a brand new vm instance. You'll have to do this as root or with doas. You can download a -CURRENT install kernel/ramdisk (bsd.rd) from an OpenBSD mirror, or you can simply use the one that's on your existing system (/bsd.rd) like I'll do here.

The below command will start a VM named "test.vm", display the console at startup, use /bsd.rd (from our host environment) as the boot image, allocate 256MB of memory, attach the first network interface to the switch called "local" we defined earlier in /etc/vm.conf, and use the test image we just created as the first disk drive. 

doas vmctl start "test.vm" -c -b /bsd.rd -m 512M -n "local" -d ~/vmm/test.img

You should see the boot message log followed by the OpenBSD installer. Enter "I" at the prompt to install OpenBSD. I won't walk through how to install OpenBSD. If you've gotten this far, you've done this at least once before. I can say that the vm console works by emulating a serial port, so you should leave these values at their defaults (just press enter):
Change the default console to com0? [yes]
Available speeds are: 9600 19200 38400 57600 115200.
Which speed should com0 use? (or 'done') [9600]
You will also likely need to install packages without a prefetch area. Answer yes to this.
Cannot determine prefetch area. Continue without verification? [no] yes
After installation and before rebooting, I'd recommend taking note of the vio0 interface's generated MAC address (lladdr) so you can specify a static lease in dhcpd later. If you don't specify one, it seems to randomize the last few bytes of the address.
CONGRATULATIONS! Your OpenBSD install has been successfully completed!
To boot the new system, enter 'reboot' at the command prompt.
When you login to your new system the first time, please read your mail
using the 'mail' command.

# ifconfig vio0                                                               
vio0: flags=8b43 mtu 1500
        lladdr fe:e1:bb:d1:23:c4
        llprio 3
        groups: dhcp egress
        media: Ethernet autoselect
        status: active
        inet 10.13.37.39 netmask 0xffffff00 broadcast 10.13.37.255
Shut down the vm with "halt -p" and then press enter a few extra times. Enter the characters "~." (tilde period) to exit the VM console. Show the VM status, and stop the VM. I've been starting and stopping VMs all morning, so your VM ID will probably be 1 instead of 13.
# halt -p

(I hit ~. here...)
[EOT]
$ vmctl status
   ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER NAME
   13 96542     1    256M   52.6M   ttyp3         root test.vm
$ doas vmctl stop 13
vmctl: terminated vm 13 successfully

VM Configuration

Now that the VM disk image file has a full installation of OpenBSD on it, build a VM configuration around it by adding the below block of configuration (with modifications as needed for owner, path and lladdr) to /etc/vm.conf.

# H-i-R.net Test VM
vm "test.vm" {
        disable
        owner axon
        memory 512M
        disk "/home/axon/vmm/test.img"
        interface {
                switch "local"
                lladdr fe:e1:bb:d1:23:c4
        }
}


I've noticed that VMs with much less than 256MB of RAM allocated tend to be a little unstable for me. You'll also note that in the "interface" clause, I hard-coded the lladdr that was generated for it earlier. By specifying "disable" in vm.conf, the VM will show up in a stopped state that the owner of the VM (that's you!) can manually start without root access.

Go ahead and reload the VM configuration with "doas vmctl reload", then look at the status again.
$ doas vmctl reload
$ vmctl status
   ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER NAME

    -     -     1    512M       -       -         axon test.vm

DHCP Reservation (Optional)

I opted to set up a DHCP reservation for my VM so that I had a single IP address I knew I could SSH to from my VM host. This, to me, seems easier than using the VM console for everything.

Add this clause (again, modified to match the MAC address you noted earlier) to /etc/dhcpd.conf. You have to place this above the last curly-brace.

        host static-client {
                hardware ethernet fe:e1:bb:d1:23:c4;
                fixed-address 10.13.37.203;
        }

Restart dhcpd.

doas rcctl restart dhcpd

Go ahead and fire up the VM, and attach to the console.

vmctl start test.vm -c

You'll see the seabios messages, the boot> prompt, and eventually, all the kernel messages and the login console. Log in and have a look around. You're in a VM! Make sure the interface got the IP address you expected. Ping some stuff and make sure NAT works.
$ ifconfig vio0
vio0: flags=8b43 mtu 1500
        lladdr fe:e1:bb:d1:23:c4
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect
        status: active
        inet 10.13.37.203 netmask 0xffffff00 broadcast 10.13.37.255
$ ping -c4 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=9.166 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=7.295 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=9.178 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=25.935 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 7.295/12.893/25.935/7.568 ms

Again, you can use <enter>~. to exit the VM console. You probably want to log off before you do that.  Go ahead and SSH in as well.

$ ssh axon@10.13.37.203
The authenticity of host '10.13.37.203 (10.13.37.203)' can't be established.
ECDSA key fingerprint is SHA256:f39GrAZouHQ3L+3/Qy3FHBma5K+eOe84B9QoFwqNpXo.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.13.37.203' (ECDSA) to the list of known hosts.
axon@10.13.37.203's password:
Last login: Sat Apr 29 14:21:54 2017
OpenBSD 6.1 (GENERIC) #19: Sat Apr  1 13:42:46 MDT 2017

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

$

Forward ports through the NAT (optional)

One last thing I did was to port-forward SSH and HTTP (on high ports) to my VM so that I could get to the VM from anywhere on my home network.

Add these lines to /etc/pf.conf directly under the vmd NAT rule we added earlier.

pass in on $ext_if proto tcp from any to any port 2200 rdr-to 10.13.37.203 port 22
pass in on $ext_if proto tcp from any to any port 8000 rdr-to 10.13.37.203 port 80

Reload the pf configuration

doas pfctl -f /etc/pf.conf

You should now be able to ssh to port 2200 on your host VM's IP. Obviously, browsing to port 8000 won't work until you've set up a web server on your VM, but this was how I started out when working on the PHP/MySQL guides.

2017-04-15

PHP/MySQL Walk-Throughs updated for OpenBSD 6.1

I've stopped maintaining the page for nginx because it hasn't received many views since I updated it about six months ago. I suspect that nginx on OpenBSD is not all that popular because the httpd that ships with OpenBSD provides almost the same functionality as a basic installation of nginx. The pages for httpd in base and Apache2 now cover PHP7 and both configurations appear to work really well.  

PHP/MySQL on OpenBSD's relayd-based httpd 
HiR's Secure OpenBSD/Apache/MySQL/PHP Guide

Feel free to comment on this post if you find any discrepancies or have suggestions. Comments are disabled on the walk-through pages.

(2017-04-23) Edited to add: I went ahead and re-worked the nginx guide, based on the httpd guide with new formatting.
PHP/MySQL with nginx on OpenBSD

2017-04-11

OpenBSD 6.1 Released

I just saw a subtle hint from beck@ that OpenBSD 6.1 is live on the mirrors. Judging from the timestamps on the files,  it looks like the OpenBSD developers stealthily dropped 6.1-Release onto their FTP servers on April Fool's day and didn't tell anyone...

Go get it.

Edited to add: The OpenBSD 6.1 release page has been changed. It's official.