AppSec Lab: RasPwn with a MiFi-8800L JetPack Router

I'm hosting a few Application Security workshops later this year. I settled on RasPwn for the lab because it comes pre-configured with a bunch of vulnerable applications out of the box.

RasPwn acts as a stand-alone wireless access point using the Raspberry Pi's on-board Wi-Fi. If you plug in ethernet, it can route packets, but DNS forwarding seems broken. Some of the participants will have to be online and available during the workshops, so I wanted to make sure the lab has full Internet access. Additionally, it helps when folks can look up information about vulnerabilities while learning new concepts. I won't always be able to rely on on-site ethernet to provide Internet access to participants, so I decided to set up my MiFi 8800L hotspot as an Internet gateway on RasPwn, and I had to make sure DNS worked.

Hotspot setup:
When you plug in most WiFi hotspots over USB, some will only charge the internal battery, while others will immediately show up as a network device. Some also show up as a "virtual USB drive" with drivers and software. The Inseego MiFi-8800L touch-screen model prompts you when you plug it in, and has an option to serve Internet via USB only or USB+WiFi.

If you use the Web UI, you can set this option as the default. There's no way to set it up from the touch-screen interface. All other Inseego (and their previous brand, Novatel) hotspots I've used can be set up to provide USB Internet access by default, using the web admin portal. See owners' manual for details on accessing the admin portal. It'll probably vary widely by model. Example from my 8800L:

RasPwn setup:
  • Download the RasPwn software and follow the install instructions on the download page. If you've messed with Raspberry Pi distributions before, this should be pretty self-explanatory.
  • Place the card into a Raspberry Pi 3 and power it up. You won't need a screen or keyboard for anything. I did have to power-cycle the Raspberry Pi after the first boot for the WiFi network to show up. You may have to do the same.  
  • When RasPwn boots up, you'll see a new WiFi network called RasPwnOS show up. Connect to it. The default WiFi password is In53cur3! 
  • SSH to The username is pi and the password is pwnme!
    • ssh pi@
  • Set up eth1 (for the hotspot's USB interface)
    • edit /etc/network/interfaces with vi or nano
    • insert the two lines below, preferably after "eth0" is specified:
      allow-hotplug eth1
      iface eth1 inet dhcp
    • Save the file
  • Change the IP masquerading rules for iptables to use eth1
    • edit /etc/iptables.up.rules with vi or nano
    • change the MASQUERADE rule from
    • save the file
  •  Set up the DHCP server to issue an external backup resolver
    • edit /etc/udhcpd.conf with vi or nano
    • change the "opt dns" line from
      opt     dns
      opt     dns
(Note: the DNS stuff is kind of hacky. You could configure the on-board bind9 DNS server to resolve recursively, but it's more complicated and this works just fine)
  • Reboot RasPwn and test Internet connectivity.
    • sudo reboot
    • Close your SSH window
    • Wait a minute or so
    • Reconnect to the RasPwnOS wifi network
    • Try browsing the internet. If it doesn't work, make sure the HotSpot is showing a USB connection. You may need to unplug it and plug it back in, or unplug the hotspot, reboot RasPwn again, and plug the hotspot in after RasPwn boots up all the way. 
Okay, so let's hack something!
  • Go to playground.raspwn.org from your RasPwn WiFi connection 
  • Pick an app and start hacking!
The first thing in the playground is the OWASP Bricks practice application. It's intentionally vulnerable and designed to be increasingly complex with each challenge building on what you learned with the previous ones.

Some of the exercises will require an intercepting proxy such as BurpSuite, Charles Proxy, or OWASP ZAP, but the first login page can be hacked with just a browser. I actually didn't read any documentation for Bricks, and had never played with it before setting up RasPwn. My first login attempt was "admin" with a password of "admin" and it logged me in.

It wasn't until I saw the SQL in the footer that I knew this was supposed to be an SQL Injection challenge.Whoops. Okay, let's try this again with foo and a password of bar.
Okay, so that's what "access denied" looks like. Now let's throw some SQL injection into the username field. Here, I used a username of "foo ' OR 1=1 -- " (note the space after the -- comment, that's needed for MySQL and maybe other databases to acknowledge a comment).

And we've successfully used SQL Injection to hack the first Bricks challenge.

Happy hacking, friends!


OpenBSD 6.6 released early!

OpenBSD 6.6 was released earlier this week. Along with it come a number of exciting enhancements to not just hardware support, but improvements to the installer, and security enhancements to the userland as unveil and pledge continue to get integrated more. You can read the entire change log at the link above.

As I had mentioned when OpenBSD 6.5 came out, sysupgrade(8) was making snapshot upgrades a breeze for those of us running -CURRENT. A few weeks back, the team made a sysupgrade patch available to bring this functionality to OpenBSD 6.5, so that folks could take advantage of it to upgrade to OpenBSD 6.6 when it was released. I spent some time this week testing it on my VMM virtual machines running OpenBSD-Stable and indeed, it's just as easy as upgrading snapshots. Then I upgraded everything else, including my production servers.

To upgrade from OpenBSD 6.5, as root or with doas, just run these commands:
sysupgrade (it'll download what it needs, reboot, do the upgrade, and reboot again, fully unattended)
pkg_add -u (to upgrade all of your binary packages to the latest version)

As per usual, I updated the OpenBSD/httpd/MariaDB/PHP walk-through



Scams: Two Close Calls

Over on Twitter, Eric Mill tweeted:

My stories are a bit too long for Twitter, but let's unpack two scams that were pretty well done which I fell for -- at least partially -- recently. We'll also highlight some early telltale signs that something was wrong, which I embarrassingly either missed or shrugged off at first, so that perhaps you'll be able to spot these scams better.

Feel free to share your own "scammed" stories in the aforementioned Twitter thread, or here in the comments.

My bank's doppelganger phone number 

My debit card had been getting flaky in chip readers recently. In fact, the plastic around the embedded chip contacts was breaking. It was time for a new card. I turned the card over to find the customer service number, which was a bit difficult to read because it was inexplicably printed directly under the embossed credit card number. The conversation went like this:

Agent: Thank you for calling [my bank's name] Card Services, this is [name]. Who am I speaking with today?
Me: ... my real name ...
Agent: Good morning, [name]! How can we help you today?
Me: My card's pretty worn out and isn't working consistently. I need a replacement, please.
Agent: We can get your new card sent out right away! What's your card number?
Me: ... reads the card number ...
Agent: And the expiration date?
Me: ... reads the expiration date after a short pause ...
Agent: And for security purposes, the 3-digit CVV code.

I suspected something was fishy when the agent requested my expiration date, but a few things were going on in my head. *I* called them, not the other way around. Also, the agent was professional and friendly, and even answered the phone with my bank's name. But when I got asked for the CVV code, that's when I knew something was completely wrong. I took a very close look at the back of my card, and compared it to the phone number I was calling. One of the hardest-to-read digits of the phone number on the back of my card was an 8, instead of the 0 that I dialed. 

Having disclosed my first and last name, Primary Account Number and expiration date, I called my bank's real number and had my card disabled and ordered a replacement. My real bank didn't even need my whole card number to look my account up when I contacted them. That was the first red flag, and I had completely missed it.

The mobile phone carrier "Fraud Department"

I got a call unusually early on a Saturday morning from my mobile carrier's customer service number (It was in my contacts and it was the right number). The person on the line was from the fraud department, and explained that they'd seen unusual activity on my account. Someone had changed the address on my account and then mail-ordered a new iPhone X for a little over $1000. She explained that any time an address is changed right before a purchase, their system flags the transaction for review.

This sounds quite rational to me. But who had access to my account? My wife's caregiver had recently quit and moved out, and she had a phone on our account. When she quit, we made her leave the phone behind. Perhaps she was up to no good? I was confused and furious; I felt violated. I explained that our old roommate may be behind it.

"Do you recognize this address?" the agent asks, before rattling off some random address in The Bronx. There's no way our old roommate was in The Bronx. Her family is all in Texas and Oklahoma. I thought it was very strange that a fraud department representative would disclose this information to me. That was the first red flag. I chose to ignore it.

"Sir, we'll take care of this," she explains. "We never processed the transaction, because it was flagged for review. Let's get your account secured! I just sent you a one-time code to verify you're in possession of the phone tied to your account. Can you read it to me?" There's red flag #2. I was still far too flummoxed to see it.

I get an SMS message from my wireless carrier. In a rush, I don't even read the whole message. I rattle off the 8-digit number. But then something catches my eye as I bring the phone back up to my head. Right at the beginning of the text message, there's a big disclaimer along the lines of "For the security of your account, we will never contact you for this code."

Now the plot thickens, but I'm a bit sharpened up. I begin to suspect this is an elaborate ruse, and it worked pretty well.

"I am not comfortable with this call," I say, grabbing my laptop and trying to log in to my account. The password has already been changed, in under a minute. Even the temporary code I'd just been texted isn't working. "Is there a direct number I can call you back at?"

"I'm securing your account, sir. I assure you, I'm from the fraud department. We do this all the time. I understand you're angry. We're almost finished with the password reset," she says, quite professionally. Meanwhile, I'm issuing my own password reset through my carrier's website. I get the exact same text message this agent had "sent" me, with a different one-time code, obiously.

I tell her that something has come up and I'll have to call her back in a few minutes. I ask for her name again. "Jessica," she says.

"May I please have your direct phone number so I can call you back in a bit, Jessica?"

"Sure thing! Just call me back at the customer service number I'm calling from. I'm at extension 105."

I hang up, and finish resetting my password.

I call back while scrolling through my account to make sure nothing's been purchased or changed. I get my carrier's customer service, as expected. There is no way to dial an extension. I get a real human on the phone, and ask to be transferred to extension 105. There is no extension 105. There *IS* a fraud department, but they do not call subscribers directly. Customer service pulls up my account's history. They keep a record of every agent who's looked at my account. No one had opened my account since we added a line to the account for our recently-departed caregiver.

Attackers can easily spoof a call to make it look like it came from any phone number. Armed with only my carrier's customer service phone number, and a list of phone numbers assigned to my carrier, they can trawl through the list and perpetrate this scam over and over. Having a young woman with a southern twang make the calls, with obvious call-center background noise was icing on the cake.


Former student pleads guilty in "USB Killer" case

A few weeks old, from the Department of Justice website, comes the first mention I've heard of a "USB Killer" being used nefariously at scale:

Akuthota admitted that on February 14, 2019, he inserted a “USB Killer” device into 66 computers, as well as numerous computer monitors and computer-enhanced podiums, owned by the college in Albany.  The “USB Killer” device, when inserted into a computer’s USB port, sends a command causing the computer’s on-board capacitors to rapidly charge and then discharge repeatedly, thereby overloading and physically destroying the computer’s USB port and electrical system.

Akuthota admitted that he intentionally destroyed the computers, and recorded himself doing so using his iPhone, including making statements such as “I’m going to kill this guy” before inserting the USB Killer into a computer’s USB port.  Akuthota also admitted that his actions caused $58,471 in damage, and has agreed to pay restitution in that amount to the College.

This is the predominant threat model that came to mind when USB Killer Hype kicked in about a year and a half ago. That is, someone repeatedly using it to attack unattended computers. While USB Killer devices are no longer one-off devices, and they have achieved a sort of "commercial viability," the kind that look convincing enough for a random person to insert into their own PC cost more than $60 USD. That's a lot of cash to spend on potentially destroying devices belonging to a random person by just leaving it laying around. Cheaper ones that are chunky (or have no case at all, or have cases emblazoned with menacing logos) are easier to come by, but obviously look more suspicious.
This is a pretty "clean" way for someone to destroy a computer they have physical access to, but ultimately, "physical access is total access" as the saying goes.


OpenBSD 6.5 released early

A few days late posting this, but OpenBSD 6.5 hit the wire last week, ahead of the May 1 target release date. Our OpenBSD Web Server Guide -- using the built-in httpd -- has been updated. And the PHP-FPM quirks from OpenBSD 6.4 got ironed out.

As far as installation and daily use go, you probably won't notice much has changed in OpenBSD 6.5. There was a ton of work done in areas of hardware support and network-stack enhancements.

If your console supports it, you may notice a new default console font (called "Spleen"). I've seen this on my OpenBSD-Current laptop for a few months. At first, I didn't really like it, but it's quite readable and has grown on me when working in text-only mode. I'm considering setting it as my default xterm font as well.

If you use OpenBSD-CURRENT with snapshots, however, there's already some fun stuff unfolding there, with sysupgrade(8) among them. This makes in-place upgrades a breeze. While it's not available in OpenBSD 6.5, upgrading from one release to the next should get a lot easier in about a year's time. The 6.6 to 6.7 upgrade will be the first supported release with this tool, unless they backport it to 6.5 with an errata/patch -- unlikely, indeed...


OpenBSD VMM Hypervisor Part 4: Running Ubuntu (and possibly other distros)

TL;DR: you cheat.

I've been trying for almost a year to figure out how to get the cloud-init meta-data service to work with the Ubuntu Cloud image. I've asked on misc@ and other OpenBSD groups, and no one has an answer. The documentation is vague. If anyone ever figures out how to configure meta-data, let me know. I'd still like to give it a shot.

Last week, I rescued a server from a pile of computers destined to be scrapped and recycled. For me, it's the perfect setup for getting serious with OpenBSD VMM in my home lab. Two older Xeon E5-2620 CPUs and 128 GB of RAM. No hard drives, but it came with enough empty drive trays for getting started. I threw a pair of old SAS drives into it.

No surprise, OpenBSD just worked. This renewed my fervor for replicating a bunch of my cloud instances at home, and there's a lot of Ubuntu in use.

I decided to bite the bullet and just use qemu to do the installation and configuration of Ubuntu. Install qemu from packages:

doas pkg_add qemu

Download Ubuntu Server. I've actually used both 18.04 LTS and 16.04 LTS. I'm focusing on 16.04 for this because that's what I'm running on most of my EC2 instances.

Create a disk image.

vmctl create qcow2:ubuntu16lts.qcow2 -s 20G

Boot the ubuntu ISO and attach the new ubuntu disk image to qemu:

qemu-system-x86_64 -boot d -cdrom ~/Downloads/ubuntu-16.04.5-server-amd64.iso -drive file=ubuntu16lts.qcow2,media=disk -m 640

Install Ubuntu as usual. I didn't bother adding anything other than the SSH server during installation. qemu is really slow on OpenBSD, but it works... eventually. When the install is done, shut down and then restart qemu without the installation ISO attached.

qemu-system-x86_64 -drive file=ubuntu16lts.qcow2,media=disk -m 640

Log in with the user-level account you created. There are only two things to tweak before it's ready to run in vmm: Configuring the serial console, and the network interface.

Under qemu, Ubuntu sees "ens3" as the network interface. Under vmm, the network interface is "enp0s3". Change "ens3" to "enp0s3" in /etc/network/interfaces if you're using 16.04. On Ubuntu 18.04, you must instead change the "netplan" config file in /etc/netplan/50-cloud-init.yaml with the same kind of change, ens3 to enp0s3.

To configure the serial console, edit /etc/default/grub and change this line:



GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

then run

sudo update-grub

Shut down qemu again. Your disk image is basically ready to go under vmm.

To save the trouble of having to mess with qemu again, I recommend creating derivative images of the one you just created, and using those for vmm.

vmctl create qcow2:ubuntu16lts-1.qcow2 -b ubuntu16lts.qcow2

Add the new disk image to a configuration clause in /etc/vm.conf on your OpenBSD host system. Mine looks like this:

vm "Ubuntu16.04" {
        owner axon
        memory 4096M
        disk "/home/axon/vmm/ubuntu16lts-1.qcow2"
        interface {
                switch "local"
                lladdr fe:e1:ba:f0:eb:b0

For more information about setting up switches and networks in vmm, see Part 2 of my VMM series.

Voila! Ubuntu in VMM!

Although the configuration files you must edit to make it work might vary, you can do the same thing and it may very well work for text-mode-only distributions.

I actually didn't need to use qemu to get arch linux installed in vmm, but it was somewhat tedious to do entirely in vmm, and it took me a few tries to get it right. Qemu might have been easier.


OpenBSD vmm Hypervisor Part 3: qcow2 and derived disk images

With OpenBSD 6.4, the VMM hypervisor got support for qcow2 disk images. This format is used by QEMU, but it has several features that make it a better choice than raw image files. The images are dynamically-allocated, so the disk image file grows as you use more space instead of taking up the entire filesystem size when the image is created. It won't ever shrink, though. "Derived images" are also supported. While VMM doesn't officially support snapshots yet, you can kind of get away with using derived images to do something similar. I'll cover that toward the end of this article.

You will probably want to have the networking set up on your OpenBSD VM host before you continue. That information is covered in Part 2 of my VMM series.

To create a qcow2 image, prefix the image file name with qcow2:

vmctl create qcow2:obsd64-base.qcow2 -s 10G


You can also use the qemu-img utility (from qemu in the package repository) to convert an existing raw image to qcow2 format, if you've already been using VMM before OpenBSD 6.4 was released. This image file will not be dynamically sized, but it can serve as a base image for derivatives:

qemu-img convert obsd64.img obsd64-base.qcow2

Start the VM using your bsd.rd as the boot image, then follow the installer prompts. 

doas vmctl start obsd64-base -n local -m 512m -d obsd64-base.qcow2 -b /bsd.rd -c

When the install is done, rebooting will just bring the installer back. Exit to shell instead, type "halt -p" and use the ~. command sequence to exit the VM. Anything else you press will probably reboot the system (back into the installer). Now you have a pristine, freshly-installed OpenBSD image to start from.

To create a derived image, select your base image with the -b option to vmctl create:

vmctl create qcow2:obsd64-test1.qcow2 -b obsd64-base.qcow2

WARNING: If you make any changes to the base image, all derived image files it was based on will become corrupt and unusable. You can remove write access to the base image if you want. The VMs relying on derived images will run fine. 

chmod 400 obsd64-base.qcow2

Now, create a VM in /etc/vm.conf with the new obsd64-test1.qcow2 image file. All changes will be stored in this new image file. The original filesystem image will remain unchanged, and you can make as many derived images as you want from it.

# bridge0 for VMs, NAT and dhcpd (required for networking in this example)
switch "local" {
interface bridge0

# OpenBSD Stable
vm "test.vm" {
owner axon
memory 512M
disk "/home/axon/vmm/obsd64-test1.qcow2"
interface {
switch "local"
lladdr fe:e1:ba:d0:eb:ab

Reload vmm's configuration:
doas vmctl reload

Then go ahead and boot it up with the console attached:
vmctl start test.vm -c

For snapshot-like functionality, you can make a copy of your derived image and save it with another file name in the same directory. You should shut down the VM before you do this, though. To restore, just copy it back over the derived image, or create a new vm clause in /etc/vm.conf pointing to your saved derived image file.

cp obsd64-test1.qcow2 snapshot-2018-11-01_obsd64-test1.qcow2

You can run multiple VMs at the same time, with different derived images from the base image as well. If I create a new derived image file and add a vm clause for it, both VMs can run at the same time.

vmctl create qcow2:obsd64-test2.qcow2 -b obsd64-base.qcow2

I added this to /etc/vm.conf:
# OpenBSD test2
vm "test2.vm" {
owner axon
memory 512M
disk "/home/axon/vmm/obsd64-test2.qcow2"
interface {
switch "local"
lladdr fe:e1:ba:d0:eb:ac

Reload vmm, and start up your VMs!
doas vmctl reload
vmctl start test.vm
vmctl start test2.vm

You can attach to the consoles of each to see that they're running. Remember that you can use the [RETURN]~. key sequence to exit the console.

vmctl console test.vm
vmctl console test2.vm