Logical Domains on SunFire T2000 with OpenBSD/sparc64

I've been on a bit of a virtualization kick. This is about OpenBSD, but not vmm.

A couple of years ago, I picked up a Sun Fire T2000. This is a 2U rack mount server. Mine came with four 146GB SAS drives, a 32-core UltraSPARC T1 CPU and 32GB of RAM. Standard hardware includes redundant power supplies, network and serial Advanced Lights Out Management (ALOM), four gigabit ethernet interfaces and a number of USB ports. A number of PCIe and PCI-X slots are available for adding things like RAID controllers, additional network adapters or what-have-you. This system is a decade old, give or take a year or two.

Sun Microsystems incorporated Logical Domains (LDOMs) on this class of hardware. You don't often need 32 threads and 32GB of RAM in a single server. LDOMs are a kind of virtualization technology that's a bit closer to bare metal than vmm, Hyper-V, VirtualBox or even Xen. It works a bit like Xen, though. You can allocate processor, memory, storage and other resources to virtual servers on-board, with a blend of firmware that supports the hardware allocation, and some software in userland (on the so-called primary or control domain, similar to Xen DomU) to control it.

LDOMs are similar to what IBM calls Logical Partitions (LPARs) on its Mainframe and POWER series computers. My day job from 2006-2010 involved working with both of these virtualization technologies, and I've kind of missed it.

While upgrading OpenBSD to 6.1 on my T2000, I decided to delve into LDOM support under OpenBSD. This was pretty easy to do, but let's walk through it. Resources I used:
Once you get comfortable with the fact that there's a little-tiny computer (the ALOM) powered by VXWorks inside that's acting as the management system and console (there's no screen or keyboard/mouse input), Installing OpenBSD on the base server is pretty straightforward. The serial console is an RJ-45 jack, and, yes, the ubiquitous blue-colored serial console cables you find for certain kinds of popular routers will work fine. The networked part of ALOM, if enabled, will probably get a DHCP address and listen for an SSH connection. "admin/changeme" is the default. Resetting it isn't too hard, if you need to. The Internet can help you if you know how to search.

If you have seven minutes to spare to watch the ALOM and the T2000 itself boot up, watch the below video. Put on your earplugs (not your headphones), because this sucker is *loud*. Otherwise, just skip past it.

If there isn't an operating system on the box, the "ok" prompt should show up in the console eventually. If there is an operating system, you can use the "break" command to get to "ok". From there, cd booting is as easy as:
setenv boot-device cdrom
OpenBSD installs quite easily, with the same installer you find on amd64 and i386. I chose to install to /dev/sd0, the first SAS drive only, leaving the others unused. It's possible to set them up in a hardware RAID configuration using tools available only under Solaris, or use softraid(4) on OpenBSD, but I didn't do this.

Enable ldomd:
doas rcctl enable ldomd

I set up the primary LDOM to use the first ethernet port, em0. I decided I wanted to bridge the logical domains to the second ethernet port. You could also use a bridge and vether interface, with pf and dhcpd to create a NAT environment, similar to how I networked the vmm(4) systems.

Cable the second ethernet port to the desired network segment. I'm assuming you got the first ethernet port configured by now. This second port doesn't need its own IP address from the primary LDOM. Configure it by putting this in /etc/hostname.em1:
While you're at it, put those same two lines in /etc/hostname.vnet0. Create additional hostname.vnetN files for as many LDOMs as you plan to set up.

Create a bridge interface configuration with this chunk of text in /etc/hostname.bridge0. Add the rest of your vnet interfaces if you made more than one.
add em1
add vnet0
Fetch the sparc64 minirootXX.fs file from an OpenBSD mirror. Make a blank 4GB disk image for the VM, then write the miniroot to the beginning of it. This will be the easiest way to boot to the OpenBSD installer inside the VM console.

mkdir ~/vm
dd if=/dev/zero of=~/vm/ldom1 bs=1m count=4000 
dd if=miniroot61.fs of=~/vm/ldom1 bs=64k conv=notrunc
Create an LDOM configuration file. You can put this anywhere that's convenient. All of this stuff was in a "vm" subdirectory of my home. I called it ldom.conf:
domain primary {
    vcpu 8
    memory 8G
domain puffy {
    vcpu 8
    memory 4G
    vdisk "/home/axon/vm/ldom1"

Make as many disk images as you want, and make as many additional domain clauses as you wish. Be mindful of system resources. I couldn't actually allocate a full 32GB of RAM across all the LDOMs. I ended up allocating 31.

We're going to dump the factory default LDOM configuration (that is, everything assigned to the primary LDOM, with no actual VMs). We need to copy this as a template, so that we can modify it and send it back to the firmware, telling it to re-allocate hardware assets to an LDOM.

# dump the factory
mkdir ~/vm/default 
cd ~/vm/default 
doas ldomctl dump 

# copy the configuration to the "myldom" directory
cd .. 
cp -r default myldom
cd myldom

# Modify the configuration using the ldom.conf we created
doas ldomctl init-system ~/vm/ldom.conf

# Push ("download") the new "myldom" configuration to the system controller.
cd ..
doas ldomctl download myldom

At this point, you can list the configurations and see that your new one will boot next.

$ doas ldomctl list
factory-default [current]
myldom [next]

Halt the system.
doas halt

Exit the console with "#." and at the system controller prompt, run "reset -c"

Once the system boots up and you log in, check out the running LDOMS.

$ doas ldomctl status
primary           running           OpenBSD running                     0%
ldom1              running           OpenBSD running                     0%

Connect to the console of ldom1 (ttyV0). Additional LDOMs will have consoles at ttyV1, ttyV2, etc. Voila. OpenBSD installer! Assuming your network configuration and cabling is right, it will show up as a machine directly on the LAN.

$ doas cu -l ttyV0
Connected to /dev/ttyV0 (speed 9600)

(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? I
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN.  You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.

System hostname? (short form, e.g. 'foo')

I eventually provisioned seven LDOMs (in addition to the primary) on the T2000, each with 3GB of RAM and 4 vcpu cores. If you get creative with use of network interfaces, virtual ethernet, bridges and pf rules, you can run a pretty complex environment on a single chassis, with services that are only exposed to other VMs, a DMZ segment, and the internal LAN.

If, like me, you end up with an unbootable LDOM configuration, you can always revert back to the factory default, or switch between stored configurations from the system controller. At the sc> prompt, use:

bootmode config="factory-default"
reset -c

Any configuration that you've downloaded to the controller can be used. They will show up by name in the "ldomctl list" command.