What I personally learned at CyberRAID

One last post from me on the inaugural CyberRAID event here in Kansas City.

Leadership

I'm not sure why I was chosen as team captain. Maybe it was because I knew more people on my team than everyone else, because I seemed more confident or because I was one of the first ones present. I wasn't nervous. I felt like I was as prepared as I was going to be. I had my plan, tools and more than half my life behind me that I've spent doing security work in some capacity or another. I thought I was ready to orchestrate a defense team, too, so I happily accepted the position of Captain at the start of the game when people told me "Go up to Dwight! Get our info!" It's not that I can't orchestrate a defense team, but I wasn't as ready as I had thought I'd be, and my plan fell apart pretty quickly.

My plan was to get people organized by what they're good at, and set them on a task. I kind of succeeded at that, but I did plenty of things wrong from the beginning. First, I was stressed but I haven't completely forgotten my manners. My requests probably sounded like a bossy demand followed by "please." Next, I didn't enforce these roles nor did I evaluate how it was going. Some roles changed without much communication. More than once, someone stepped on the toes of someone else who was already working on something. I'm thankful that there wasn't any apparent infighting on my team. Everyone remained rational.

I've lead teams on many projects in my career, but I have no formal management experience or training. I've been an IT worker in a crisis situation more times than I can count. I've never had to lead a crisis response, though. To manage people and work on technical things at the same time is a truly herculean task -- one that I feel I only barely stumbled through.

I learned a lot about myself and hardships faced by leaders in a crisis situation. I also gained a new perspective on IT workers. Among my team, I had some of the most brilliant and capable security minds I know of in the region. On the first day, we had grand ideas and the right mindset, but our implementation of them was slipshod at best. Our good communication skills were the only thing standing between what we had (which was still a good effort) and unabashed anarchy.

Things I learned about leadership and IT teams (and thus, what I need to work on myself):

#1: Technical leaders must lead first and foremost, and help second.

#2: Groups of geeks require a little bit of guidance to avoid replicating (or undoing) work.

#3: Good communication is vital, and its prerequisite is good rapport.

#4: Change management is an extension of good communication. Yeah, I went there.

On the technical side, there were so many things that I'd do different right from the start.

#5: Get visibility to the DMZ network and I'd immediately scan it from the outside while the firewall and system admins work to start locking down obvious things.

#6: Additional Virtual Machines on the DMZ. It would have been great to have a decent (and familiar) IDS on a Span Port. It might have also been fun to have a few honey pots sitting on the DMZ. Had I thought about this ahead of time, I could have totally made it happen. These are tools any one of us could have brought along for the ride in VMs.

#7: Egress filters from the start. There's no good reason not to. They make sense in the enterprise, and they make sense in the game.

Things I'm taking back to the office:

#7: Egress filters! I'm mentioning it twice. Blind SQL injection and RCE exploits are very popular, so crafty hackers and pen-testers often try to leverage these vulnerabilities to launch some process that can notify them that their exploit has worked. This might be popping an xp_cmdshell to launch ping with a special payload they can look for in return, or it may be something much more quotidian, such as a reverse_tcp or meterpreter call-back from metasploit. Again, egress filters make sense in the real world. To further this point, watch out for obscure tunneling through ICMP and DNS. Ideally the DNS server your DMZ uses should not allow recursion.

#8: We all know that despite our best efforts, a dedicated adversary will find a way in. For some reason, this exercise made it sink in a little better. It's been a while since I've worked somewhere that was breached on my watch. This further boosts my desire to learn more about modern post-breach activities both defensive (forensics, containment) and offensive (post-exploitation, pivoting). I have some serious reading and research to do!

Who else was at CyberRAID, CCDC, SANS ICE II or any other recent exercise like this? What did you get out of it?

Cyber-RAID 0 - Blue Team Wrap-Up

First and foremost, I owe a huge thank you to all my team-mates. Blue Team 2 (which I was on) took home first place in the Defense category by a margin of 5% or so. Like I said yesterday, we had an all-star team with such an awesome range of talents. There's no way any one or two of us could have done as well on our own. We used everyone's skills, and we learned quite a bit from one another. I also learned a lot from watching the Red Team work.

The VMs on each team's network for this exercise:
10.10.9.11 - Secondary DNS Server - Red Hat Linux 7.3 (Valhalla)

• DNS had to remain open to the outside world. Marks would occur against your network if the scorebot system could not get your system to resolve the addresses it's supposed to be hosting.
• SSH: the scorebot user must be able to log in.

10.10.9.14 - Active Directory - Windows Server 2003

• DNS had to remain open to the outside world. This one was properly configured.

10.10.9.15 - Exchange Server - Windows Server 2003

• OWA on port 80 had to remain open to the outside world.

10.10.9.56 - PBX Server - Older Debian install

• HTTP had to remain open to the outside world.

10.10.9.69 - DB Server - Older Debian install

• SSH: the scorebot user must be able to log in.
• MySQL had to be open to the outside world, allowing the scorebot user to select from a special table.

10.10.9.70 - Web Server - Older Debian Install

• HTTP had to remain open to the outside world, and a flag file in the web root had to remain intact (subject to MD5 checksum)
• FTP had to be open to the outside world, allowing the scorebot user to retrieve a file.
• SSH: the scorebot user must be able to log in.

10.10.9.89 - DB Server - Older Debian install

• SSH: the scorebot user must be able to log in.
• MySQL had to be open to the outside world, allowing the scorebot user to select from a special table.

10.10.9.91 - Web Server - Older Debian Install

• HTTP had to remain open to the outside world, and a flag file in the web root had to remain intact (subject to MD5 checksum)
• FTP had to be open to the outside world, allowing the scorebot user to retrieve a file.
• SSH: the scorebot user must be able to log in.

10.10.9.253 - Solera IDS and packet capture VM

• This wasn't a target, and didn't have anything the attackers could get their hooks into. It was provided to us, but none of us knew how to use it, nor did any of us have time to play with it. We instead relied on local logs and traditional packet captures from Wireshark and tcpdump.

Gotchas:

All the systems had many extra services running. There were default passwords on database, web-app and shell accounts. Anonymous FTP (with upload ability) was allowed on the two FTP servers.

This was a self-contained and isolated network, making traditional methods of patching impossible. What's more: in order to protect the Blue Team laptops, the VMWare server had dual interfaces. Blue Teams were only given enough infrastructure to connect to the management interface of the virtual environment. That means that Blue Team only got console access to the environment. The hosts themselves were natted to the hostile network through a Cisco ASA.

Day 1, with more detail:

Our first action was to snapshot all of the VMs. This would let us revert them in the case of complete ownage or if one of the red-teamers rm'd our boxes. Next, we figured out who was good at what kinds of things. This would come in handy later on. Next, we all started exploring the VMs through the VMWare consoles. There were eight VMs and eight people. You'd figure we could have each taken one of the VMs and started hardening or patching them. But no, everyone immediately opened 3, 4, or 5 VMs at once and started trampling on other peoples' sessions. I was just as guilty as everyone else.

I think most of us were more prepared for a totally Windows-centric environment. Several of us brought Windows Update patches on DVD. Our team started patching those right away, while two guys with firewall experience worked to restrict only incoming services we needed.

It was already too little, too late. Surbo from i-Hacked completely and totally pwn3d our exchange server early on in the game while my friend Eric was busy trying to figure out what in the hell was going on. At this point, scoring hadn't started yet, so the firewall guys shut down all incoming traffic.

Before lunch, the systems people started hardening and patching the boxes as best as we could without having Internet access. We all had reasonably good communication, and we all had some good ideas. Egress filtering came into mention before lunch. The firewall guys built up a good rule-set that would allow all scored incoming services to work properly once the "deny all" rule was pulled, while disallowing any connections out from the systems that weren't absolutely needed. All of this was to shut down any phone-home scripts.

After lunch, we opened up the firewall ruleset to the world and started watching the score board. A lot of time was spent yesterday troubleshooting the firewall and various services, making sure we had as few marks against us as possible.

BIND on our Red Hat Linux box (using a 2002-era version of that distro!) ended up being brutally misconfigured. It was also a horribly vulnerable version, but no red-teamers seemed to exploit it to properly pop a shell. Yesterday, no blue team could solve the issue.

Toward the end of the day, our Pointy-Haired Boss demanded that he wanted the company's computers to be able to access http, https, smtp, dns, mysql and postgresql outside the network. Firewall changes that we made broke all our incoming services for a while, due to using the wrong IP addresses in the DMZ rules.

We got these changes implemented, and then found that our PHB was using his personal laptop to connect out over those ports to verify that the changes were made. After that, scoring was complete for Day One.

Day 2

One of the game rules was that we couldn't explicitly block anything by IP address either inbound or outbound. First thing in the morning, we created a group for our DMZ servers then placed full egress filtering on them. At some point in the morning, the PHB verified he could still get out to the other networks from his laptop on our DMZ. He had no idea what we had done.

In normal IT shops, this is known as adhering to the letter of the law instead of the spirit of it. It's a type of insubordination that IT guys can use against management when they really think they know what's better for the company. I really don't like resorting to these kinds of tricks in the real world, but since it's a tool that IT guys have in the real world, it's a tool my team was willing to use. Later on in the day, the PHB came back around and asked us to prove to him that our DMZ servers could also get out. This was a much more blatant deception. Our firewall guy dropped the egress rule while one of our Linux guys feigned a bit of ignorance to stall the boss for 15 seconds or so while we pulled the shields down.

These were risky moves. If at any time we would have been found to be out of compliance with the boss's demands, it would have cost us an instant 5,000 marks against us. In the real world, moves like this could potentially cost you your job. Of course, in the real world, you can often talk the boss out of making bad decisions, and the boss would likely have to provide a good business reason to implement the features that were being asked of us.

Things were going more smoothly on day two. We hooked up a switch to the DMZ to give our personal laptops some visibility to the front-end of our network. This helped things immensely, and it's something I'd do immediately on the first day of an event like this in the future. I brought speakers, too, so that my team had some tunes. We rocked out to 808 State, Paul Van Dyk, Daft Punk, Juno Reactor, Steve Porter, Orbital, White Zombie and more. At a reasonable volume, of course.

With the exception of the badly configured BIND server, our team and Team 3 were holding steady on marks against us. We both started to identify some SQL injection and default usernames being exploited on our networks. Team 3 had gained on us a lot at the end of Day One, but we held a margin of about 200 points ahead of their team for the entirety of Day Two.

More demands rolled in. We had 30 minutes to open Exchange's SMTP to the world, and we had to add a new zone with MX, A and NS records to the active directory's DNS server. Our Windows admins aced it.

While that was going on, I was digging into BIND on the redhat box. I was the first blue-teamer to finally dig into the issue and fix it. Only one other team was able to resolve this issue, and it took them a while after I had fixed my team's. I'll spare you the details, but it was really, really misconfigured.

At this point, several attackers had some kind of hooks into a few of our systems, but they were having trouble scoring phone-homes. With all our tasks out of the way, we went into incident response mode, booting and cleaning up after red teamers.

One last PHB demand for the day: Set up VoIP trunks and extensions on our Asterisk server, and allow the protocols needed to talk between networks. We weren't going to actually communicate over it, but we needed to at least show that we could get into the interface and configure Asterisk with FreePBX. Since my machine was plugged into the external switch and since no one on my team (even me) had any Asterisk experience that would make them any better for the job, I went ahead and took the charge on that one. It required collaborating with my team-mates to grok the config files and tweak the databases so I could get logged in.

There was about half-an-hour of incident response left after that. About 10 minutes before scoring closed, the game organizers joked that he'd remove 10,000 marks from any team who could provide him with full packet captures of the entire exercise from Solera by the time scoring closed. I say he was joking, because that VM had only 2GB of hard drive space on it, so there's no way it could have stored it all. Even if it could have stored it, I doubt anyone could have exported all that data in 10 minutes.

All of the tasks required collaboration, and the entire exercise calls for a team of people with diverse skills.

Cyber-RAID 0, Day One - Blue Team

Asmodian X is Red-Teaming, but here are some of my thoughts on Cyber-RAID 0 from the Blue Team side.

First: today was one of the most frustrating and stressful days of my entire IT career. That's saying something, considering that I'm officially off the clock, on vacation with a "four-day weekend." I'm burning vacation days to participate, and some good friends of mine with strong ties to the financial, law enforcement and education industries sponsored my attendance at this event. Being stressed out doesn't mean I'm not having fun, though. This is my first time in a game like this, so it's new and exciting to me.

Next, I have an all-star team working with me. We got to self-organize into groups, so I already knew some of the people on my team, and what they're capable of.

Blue Teams
The "Blue Team" is actually 4 teams with 8 members each. The goal of each blue team is to get as few marks against their network as possible. Each "network" is a VMWare server with 8 VMs. Each team gets a nearly identical setup, save for a few passwords being different. Marks are racked up based on the integrity of mandatory services. Your exchange server goes down? That's a certain number of marks against your team. An attacker deletes or modifies a certain file on your web server? More marks against you. These accumulate periodically until you restore the services to their intended state. I won't go into what all services are checked or what kinds of virtual machines we're running, since some of my red-team buddies might take advantage of the information. It's safe to say that there were many services running that didn't need to be.

By gathering enough data to implicate a specific attacker, each Blue Team can recover some of the marks against their network as well as getting the attacker "arrested" - sidelined for 30 minutes.

The Red "Team"
The Red Team is full of lone-gunmen who are free to collaborate if they wish, but they're much less structured than the Blue Teams are. Each Red Team member scores points for themself by getting phone-home scripts or binaries to run from the Blue Team network. Ideally, they exploit remote-code-execution vulnerabilities, pop a box to get a session or shell, or otherwise get the Blue Team's systems to contact the scoring server on their behalf. The goal of the Red Team attackers is to score as many points as possible. If they can persist their hold on a Blue Team network, they can continue to rack up points by running their phone-home processes repeatedly. Note: these scripts can't be run in an infinite loop effectively to rack up tens of thousands of points per minute.

The Pointy-Haired Boss
Toward the end of the day, our Virtual CEO decided to DEMAND that we change our firewall rulesets to open certain ports for outbound access to any remote server. As you can imagine, per the rules of the game, many of the blue teams had opted to implement egress filtering rules that would allow the services to be contacted from the outside, but to disallow any outgoing connections originating from our servers in order to foil any successful "phone home" attempts, even in the event of a complete system compromise. This demand was certain to throw a wrench into egress filtering rules, but the team I'm on dealt with it well enough. Tomorrow, more demands will be thrown at us, and the usual fare of IT issues will be simulated: password resets, account creation, etc.

Results
"We have met the enemy, and he is us!"

At the start of the game, each of the Blue Teams caused more problems for themselves than the attackers did: team-mates accidentally knocking out power to production systems, intentionally telling Red-Teamers to "Piss Off" by modifying an integrity-monitored web page, and failing to fully understand this network that was just dropped into our laps are only three examples of the sort of frustrating things I saw today, and pretty much every team had the same problems.

There's still a half-day ahead of us, but the last time I checked, our Blue Team team was in the lead (by virtue of having nearly a thousand fewer "marks" against us than our closest competitor) but I have a feeling we'll need to work hard to stay in the lead. The members of the Red Team seem to be having a very, very rough go of things as well. The top attacker, last I saw, had a mere dozen points. My guess is that the attackers are landing a few successful exploits, but are having difficulty with the way points are awarded.

We'll see how it turns out at 13:00 tomorrow afternoon. B-Sides runs all day tomorrow as well, but Cyber-RAID participants will miss out on the first half.