2010-09-17

Cyber-RAID 0 - Blue Team Wrap-Up

First and foremost, I owe a huge thank you to all my team-mates. Blue Team 2 (which I was on) took home first place in the Defense category by a margin of 5% or so. Like I said yesterday, we had an all-star team with such an awesome range of talents. There's no way any one or two of us could have done as well on our own. We used everyone's skills, and we learned quite a bit from one another. I also learned a lot from watching the Red Team work.

The VMs on each team's network for this exercise:
10.10.9.11 - Secondary DNS Server - Red Hat Linux 7.3 (Valhalla)

  • DNS had to remain open to the outside world. Marks would occur against your network if the scorebot system could not get your system to resolve the addresses it's supposed to be hosting.
  • SSH: the scorebot user must be able to log in.
10.10.9.14 - Active Directory - Windows Server 2003
  • DNS had to remain open to the outside world. This one was properly configured.
10.10.9.15 - Exchange Server - Windows Server 2003
  • OWA on port 80 had to remain open to the outside world.
10.10.9.56 - PBX Server - Older Debian install
  • HTTP had to remain open to the outside world.
10.10.9.69 - DB Server - Older Debian install
  • SSH: the scorebot user must be able to log in.
  • MySQL had to be open to the outside world, allowing the scorebot user to select from a special table.
10.10.9.70 - Web Server - Older Debian Install
  • HTTP had to remain open to the outside world, and a flag file in the web root had to remain intact (subject to MD5 checksum)
  • FTP had to be open to the outside world, allowing the scorebot user to retrieve a file.
  • SSH: the scorebot user must be able to log in.
10.10.9.89 - DB Server - Older Debian install
  • SSH: the scorebot user must be able to log in.
  • MySQL had to be open to the outside world, allowing the scorebot user to select from a special table.
10.10.9.91 - Web Server - Older Debian Install
  • HTTP had to remain open to the outside world, and a flag file in the web root had to remain intact (subject to MD5 checksum)
  • FTP had to be open to the outside world, allowing the scorebot user to retrieve a file.
  • SSH: the scorebot user must be able to log in.
10.10.9.253 - Solera IDS and packet capture VM
  • This wasn't a target, and didn't have anything the attackers could get their hooks into. It was provided to us, but none of us knew how to use it, nor did any of us have time to play with it. We instead relied on local logs and traditional packet captures from Wireshark and tcpdump.

Gotchas:
All the systems had many extra services running. There were default passwords on database, web-app and shell accounts. Anonymous FTP (with upload ability) was allowed on the two FTP servers.

This was a self-contained and isolated network, making traditional methods of patching impossible. What's more: in order to protect the Blue Team laptops, the VMWare server had dual interfaces. Blue Teams were only given enough infrastructure to connect to the management interface of the virtual environment. That means that Blue Team only got console access to the environment. The hosts themselves were natted to the hostile network through a Cisco ASA.

Day 1, with more detail:
Our first action was to snapshot all of the VMs. This would let us revert them in the case of complete ownage or if one of the red-teamers rm'd our boxes. Next, we figured out who was good at what kinds of things. This would come in handy later on. Next, we all started exploring the VMs through the VMWare consoles. There were eight VMs and eight people. You'd figure we could have each taken one of the VMs and started hardening or patching them. But no, everyone immediately opened 3, 4, or 5 VMs at once and started trampling on other peoples' sessions. I was just as guilty as everyone else.

I think most of us were more prepared for a totally Windows-centric environment. Several of us brought Windows Update patches on DVD. Our team started patching those right away, while two guys with firewall experience worked to restrict only incoming services we needed.

It was already too little, too late. Surbo from i-Hacked completely and totally pwn3d our exchange server early on in the game while my friend Eric was busy trying to figure out what in the hell was going on. At this point, scoring hadn't started yet, so the firewall guys shut down all incoming traffic.

Before lunch, the systems people started hardening and patching the boxes as best as we could without having Internet access. We all had reasonably good communication, and we all had some good ideas. Egress filtering came into mention before lunch. The firewall guys built up a good rule-set that would allow all scored incoming services to work properly once the "deny all" rule was pulled, while disallowing any connections out from the systems that weren't absolutely needed. All of this was to shut down any phone-home scripts.

After lunch, we opened up the firewall ruleset to the world and started watching the score board. A lot of time was spent yesterday troubleshooting the firewall and various services, making sure we had as few marks against us as possible.

BIND on our Red Hat Linux box (using a 2002-era version of that distro!) ended up being brutally misconfigured. It was also a horribly vulnerable version, but no red-teamers seemed to exploit it to properly pop a shell. Yesterday, no blue team could solve the issue.

Toward the end of the day, our Pointy-Haired Boss demanded that he wanted the company's computers to be able to access http, https, smtp, dns, mysql and postgresql outside the network. Firewall changes that we made broke all our incoming services for a while, due to using the wrong IP addresses in the DMZ rules.

We got these changes implemented, and then found that our PHB was using his personal laptop to connect out over those ports to verify that the changes were made. After that, scoring was complete for Day One.

Day 2
One of the game rules was that we couldn't explicitly block anything by IP address either inbound or outbound. First thing in the morning, we created a group for our DMZ servers then placed full egress filtering on them. At some point in the morning, the PHB verified he could still get out to the other networks from his laptop on our DMZ. He had no idea what we had done.

In normal IT shops, this is known as adhering to the letter of the law instead of the spirit of it. It's a type of insubordination that IT guys can use against management when they really think they know what's better for the company. I really don't like resorting to these kinds of tricks in the real world, but since it's a tool that IT guys have in the real world, it's a tool my team was willing to use. Later on in the day, the PHB came back around and asked us to prove to him that our DMZ servers could also get out. This was a much more blatant deception. Our firewall guy dropped the egress rule while one of our Linux guys feigned a bit of ignorance to stall the boss for 15 seconds or so while we pulled the shields down.

These were risky moves. If at any time we would have been found to be out of compliance with the boss's demands, it would have cost us an instant 5,000 marks against us. In the real world, moves like this could potentially cost you your job. Of course, in the real world, you can often talk the boss out of making bad decisions, and the boss would likely have to provide a good business reason to implement the features that were being asked of us.

Things were going more smoothly on day two. We hooked up a switch to the DMZ to give our personal laptops some visibility to the front-end of our network. This helped things immensely, and it's something I'd do immediately on the first day of an event like this in the future. I brought speakers, too, so that my team had some tunes. We rocked out to 808 State, Paul Van Dyk, Daft Punk, Juno Reactor, Steve Porter, Orbital, White Zombie and more. At a reasonable volume, of course.

With the exception of the badly configured BIND server, our team and Team 3 were holding steady on marks against us. We both started to identify some SQL injection and default usernames being exploited on our networks. Team 3 had gained on us a lot at the end of Day One, but we held a margin of about 200 points ahead of their team for the entirety of Day Two.

More demands rolled in. We had 30 minutes to open Exchange's SMTP to the world, and we had to add a new zone with MX, A and NS records to the active directory's DNS server. Our Windows admins aced it.

While that was going on, I was digging into BIND on the redhat box. I was the first blue-teamer to finally dig into the issue and fix it. Only one other team was able to resolve this issue, and it took them a while after I had fixed my team's. I'll spare you the details, but it was really, really misconfigured.

At this point, several attackers had some kind of hooks into a few of our systems, but they were having trouble scoring phone-homes. With all our tasks out of the way, we went into incident response mode, booting and cleaning up after red teamers.

One last PHB demand for the day: Set up VoIP trunks and extensions on our Asterisk server, and allow the protocols needed to talk between networks. We weren't going to actually communicate over it, but we needed to at least show that we could get into the interface and configure Asterisk with FreePBX. Since my machine was plugged into the external switch and since no one on my team (even me) had any Asterisk experience that would make them any better for the job, I went ahead and took the charge on that one. It required collaborating with my team-mates to grok the config files and tweak the databases so I could get logged in.

There was about half-an-hour of incident response left after that. About 10 minutes before scoring closed, the game organizers joked that he'd remove 10,000 marks from any team who could provide him with full packet captures of the entire exercise from Solera by the time scoring closed. I say he was joking, because that VM had only 2GB of hard drive space on it, so there's no way it could have stored it all. Even if it could have stored it, I doubt anyone could have exported all that data in 10 minutes.

All of the tasks required collaboration, and the entire exercise calls for a team of people with diverse skills.

blog comments powered by Disqus