Parallels between the Titanic and IT Security

Unbelievably awesome stuff by Guerilla CISO via Anton Chuvakin:

...the problem here was that the Titanic indeed did meet all of the safety requirements of the time. And that a big part of the problem was that the safety requirements were drafted in 1894 at a time when there were rapid changes and in the size and design of ships of this kind [...] the bottom-line was that when the Titanic was reviewed by the safety accountants, they took out their check-list and went over the ship with a fine tooth comb. When the day was done the ship fully met all the safety criteria and was certified as safe.

This is where I see the parallels between root causes of the Titanic disaster and the odd situation we find ourselves in today in terms of IT security. Security by checklist –especially out of date checklists—simply doesn’t work. Moreover, the entire mental framework that mixes up accounting practices and thoughts with security discipline and research is an utter failure. Audits only uncover the most egregious security failures.

Compliance is a double-edged sword. For many security teams, especially those who work in an organization without a CISO or CSO to help cut through red tape, actual enforcement of policy is hard to do. Many of us aren't given any "teeth" so we're restricted to a bunch of hand waving and hollering and tantrum-throwing whilst being unable to actually make things more secure. Compliance can be used to acquire funding and hopefully get some traction on enterprise security initiatives. Security needs more than compliance, though. This is a great (but overtly dramatic) example of that.

Audit Rant

I'm in the middle of our twice-annual SAS 70 audit.  Audits are pretty much a part of any InfoSec professional's life at one time or another.  Some of us act as the proxy for our organization, providing what's needed to the auditors.  Others of us are responsible for doing the auditing.  There are several jobs in InfoSec that don't really deal with Audits, but if you work in the industry long enough, you're likely to experience audits eventually.

Some have asked me what exactly an audit is.  Those who've been through a tax audit know that audits aren't fun.  There are many types of audits depending on the industry you're in.  Hospitals and Medical firms must comply to HIPAA.  Firms that handle payment card information must comply to PCI DSS, and if you work for a publicly traded company in the US, you can thank the turn-of-the-century financial scandals of Enron, Tyco, WorldCom and others for the Sarbanes-Oxley Act and the audits that ensued.  The list goes on and on, and there are audits to ensure proper controls are in place.

Being on the "receiving" end of the SAS 70 this time around, I can try to convey what happens.  Imagine if someone comes by and asks for your car's odometer readings at each fill-up, all of your phone bills and bank statements for the past six months.

Imagine after having turned that over, you get them back a while later, with a bunch of items highlighted.  The auditor wants to know where you drove on 20 specific tanks of gas or on specific days.  He wants to know what you purchased with 50 randomly-highlighted withdrawals from your bank statement, and wants to know the nature of a bunch of your phone calls.

Now, take that a step further, and widen the scope. I have to provide that kind of detail, but not about personal finances and the like, but about my entire enterprise.  The people, the systems, the procedures, the documentation authorizing certain actions to be done, and all that.  Yes, at this point, the "auditee" (that's me) is tasked with answering for pretty much every action that's taken place within their organization, regardless if they had anything to do with the actions taken or not.

I've been on the auditor end of things as well, but it's been quite a few years.  It comes with its own challenges.  That's another rant for another day... or does someone feel like doing a guest post with a counter-point to this article?

Sysadmin Sunday: Process Accounting

Introduction
lastlog, last, and the contents of the various text log files are a good start for backtracking on suspicious activity and performance issues, but what if you need more detail? Does the system seem to really slow down at certain times of the day? Do you think that call center plebe might be using his shell account for something other than that decade-old CRM tool? Has an account has been compromised and used to leverage more access to your systems? If so, what is the attacker trying to run? It's pretty hard to diagnose those problems with the traditional tools provided.

This week, I'll walk you through using the basic process accounting tools that are available for many UNIX-Like operating systems out there. Process accounting affords the system administrator a totalitarian view of what exactly users are up to -- or, more importantly, what they have been up to!

These tools are included by default on Solaris as well as all BSD systems (including OpenBSD, FreeBSD and even the BSD-derived Mac OS X), and might come standard with some Linux distributions. It's never enabled by default, however. This is partially because the amount of data generated can be quite bulky. I recommend adding the process accounting log file to the log rotating scheme (which varies from flavor to flavor and is beyond the scope of this article)

As usual, green fixed-width text is in a user-level shell, red fixed-width text represents commands run as root (or with sudo).

Enabling Accounting
Once you properly configure and enable accounting, you will begin seeing the logs work their magic.

Debian
On Debian-based Linux distributions such as Ubuntu, you'll have to install the acct package. Once you install it, the accounting will enable by itself, without any further action needed by you.

# apt-get install acct

Red Hat
On Red-Hat based Linux distributions such as (well, duh) Red Hat Enterprise Linux or CentOS, you may need to manually enable it after installing. The package is known as psacct on most RH based Linux distros. The exception that I know of is SuSE, which still calls it acct. On SuSE, I believe you may replace "psacct" below with "acct" and it should work.

# yum install psacct
# chkconfig psacct on
# /etc/init.d/psacct start

BSD
In BSD, you need to first create and set permissions on the file used for accounting information and manually start it:

# touch /var/account/acct
# accton /var/account/acct

To enable accounting by default in FreeBSD, put the following line in /etc/rc.conf:

accounting_enable="YES"

For OpenBSD, add the following line to /etc/rc.conf:

accounting=YES

After installing process accounting packages, I highly recommend finding the nearest opportunity to reboot the system and test to ensure that accounting starts after reboot.

Using the reporting tools
There are 3 main commands used by the process accounting system.

lastcomm (report on commands that have been run by users)
sa (report on system accounting logs)
ac (connect time accounting)

The most obvious of these commands is lastcomm. If you are suspicious of a user's activity, or are simply bored and want to see what's been running recently, lastcomm is the hot ticket. by default, it shows the last commands that have been run at the top of the list and works its way back through time. It's almost too granular at times, so your mileage may vary. Here's a look at lastcomm's output while root is in the middle of installing something with pkg_add on OpenBSD:

$ lastcomm | less
lastcomm   -X      axon     ttyp2      0.03 secs Sat Dec  1 13:01 (0:03:28.75)
less       -       axon     ttyp2      0.05 secs Sat Dec  1 13:01 (0:03:28.75)
arch       -S      root     ttyp1      0.03 secs Sat Dec  1 13:02 (0:00:00.02)
gzip       -SX     root     ttyp1      0.34 secs Sat Dec  1 13:01 (0:00:19.42)
ftp        -S      root     ttyp1      1.08 secs Sat Dec  1 13:01 (0:00:19.42)
arch       -S      root     ttyp1      0.03 secs Sat Dec  1 13:01 (0:00:00.03)
gzip       -S      root     ttyp1      0.12 secs Sat Dec  1 13:01 (0:00:13.02)
ftp        -S      root     ttyp1      0.34 secs Sat Dec  1 13:01 (0:00:12.97)
arch       -S      root     ttyp1      0.02 secs Sat Dec  1 13:01 (0:00:00.02)
gzip       -S      root     ttyp1      0.08 secs Sat Dec  1 13:01 (0:00:06.25)
ftp        -S      root     ttyp1      0.11 secs Sat Dec  1 13:01 (0:00:06.25)
arch       -S      root     ttyp1      0.03 secs Sat Dec  1 13:01 (0:00:00.03)
less       -       axon     ttyp2      0.03 secs Sat Dec  1 13:01 (0:00:20.05)
...

Lastcomm's command-line options allow you to manually filter things out by command, user, or terminal as well. I truncated output to 5 lines in the following examples:

$ lastcomm user axon 
xterm      -       axon   __         0.23 secs Sat Dec  1 13:11 (0:00:26.91)
bash       -X      axon   ttyp3      0.06 secs Sat Dec  1 13:11 (0:00:26.19)
WPrefs     -       axon   __         0.36 secs Sat Dec  1 13:11 (0:00:18.25)
lastcomm   -       axon   ttyp2      0.03 secs Sat Dec  1 13:11 (0:00:00.06)
head       -       axon   ttyp2      0.03 secs Sat Dec  1 13:11 (0:00:00.06)

$ lastcomm command sendmail
sendmail   -F      smmsp  __         0.02 secs Sat Dec  1 13:00 (0:00:01.03)
sendmail   -S      smmsp  __         0.11 secs Sat Dec  1 13:00 (0:00:01.12)
sendmail   -SF     root   __         0.03 secs Sat Dec  1 12:46 (0:00:01.03)
sendmail   -F      smmsp  __         0.00 secs Sat Dec  1 12:30 (0:00:01.03)
sendmail   -S      smmsp  __         0.11 secs Sat Dec  1 12:30 (0:00:01.11)

$ lastcomm terminal ttyC0 
xdm        -S      root   ttyC0      0.08 secs Sat Dec  1 11:16 (0:00:00.36)
sh         -S      root   ttyC0      0.25 secs Sat Dec  1 11:15 (0:00:10.58)
date       -S      root   ttyC0      0.02 secs Sat Dec  1 11:16 (0:00:00.08)
ifconfig   -S      root   ttyC0      0.00 secs Sat Dec  1 11:16 (0:00:00.00)
cron       -S      root   ttyC0      0.02 secs Sat Dec  1 11:16 (0:00:00.05)

sa is a very, very powerful tool with quite a few options. Check out the sa(8) man page to get a feel for all of the capabilities. Some useful highlights are as follows.

Get a system utilization stats for each user:

$ sa -m      
root           131         0.43cpu         9168tio           0k*sec
smmsp            9         0.01cpu          165tio           0k*sec
sshd             1         0.00cpu            4tio           0k*sec
_identd          1         0.00cpu            1tio           0k*sec
_x11             3         0.00cpu           67tio           0k*sec
axon           182         0.11cpu          721tio           0k*sec

Show the processes with the most CPU Time:

$ sa -t
     360      143.90re        0.57cp   252.09re/cp       29avio        0k   
      18       13.19re        0.16cp    84.11re/cp        5avio        0k   ftp
       4       13.43re        0.12cp   107.85re/cp     1788avio        0k   perl
      29        3.72re        0.08cp    47.93re/cp       67avio        0k   ***other
      16       13.04re        0.06cp   231.90re/cp        0avio        0k   gzip
      45        4.08re        0.03cp   140.01re/cp        2avio        0k   lastcomm
      36        0.16re        0.02cp     7.54re/cp        7avio        0k   sa
      20        1.92re        0.02cp   120.72re/cp        0avio        0k   sed
       3       58.33re        0.01cp  3929.74re/cp       18avio        0k   ssh
       7        0.15re        0.01cp    10.27re/cp       25avio        0k   sendmail
      38       18.59re        0.01cp  1552.26re/cp        3avio        0k   sh
...

Finally, ac shows how long each user has been connected to the system. This uses the normal wtmp log (not the process accounting log) just like the "last" command. By default, it shows the connect time for all users totalled up. The -d argument gives day-to-day totals, and -p displays the totals for each user. These stats aren't very exciting since I just recently enabled it for the purpose of documenting it:

$ ac
total                               13.51

$ ac -d
Nov 30  total        9.73
Dec  1  total        3.78

$ ac -p
axon                                13.51
total                               13.51

Locking it down
It's worth mentioning that if you're on a system that's not yours to administer (for instance, as a user) with process accounting enabled, there's also a good chance that you can gather potentially sensitive information about the system with these tools. The same goes for all the users on your systems. While the information stored is generally benign, there's not much use in leaving them out in the open. Note that in all of the above report examples, I was using my user level account. I highly recommend setting mode 600 on the accounting files and the directory they're in, and making sure they're owned by the root user. This should keep prying eyes out of the accounting logs. In the example below, I use /var/account which is the default for BSD. It varies from one distribution to the next. For example, on Ubuntu, the accounting logs are stored in /var/log/account.

# chmod -R 600 /var/account                                                                      
# chown -R root /var/account

Now, I must be root to run sa and lastcomm. They don't work with my user-level account.

$ sa
sa: retrieving process accounting summary: Permission denied
sa: process accounting initialization failed

$ lastcomm
lastcomm: /var/account/acct: Permission denied