Introduction
lastlog, last, and the contents of the various text log files are a good start for backtracking on suspicious activity and performance issues, but what if you need more detail? Does the system seem to really slow down at certain times of the day? Do you think that call center plebe might be using his shell account for something other than that decade-old CRM tool? Has an account has been compromised and used to leverage more access to your systems? If so, what is the attacker trying to run? It's pretty hard to diagnose those problems with the traditional tools provided.
This week, I'll walk you through using the basic process accounting tools that are available for many UNIX-Like operating systems out there. Process accounting affords the system administrator a totalitarian view of what exactly users are up to -- or, more importantly, what they have been up to!
These tools are included by default on Solaris as well as all BSD systems (including OpenBSD, FreeBSD and even the BSD-derived Mac OS X), and might come standard with some Linux distributions. It's never enabled by default, however. This is partially because the amount of data generated can be quite bulky. I recommend adding the process accounting log file to the log rotating scheme (which varies from flavor to flavor and is beyond the scope of this article)
As usual, green fixed-width text is in a user-level shell, red fixed-width text represents commands run as root (or with sudo).
Enabling Accounting
Once you properly configure and enable accounting, you will begin seeing the logs work their magic.
Debian
On Debian-based Linux distributions such as Ubuntu, you'll have to install the acct package. Once you install it, the accounting will enable by itself, without any further action needed by you.
# apt-get install acct
Red Hat
On Red-Hat based Linux distributions such as (well, duh) Red Hat Enterprise Linux or CentOS, you may need to manually enable it after installing. The package is known as psacct on most RH based Linux distros. The exception that I know of is SuSE, which still calls it acct. On SuSE, I believe you may replace "psacct" below with "acct" and it should work.
# yum install psacct
# chkconfig psacct on
# /etc/init.d/psacct start
BSD
In BSD, you need to first create and set permissions on the file used for accounting information and manually start it:
# touch /var/account/acct
# accton /var/account/acct
To enable accounting by default in FreeBSD, put the following line in /etc/rc.conf:
accounting_enable="YES"
For OpenBSD, add the following line to /etc/rc.conf:
accounting=YES
After installing process accounting packages, I highly recommend finding the nearest opportunity to reboot the system and test to ensure that accounting starts after reboot.
Using the reporting tools
There are 3 main commands used by the process accounting system.
lastcomm (report on commands that have been run by users)
sa (report on system accounting logs)
ac (connect time accounting)
The most obvious of these commands is lastcomm. If you are suspicious of a user's activity, or are simply bored and want to see what's been running recently, lastcomm is the hot ticket. by default, it shows the last commands that have been run at the top of the list and works its way back through time. It's almost too granular at times, so your mileage may vary. Here's a look at lastcomm's output while root is in the middle of installing something with pkg_add on OpenBSD:
Lastcomm's command-line options allow you to manually filter things out by command, user, or terminal as well. I truncated output to 5 lines in the following examples:$ lastcomm | less
lastcomm -X axon ttyp2 0.03 secs Sat Dec 1 13:01 (0:03:28.75)
less - axon ttyp2 0.05 secs Sat Dec 1 13:01 (0:03:28.75)
arch -S root ttyp1 0.03 secs Sat Dec 1 13:02 (0:00:00.02)
gzip -SX root ttyp1 0.34 secs Sat Dec 1 13:01 (0:00:19.42)
ftp -S root ttyp1 1.08 secs Sat Dec 1 13:01 (0:00:19.42)
arch -S root ttyp1 0.03 secs Sat Dec 1 13:01 (0:00:00.03)
gzip -S root ttyp1 0.12 secs Sat Dec 1 13:01 (0:00:13.02)
ftp -S root ttyp1 0.34 secs Sat Dec 1 13:01 (0:00:12.97)
arch -S root ttyp1 0.02 secs Sat Dec 1 13:01 (0:00:00.02)
gzip -S root ttyp1 0.08 secs Sat Dec 1 13:01 (0:00:06.25)
ftp -S root ttyp1 0.11 secs Sat Dec 1 13:01 (0:00:06.25)
arch -S root ttyp1 0.03 secs Sat Dec 1 13:01 (0:00:00.03)
less - axon ttyp2 0.03 secs Sat Dec 1 13:01 (0:00:20.05)
...
sa is a very, very powerful tool with quite a few options. Check out the sa(8) man page to get a feel for all of the capabilities. Some useful highlights are as follows.$ lastcomm user axon
xterm - axon __ 0.23 secs Sat Dec 1 13:11 (0:00:26.91)
bash -X axon ttyp3 0.06 secs Sat Dec 1 13:11 (0:00:26.19)
WPrefs - axon __ 0.36 secs Sat Dec 1 13:11 (0:00:18.25)
lastcomm - axon ttyp2 0.03 secs Sat Dec 1 13:11 (0:00:00.06)
head - axon ttyp2 0.03 secs Sat Dec 1 13:11 (0:00:00.06)
$ lastcomm command sendmail
sendmail -F smmsp __ 0.02 secs Sat Dec 1 13:00 (0:00:01.03)
sendmail -S smmsp __ 0.11 secs Sat Dec 1 13:00 (0:00:01.12)
sendmail -SF root __ 0.03 secs Sat Dec 1 12:46 (0:00:01.03)
sendmail -F smmsp __ 0.00 secs Sat Dec 1 12:30 (0:00:01.03)
sendmail -S smmsp __ 0.11 secs Sat Dec 1 12:30 (0:00:01.11)
$ lastcomm terminal ttyC0
xdm -S root ttyC0 0.08 secs Sat Dec 1 11:16 (0:00:00.36)
sh -S root ttyC0 0.25 secs Sat Dec 1 11:15 (0:00:10.58)
date -S root ttyC0 0.02 secs Sat Dec 1 11:16 (0:00:00.08)
ifconfig -S root ttyC0 0.00 secs Sat Dec 1 11:16 (0:00:00.00)
cron -S root ttyC0 0.02 secs Sat Dec 1 11:16 (0:00:00.05)
Get a system utilization stats for each user:
$ sa -m
root 131 0.43cpu 9168tio 0k*sec
smmsp 9 0.01cpu 165tio 0k*sec
sshd 1 0.00cpu 4tio 0k*sec
_identd 1 0.00cpu 1tio 0k*sec
_x11 3 0.00cpu 67tio 0k*sec
axon 182 0.11cpu 721tio 0k*sec
Show the processes with the most CPU Time:
$ sa -t
360 143.90re 0.57cp 252.09re/cp 29avio 0k
18 13.19re 0.16cp 84.11re/cp 5avio 0k ftp
4 13.43re 0.12cp 107.85re/cp 1788avio 0k perl
29 3.72re 0.08cp 47.93re/cp 67avio 0k ***other
16 13.04re 0.06cp 231.90re/cp 0avio 0k gzip
45 4.08re 0.03cp 140.01re/cp 2avio 0k lastcomm
36 0.16re 0.02cp 7.54re/cp 7avio 0k sa
20 1.92re 0.02cp 120.72re/cp 0avio 0k sed
3 58.33re 0.01cp 3929.74re/cp 18avio 0k ssh
7 0.15re 0.01cp 10.27re/cp 25avio 0k sendmail
38 18.59re 0.01cp 1552.26re/cp 3avio 0k sh
...
Finally, ac shows how long each user has been connected to the system. This uses the normal wtmp log (not the process accounting log) just like the "last" command. By default, it shows the connect time for all users totalled up. The -d argument gives day-to-day totals, and -p displays the totals for each user. These stats aren't very exciting since I just recently enabled it for the purpose of documenting it:
$ ac
total 13.51
$ ac -d
Nov 30 total 9.73
Dec 1 total 3.78
$ ac -p
axon 13.51
total 13.51
Locking it down
It's worth mentioning that if you're on a system that's not yours to administer (for instance, as a user) with process accounting enabled, there's also a good chance that you can gather potentially sensitive information about the system with these tools. The same goes for all the users on your systems. While the information stored is generally benign, there's not much use in leaving them out in the open. Note that in all of the above report examples, I was using my user level account. I highly recommend setting mode 600 on the accounting files and the directory they're in, and making sure they're owned by the root user. This should keep prying eyes out of the accounting logs. In the example below, I use /var/account which is the default for BSD. It varies from one distribution to the next. For example, on Ubuntu, the accounting logs are stored in /var/log/account.
Now, I must be root to run sa and lastcomm. They don't work with my user-level account.# chmod -R 600 /var/account
# chown -R root /var/account
$ sa
sa: retrieving process accounting summary: Permission denied
sa: process accounting initialization failed
$ lastcomm
lastcomm: /var/account/acct: Permission denied
Posts
