Tracing processes with a laugh

I saw this on XKCD yesterday and had a good chuckle:

Of course, If I ran into something that was segfaulting, I'd pick up the pieces, and break out the tracer. If you've got a daemon or program that keeps segfaulting for no known reason, tracing is a great place to start.

In these examples, I'll just trace a quick ls command. In our case, ls doesn't have any problems, but the trace will contain all of the system calls that were executed. If you can replicate problems or crashes while tracing, you can spot where they're happening to report the problem to the developer or vendor of the application. Or, you can go back and double-check your damn pointers, human -- lest the computer eat your comp-sci homework.


Solaris
Truss is a command on Solaris that dumps all of the syscalls for a process. In its most basic form, you launch truss around the program you're going to troubleshoot. The below command-line takes truss' output and puts it in ls.truss.out before running ls normally, listing the files.

$ truss -o ls.truss.out ls

chuser.sh find.truss.out ls.truss.out megascan.sh test.pl

Or you can use truss to get system calls from a running process. For daemons you should launch truss as root or with sudo. -o sshd.truss.out tells it to write the data to sshd.truss.out, whereas -p 3088 tells truss to attach to process ID 3088, the made-up PID for our made-up instance of the ssh daemon.

# truss -o sshd.truss.out -p 3088

You can view ls.truss.out to see what it found.

BSD
A little more complicated, you can use ktrace to do something similar. By default, ktrace creates a (non-human-readable) file called ktrace.out. You can specify the output file with -f.

$ ktrace -f ls.ktrace.out ls
ls.ktrace.out pkgscripts obsd_pkgscripts-1.00.tar.gz static.key

Similarly, with the -p option, ktrace accepts a pid:

# ktrace -f sshd.ktrace.out -p 3088

Then, the fun begins. You have to use kdump to read the syscalls from the file.

$ kdump -f ls.ktrace.out > ls.ktrace.txt

Have a look at the results, if you wish.