Sysadmin Sunday: Process Management

Managing processes in UNIX operating systems is a critical function. This goes beyond just killing runaway processes, though. This week in Sysadmin Sunday, we'll take a look at managing processes to make your life a little easier.

When you're in a shell, there are 3 kinds of processes: foreground, background, and stopped (sometimes called suspended). This is pretty straight-forward.

Shelling out of a running program

If you're in the middle of a really long-running program and don't want to or cannot switch consoles, you can normally shell out of that that process with the (Ctrl-Z) key combination. For example, encrypting a huge file with GNU Privacy Guard:

$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
(moments pass...)
^Z (Ctrl-Z)
[1]+ Stopped gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
Pressing (Ctrl-Z) gives you a shell but puts the running process in a stopped state. From there, you can run something else, such as vi. Just remember that the process that you escaped from is stopped and won't make any progress at this point. That also means it shouldn't display any output that might interrupt you if you decide to work on something else while that process is stopped. It's more "paused" than "stopped". Take note of the job ID [in brackets] of the stopped process. That will come in handy later on.

See what jobs your current shell has
I've taken the liberty of running and shelling out of vi as well for this example. Keep in mind that GPG is still sitting there doing nothing, too.

To see what's running in your current shell, use the jobs tool.
$ jobs
[1]- Stopped gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
[2]+ Stopped vi /etc/resolv.conf
Here, you can see both gpg and vi are stopped. Again, the job ID for each process shows up in brackets. The + shows which of the jobs is the most recent one you have dealt with and will be used as the default job later on. Next, the status of the job is shown, followed by the actual command-line (or as much of it as can be displayed if it's a particularly long command line).

Switching between processes
We take a lot of things for granted, like being able to tab between 20 different windows on a GUI terminal. Not to be a curmudgeonly olde-school sysadmin, but back in the day, we didn't have that luxury. A lot of my early interactions with UNIX were over a serial terminal or a dial-up connection. Virtual consoles were not common, so switching between jobs was just part of working with UNIX on a daily basis for many people.

To bring a stopped process to the foreground again, use the fg command. If you just type fg, it will bring back the most recent job, the one marked with a + when you run the jobs command. If you wish to tell it to bring back a specific command, use jobs %2, or replace 2 with the job number you wish to bring to the foreground. You could have two different files open with two different instances of vi for example, and use ^Z, jobs, and fg to switch between them. This, of course, can get confusing, so be careful when multi-tasking this way.
$ fg %2
(vi opens back up)

Background processes
For programs that you don't expect much output from (such as gpg) you can simply background the job. Instead of fg, you use the bg command. Again bg %1 would work to background a specific job number. A background job is much like a stopped one, except it actually goes on about its business. If there is output, it will display the output onto your screen regardless what you're doing. Otherwise, it will silently chug away out of sight. When the process finishes, you'll be notified that it has completed. I'll background the gpg job to show how this works:
$ bg %1
[1]+ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
[1]+ Done gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz

Alternatively, if you want to run something that you know will take a while and you're certain you won't have to actually interact with it, you can launch it in the background by placing an ampersand (&) at the end of the command line. You'll immediately get the job ID and process ID, then you'll get your shell back. When it's done running, you'll be notified.
$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
[1] 1567
[1]+ Done gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz

What if you log off with stuff still running?
It really boils down to what's running and how it handles the various interrupt signals. As a general rule, any child processes under the shell get sent a HUP (HangUP) signal which is analogous to kill -1 or kill -HUP . This usually terminates the processes so that they're not out there hanging around forever. If you log off, there's a good chance that whatever you were running will stop.

If you really want the things to stay running, however, there are a few viable options. The first one good if you know you may need to log off while something is still running. It's the nohup command. Put it before your usual commnd and background it either with an ampersand at the end or by (Ctrl-Z) and bg. When you log off, the process should stay running because nohup doesn't pass the signal along to the child process. This is great for huge software compiles or other time-consuming, non-interactive tasks. Furthermore, nohup will re-direct all the output to a file (by default, nohup.out) so that you can go back and look at it later. This has the additional benefit of not barfing stuff all over your screen while you may be trying to get work done.

$ nohup nmap localhost&
[1] 1612
$ appending output to nohup.out
$ exit

At this point, you can log off the system and nmap will continue running. You can log in later to see the results of the nmap by checking the nohup.out file:
$ cat nohup.out

Starting Nmap 4.20 ( http://insecure.org ) at 2008-02-02 03:55 CST
Interesting ports on localhost (
Not shown: 1690 closed ports
22/tcp open ssh
88/tcp open kerberos-sec
139/tcp open netbios-ssn
445/tcp open microsoft-ds
548/tcp open afpovertcp
631/tcp open ipp
3689/tcp open rendezvous

Nmap finished: 1 IP address (1 host up) scanned in 15.157 seconds

But what if you're in the middle of something that you didn't expect to take so long, for example, compiling Mozilla Firefox from source? You have to log off, but want it to keep running. You don't want to stop it, then restart it with nohup. Not all hope is lost. The second method is to simply tell the shell not to send the HUP signal to any of the processes. Make sure only the things you want to remain running in the background are properly backgrounded, then on the command-line, type set NOHUP, then log off. Unlike the nohup command, output is not re-directed, so you won't be able to see any of the output that happens while you're logged off. For things like software compiling or moving large amounts of data, this isn't much of a problem.
$ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
[1]+ Stopped gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz
$ bg
[1]+ gpg -r ax0n@h-i-r.net --encrypt-file axon.home.tgz &
$ set NOHUP
$ exit

The process will finish normally.

There are other options that are more modern than using the jobs, bg, and fg commands. These are beyond the scope of this article, however my favorite among these options is the screen command, which gives you virtual consoles that persist even after you log out. Screen comes with many Linux distributions, but it's still a little-known utility except for the fairly hard-core command-line junkies. This allows you to leave things like text-mode IRC clients or other interactive programs running at all times, even if you're not actually logged in.

blog comments powered by Disqus