2011-11-18

Shell Script: Parse Juniper firewall logs

Juniper firewalls (at least the ScreenOS-based one I have in the lab) have an interesting format for their syslog entries. It's a whole line full of variable=parameter type stuff. Usually, these are in a pretty predictable order, but you can't rely on the nth parameter to be the same in every log entry just due to the fact that different types of traffic have different parameters. You don't see a source or destination port on ICMP traffic, for example.  This script reads a log file (I'm using Syslog-NG in the lab), splits the entry up into one parameter per line, grabs only parameters matching the list you provide on the command line, then stitches it back together into one line. I saw no benefit in trying to make the output order match the order provided on the command-line, since some columns are prone to be blank in certain situations anyway. This is quick and dirty, but it does the job for what I need, and I thought I'd share.

axon@moo:~$ ./juniper-parse.sh

Parse columns from Juniper firewall logs
  Syntax:
  ./juniper-parse.sh logfile column [column ...]

Common columns:
  action, device_id, dst, dst_port, duration, ip,
  policy_id, port, proto, rcvd, reason, sent, service,
  session_id, src, src_port, start_time, zone


Here, you can see some of my lab machines trying to fetch updates from an Ubuntu mirror, getting denied because I only allow requests through my proxy server:

axon@moo:~$ sudo ./juniper-parse.sh /var/log/firewall.log action policy_id proto src dst dst_port

2011-11-13T08:35:09-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.75 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:09-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.112 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:10-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.171 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:13-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.171 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:15-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.77 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:18-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.75 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:18-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.77 dst=91.189.92.169 dst_port=80
...

Script below:

#!/bin/sh
#juniper-parse.sh
if [ $# -lt "2" ]
then
echo "
Parse columns from Juniper firewall logs
  Syntax:
  $0 logfile column [column ...]

Common columns:
  action, device_id, dst, dst_port, duration, ip,
  policy_id, port, proto, rcvd, reason, sent, service,
  session_id, src, src_port, start_time, zone

"
exit 1
fi

filename=$1; shift
until [ $# = "0" ]
do
  pattern="$pattern -e ^$1="
  shift
done
while read line
do
  timestamp=`echo $line | cut -f1 -d" "`
  echo "$timestamp `echo $line | tr ' ' '\n' | grep $pattern | tr '\n' ' '`"
done < $filename

2011-11-12

OpenBSD 5.0: Apache, PHP and MySQL

The basic steps for taking a bare-bones install of OpenBSD and adding an AMP stack for web applications hadn't changed much in the past 2 years. Although 5.0 is not seen as a "major update", but simply a continuation of the normal development cycle, the OpenBSD team did a few things that make the installation a bit different this go around. I'll cover some of the recent changes to OpenBSD and its packages in this post, then you can dive right into the updated OpenBSD Chroot Apache, PHP, MySQL setup guide.

Initialization scripts: Starting with OpenBSD 4.9, the RC scripts became modular. This is similar to how NetBSD and FreeBSD have worked for many years. Individual daemons have startup scripts in /etc/rc.d, but rc.conf, rc.local and rc.conf.local still work the same, so it wouldn't surprise me if many OpenBSD users didn't even notice the change. It seems like OpenBSD 5.0 packages for most of the popular services (samba, cups, postgres, etc) are actually creating these startup scripts now. If a third-party application from packages installs a startup script in /etc/rc.d, you can add it to the pkg_scripts variable in /etc/rc.conf.local like so, and it'll magically start.

pkg_scripts="mysqld cups samba"

See the OpenBSD man page for rc.d for more details.

MySQL and chroot: I'm not sure why it didn't dawn on me before, but one really simple way around hard-linking MySQL's socket file into the chroot environment is to simply have your web applications connect to MySQL's TCP port on 127.0.0.1 (not localhost, because that means "use the socket file" in MySQL-ese) I think I'll be doing it this way in the future.

PHP: PHP 5.3 added a lot of features, such that PHP 5.2.x and PHP 5.3 might as well be different major version numbers when complex web applications are considered. OpenBSD 5.0 adds packages for PHP 5.3 for the first time, but PHP 5.2 is still available in the repository. For this reason, the PHP core and module packages are no longer prefixed by php5-, they're just php, php-mysql, etc. Since PHP 5.2 is still supported, the version numbers are used to distinguish them, like this:

$ sudo pkg_add php-mysql
Ambiguous: php-mysql could be php-mysql-5.2.17p3 php-mysql-5.3.6

Further, some configuration file locations have been changed. If you pay attention to the text after installing packages, this shouldn't make much difference. Hint: you can always re-read the post-install notes afterward by using pkg_info -M.

All in all, the recent changes to OpenBSD are welcome. Some of the things I outlined here are areas where OpenBSD had some catching up to do. They've managed to do a lot of proactive stuff in the realm of hardware support and (of course) security and encryption. This is still one of my favorite platforms to tinker with.