2011-11-18

Shell Script: Parse Juniper firewall logs

Juniper firewalls (at least the ScreenOS-based one I have in the lab) have an interesting format for their syslog entries. It's a whole line full of variable=parameter type stuff. Usually, these are in a pretty predictable order, but you can't rely on the nth parameter to be the same in every log entry just due to the fact that different types of traffic have different parameters. You don't see a source or destination port on ICMP traffic, for example.  This script reads a log file (I'm using Syslog-NG in the lab), splits the entry up into one parameter per line, grabs only parameters matching the list you provide on the command line, then stitches it back together into one line. I saw no benefit in trying to make the output order match the order provided on the command-line, since some columns are prone to be blank in certain situations anyway. This is quick and dirty, but it does the job for what I need, and I thought I'd share.

axon@moo:~$ ./juniper-parse.sh

Parse columns from Juniper firewall logs
  Syntax:
  ./juniper-parse.sh logfile column [column ...]

Common columns:
  action, device_id, dst, dst_port, duration, ip,
  policy_id, port, proto, rcvd, reason, sent, service,
  session_id, src, src_port, start_time, zone


Here, you can see some of my lab machines trying to fetch updates from an Ubuntu mirror, getting denied because I only allow requests through my proxy server:

axon@moo:~$ sudo ./juniper-parse.sh /var/log/firewall.log action policy_id proto src dst dst_port

2011-11-13T08:35:09-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.75 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:09-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.112 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:10-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.171 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:13-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.171 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:15-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.77 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:18-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.75 dst=91.189.92.169 dst_port=80
2011-11-13T08:35:18-06:00 policy_id=12 proto=6 action=Deny src=192.168.42.77 dst=91.189.92.169 dst_port=80
...

Script below:

#!/bin/sh
#juniper-parse.sh
if [ $# -lt "2" ]
then
echo "
Parse columns from Juniper firewall logs
  Syntax:
  $0 logfile column [column ...]

Common columns:
  action, device_id, dst, dst_port, duration, ip,
  policy_id, port, proto, rcvd, reason, sent, service,
  session_id, src, src_port, start_time, zone

"
exit 1
fi

filename=$1; shift
until [ $# = "0" ]
do
  pattern="$pattern -e ^$1="
  shift
done
while read line
do
  timestamp=`echo $line | cut -f1 -d" "`
  echo "$timestamp `echo $line | tr ' ' '\n' | grep $pattern | tr '\n' ' '`"
done < $filename

blog comments powered by Disqus