Re: Tomcat and penlog

I found a solution to my logging problems anyway.

It seems that getting penlogd to merge the pen and web logs is pretty
redundant as long as you enable the X-Forwarded-For header in pen and
then get your webserver to see the header and rewrite the logs itself. I
managed to do that in Tomcat with the following code:-

<Valve className="org.apache.catalina.valves.AccessLogValve"
 directory="logs" prefix="fifo" rotatable="false"
 pattern='%{X-Forwarded-For}i %l %u %t "%r" %s %b'
 resolveHosts="false"/>

This works fine of HTTP... but HTTPS still has its old problems. pen is
unable to insert X-Forwarded-For headers into encrypted SSL that goes
over it, and penlogd cannot match the encrypted content pen gets with
the unencrypted content the webserver gets.
Anyone have a solution for this?

As for reliable piped log in tomcat... well, I could have used the "tail
-f" kludge suggested in the howto, but I decided to create my own kludge
in the form of a fifo and a perl script.
I did it this way because I didn't want to tail a file which would have
to eventually rollover (or fill up the disk entirely) and leave me
hanging. The end result is something that will work for anything else
other than Tomcat too.

Instead I made a fifo in the tomcat log directory ("mkfifo fifo" in
FreeBSD) and set up Tomcat to log to that (using the above configuration
example), with log rotation off because the fifo is a log blackhole.

Then I wrote a script and start script to get these logs from the fifo
and push them onto penlogd.

------------------------------

#!/bin/sh
# Repilog start script

pid=`ps -ax | grep repilog | grep -v grep | grep -v repilog.sh | awk '{print $1}'`

case "$1" in
       start)
               if [ $pid ]
               then
               else
                       [ -x /usr/local/sbin/repilog ] && /usr/local/sbin/repilog &
                       echo "REPILOG started!"
               fi
               ;;
       stop)
               if [ $pid ]
               then
                       kill $pid && echo "REPILOG killed!"
               fi
               ;;
       *)
               echo ""
               echo "Usage: `basename $0` { start | stop }"
               echo ""
               exit 64
               ;;
esac

---------------------------------------------------

#!/usr/bin/perl

# REPILOG - REliable PIped LOGs Simulator
# By Nathan Butcher 2006/11/21
#
# A shim to allow a logging program to spit out logs to a fifo
# where they can be piped to another waiting program

# Please run me in the background, otherwise I will take over your console!
# or better yet, use the start script!

use strict;

### configurables
my $fifo="/usr/local/tomcat5.5/logs/fifo"; # Location of fifo
my $command='|/usr/local/bin/penlog 192.168.0.5 10000'; # Command
###

### check to see if fifo exists or not
unless ( -p $fifo) {
        die "Repilog has no fifo to use. Please make it.\n";
}

### Subroutine to handle interrupts and close cleanly
sub handler {
       close (LOG);
       close (FIFO);
       exit 0;
}

### Interrupt signal handling
$SIG{INT} = 'handler';
$SIG{KILL} = 'handler';
$SIG{HUP} = 'handler';

### Main (endless) loop
while (1) {
       open(FIFO, "$fifo") || die "Repilog can't access the fifo!\n";
       while (<FIFO>) {
               open(LOG, "$command") || die "Repilog cannot access command!\n";
               print LOG "$_";
               close (LOG);
       }
       close(FIFO);
       warn "Repilog pipe closed. Restarting\n";
}

-------------

Nathan Butcher wrote:
> Is anyone aware of a way to configure Tomcat to do "reliable piped
> logs" with penlog, similar to what can be done with Apache as shown in
> the penlog manpage?
>
> I'm trying to merge the penlogs with the Tomcat access logs, in order
> to run merged logs though a traffic analysis program. Not having much
> luck here.
>
Received on Fri Nov 24 2006 - 04:39:14 CET