Webresolve

#!/bin/sh # If the file oldnews.shtml is present, it will contain a list of # all the release notes. In that case we only need to list the most # recent one here. oldnews() { f=`echo $news|cut -f 1 -d ' '` s=`basename $f .shtml | cut -c 6-` echo "News in $s" echo "

" echo "Old news" echo "

" exit } test -n "$QUERY_STRING" || exit dir=/var/www/vhosts/"$QUERY_STRING" cd "$dir" || exit echo "Content-type: text/html" echo news=`ls -t news-*.shtml` c1="#99ccff" c2="#ffffff" bgcolor="$c1" test -f oldnews.shtml && oldnews echo "

" for f in $news; do s=`basename $f .shtml | cut -c 6-` echo "" if test "$bgcolor" = "$c1"; then bgcolor="$c2" else bgcolor="$c1" fi done echo "

" echo "News in $s" echo "

This program is used in conjunction with Webalizer to get host names rather than IP addresses in the statistics. Preprocess the log files with webresolve to replace IP addresses with host names. The program reads from stdin and writes to stdout:

	webresolve < access_log > access_log.resolved

I wrote the program to process access log files from Apache before they are analyzed by Webalizer. Recent versions of Webalizer has semi-working resolution built in, but not on all platforms, including the one siag.nu runs on.

Each line is one record consisting of whitespace-separated fields. If the first field starts with a digit, we assume that it is an IP address which we try to look up. The rest of the line is ignored. The result of the lookup is cached and the line written back with the first field replaced by the host name, if lookup was successful.

This program is pretty slow. Thanks to the caching, repeated visits from the same address will only result in one lookup. Therefore large files will be processed proportionally faster than small ones.

The load from this program is very light, because it spends most of its time waiting for the resolver. This also means that large files can take quite some time to process. The solution is to split the log file and run several resolution processes in parallel. This is done by the script splitwr:

        splitwr logfile > logfile.resolved
        webalizer logfile.resolved
        rm logfile.resolved

By default, splitwr runs 20 parallel resolution processes. The number can be changed by editing the script.

Installation

No configuration script for this program, just type:

        make
        make install

By default the programs are installed in /usr/local/bin. This can be changed like this example:

        make install PREFIX=/usr

to install into /usr/bin instead.

Download source

Freshmeat project page

Mirror page

More stuff

#!/bin/sh echo "Content-type: text/html" echo echo ""

Ulric Eriksson - July 2002 - ulric@siag.nu