Re: Pen goes extremely slow occasionally

On Mon, 3 Nov 2008, Chris St. Pierre wrote:

> We've been using pen to load balance among our four LDAP servers for
> several months now, and are quite pleased with it. Last week, quite
> suddenly, pen would occasionally (and suddenly) start taking _ages_ to
> answer requests. Sometimes restarting it would solve the problem;
> some times it wouldn't. At first we thought the problem was the
> number of connections to pen, but we've seen this happen with no
> connections at all.
>
> This is our pen command line:
>
> pen -u pen -l /var/log/pen/ldap.log -p /var/run/pen/ldap.pid \
> -n -C 127.0.0.1:10389 -x 2048 -d 10.1.1.57:389 chico groucho harpo zeppo
>
> Strangely enough, despite the fact that I've given it the -n flag, it
> doesn't appear to be in no-block mode:
>
> # penctl 127.0.0.1:10389 mode
> block no delayed_forward no hash no roundrobin no stubborn no weight no prio

The manpage is misleading, thanks for bringing that to my
attention. The -n option makes pen "not nonblocking", i.e.
nonblocking is the default. I recommend that you remove the -n
option from the command line.

The symptoms you describe indicate that pen is slow to accept the
new connection. That is usually caused by having large numbers of
simultaneous connections. Pen will then let new connections wait
until a "slot" becomes available. You say you have already ruled
out that possibility, which would indicate that some other
resource is exhausted; possibly in the operating system.

"Have you tried turning it off and on again?" ;)

Note: pen doesn't do that. We've had it running for years and
years without restarting or rebooting anything.

A few other things to check:

ulimit -n

netstat | grep 389

A tool such as tcpdump or even better wireshark (ethereal) can be
helpful to figure out what's going on.

Ulric
Received on Tue Nov 04 2008 - 09:48:25 CET