Re: Pen goes extremely slow occasionally

On Tue, 4 Nov 2008, Ulric Eriksson wrote:

> The manpage is misleading, thanks for bringing that to my attention. The -n
> option makes pen "not nonblocking", i.e. nonblocking is the default. I
> recommend that you remove the -n option from the command line.

Thanks, that clears that up. :)

> ulimit -n

65536 for the user pen runs as.

> netstat | grep 389

I'm guessing you meant 'netstat -tnp' or something similar; I just
replicated the slowness -- which is now starting almost instantly, as
soon as we turn pen on and put requests throught it -- with only 119
connections. About 6-7 of those connections were in CLOSE_WAIT, which
is interesting as you'll read below.

> A tool such as tcpdump or even better wireshark (ethereal) can be helpful to
> figure out what's going on.

I did a wireshark capture between a client and pen on a request that
took about 33 seconds to complete. (Requests directly to the backends
during this time were taking 0.012 seconds on average, so this is
clearly not a backend issue.) The request went roughly like this, if
I'm reading the capture properly:

0 seconds: SYN, SYN-ACK, ACK completed in a timely fashion. Bind
request sent.
0.2 seconds: Bind request retransmitted
0.6 seconds: Bind request retransmitted, ACK sent
29.2 seconds (!): Bind response sent. Search request sent
31.2 seconds: Search result sent, unbind request sent, FIN/ACK sent
32.7 seconds: FIN/ACK from server finally sent

So, we waited 28.6 seconds for the bind, 2 seconds for the search, and
1.5 seconds in CLOSE_WAIT.

If I had to guess, I'd say that pen isn't having problems listening to
the client, but rather with hearing back from the LDAP server, but
that's just a guess.

I suppose the next step would be to capture the massive volume of data
between pen and the correct LDAP backend and try to wade through that,
yes? Any ideas before I go down that garden path?

Thanks again!

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University
Received on Tue Nov 04 2008 - 17:18:16 CET