RE: PEN - support for thousands of connections

Thank you so very much!

-----Original Message-----
From: owner-pen_at_siag.nu [mailto:owner-pen_at_siag.nu]On Behalf Of
'ulric_at_siag.nu'
Sent: Sunday, March 09, 2008 4:19 PM
To: Crigger, Clint
Cc: 'pen_at_siag.nu'
Subject: RE: PEN - support for thousands of connections

Interesting test. Pen performance is nowhere near as well known
on Windows as it is on Linux.

It is expected that Pen will eat all of one CPU and none of the
other, since Pen is single threaded.

Regarding compiling with cygwin, there isn't much to learn once
the installation is done. The usual Gnu toolchain is there and
works pretty much the same.

I have uploaded pen-0.17.2.c5000.exe to http://siag.nu/pub/pen/.
It is built for 5000 connections; however, I have no way to test
that.

Ulric

---
On Thu, 6 Mar 2008, Crigger, Clint wrote:
> Ulric,
> 	Many thanks for your assistance and helpful response.
>
> We have been performing some detailed load and throughput testing up to
500
> concurrent connections.  If you would like I could probably share that
> information with you in the near future.
>
> We have observed plateaus of 50% sustained cpu loading while performing
> query operations via PEN against two LDAP servers (load balancing /
failover
> config).  Pen is running on a Win2k server with 2ea 3.1 ghz Intel
> Processors.  Essentially pen is consuming one cpu during the query load.
>
> Here is the test scenario:
> 	We add connections in 50 connection increments every 5 minutes up to
> the test max.  Each connections is closed, reopened, authenticated, and
used
> to perform ldap queries every 10 minutes.
>
> Each 50 connections is created by a seperate worker process that cycles
> through iterations of establishing connections, performing a set number of
> queries per connection, wait (up to 600 seconds) then close those
> connections, and start a new iteration.  (Note:  25% of the worker
processes
> iterate in 150 second intervals).
>
>     The worker processes are instantiated in a 300 second ramp-up,
> therefore the load is stair-stepped every 5 minutes in 50 connection
> increments until the max connection cound of 500 is reached.
>
> Our observations indicate that at the beginning of the test and at the end
> when larger numbers of new connections are being established, or existing
> connections are being torn down, query performance in terms of time taken
is
> quite slow (see attached charts).
>
>
> To be honest, I am a bit of a 'babe in the woods' regarding compiling C in
> Win32 / cygwin, and am limited in being able to obtain a complete cygwin
> install by organizational policy.
>
> Is there a tutorial (for noobs) regarding prerequisites to compiling on
> cygwin that you could point me to,  or is there a win32 binary that can
> handle 5k connections available?
>
> Much thanks,
> Clint
>
>
>
> -----Original Message-----
> From: 'ulric_at_siag.nu' [mailto:ulric_at_siag.nu]
> Sent: Thursday, January 17, 2008 2:39 AM
> To: Crigger, Clint
> Cc: 'pen_at_saig.nu'
> Subject: Re: PEN - support for thousands of connections
>
>
>
> On Wed, 16 Jan 2008, Crigger, Clint wrote:
>
>> A.     How would one compile a windows binary (cygwin execution
> environment)
>> capable of supporting 5,000 simultanious connections?
>>         [  Yes 5k simultanious active connections ]
>
> Just like on any sane platform, i.e. all except Linux:
>
> ./configure --with-fd_setsize=10100
>
>> B.  Is such a request an inappropriate application of pen?
>
> Not necessarily. It depends on the traffic volume. I assume that
> the connections will be permanent.
>
> If a server errors out, the clients accessing that server will
> lose their connection. So the application will need a way to
> detect that and reconnect.
>
>> C.  The log output from pen contains part of the contents of
>> the reqeust, which includes the credentials used to bind to the
>> ldap server(s).  Is it possible retain logging, but not to log
>> request contents?
>
> That is easiest to accomplish by hacking the source, I think. In
> the function copy_up you'll find this line:
>
>                         log(logfp, i, b, rc);
>
> Replace it with this:
>
>                         log(logfp, i, b, "");
>
> and the log will only contain the client ip, timestamp, and
> server ip.
>
> Ulric
>
>