Re: Difference between "-c" and "-x"

On Fri, Mar 28, 2008 at 05:15:48PM +0100, Roberto Suarez Soto wrote:
> It would be so if using select(), but it seems that using poll() this limit
> is avoided. Unfortunately, the Debian packaged version of pen doesn't
> use "--with-poll" when compiling, so I couldn't use it. I've recompiled the
> package (also updating the upstream version in Debian, 0.15, to 0.17.2) and
> enabled it.
>
> Consider it a "beta" version, but if you're interested you can download it
> from here:
>
> http://public.allenta.com/pen/
>
> I hope my belief on poll() is true, but I'd be very thankful if someone could
> confirm that it should handle many more connections than select(). Thanks in
> advance.

It can handle more connections than select(), but that doesn't mean it will
scale particularly well.

Every time round pen's mainloop_poll(), it builds an array of struct
pollfd's from scratch. So with 5,000 concurrent connections it has to create
a 5,000 element array:

                /* add sockets from open connections */
                if (debuglevel) debug("filling pollfd structure");
                for (i = 0; i < connections_max; i++) {
                        if (conns[i].downfd == -1) continue;
                        upevents = downevents = 0;

                        if (conns[i].upn == 0) downevents |= POLLIN;
                        else upevents |= POLLOUT;

                        if (conns[i].downn == 0) upevents |= POLLIN;
                        else downevents |= POLLOUT;

                        if (downevents) {
                                ufds[nfds].fd = conns[i].downfd;
                                ufds[nfds++].events = downevents;
                        }
                        if (upevents) {
                                ufds[nfds].fd = conns[i].upfd;
                                ufds[nfds++].events = upevents;
                        }
                }

Then the kernel has to process this whole array every time that poll() is
called, then wait for data; and at that point only one or two sockets might
have data to transfer, in which case all this setup overhead is
proportionately very high.

It would be better if it only rebuilt the array when necessary, and it could
also use Linux's newer epoll() interface. I've heard that systems handling
tens of thousands of concurrent client connections have been built this way.

However, maybe a userland proxy like pen isn't ideal for such large numbers
of connections. I'd suggest either doing load-balanced DNAT in kernel space
(e.g. using OpenBSD/FreeBSD pf, or I believe iptables in Linux can do this
too), or for a business-critical application then buy a pair of hardware
load balancers (e.g. Foundry, Alteon, F5, or many others to choose from).
The latter can also do heartbeat monitoring, sticky sessions and SSL
offloading.

You could also consider taking action to reduce the number of concurrent
connections you need, in particular disabling persistent HTTP connections at
the back-end webservers.

HTH,

Brian.
Received on Fri Mar 28 2008 - 22:53:58 CET