Re: threadlets as 'naive pool of threads', epoll, some measurements

Davide Libenzi
2007-02-26T21:22:39Z

On Mon, 26 Feb 2007, Ingo Molnar wrote:

>
> update:
>
> > i have tried the one Evgeniy provided in the URL:
> >
> > http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
> >
> > and 'ab -k -c8000 -n80000' almost always aborts with:
> >
> > apr_socket_recv: Connection reset by peer (104)
> >
> > in the few cases it finishes, i got the following epoll result, over
> > gigabit ethernet, on an UP Athlon64 box:
> >
> > eserver_epoll: 7800 reqs/sec
>
> eserver_epoll.c had a number of bugs. The most serious one was the
> apparently buggy use of "EPOLLET" (edge-triggered events). Removing that
> and moving epoll to level-triggered (which is slower but does not result
> in missed events) gives:
>
> eserver_epoll: 9400 reqs/sec
>
> > the same with the most naive implementation of the same, using
> > threadlets:
> >
> > eserver_threadlet: 5800 reqs/sec
>
> eserver_epoll_threadlet: 9400 reqs/sec
>
> as expected, the level of extra blocking triggered by this is low - even
> if the full request function runs without nonblock assumptions.

That looks pretty good. I started (spare time allowing) laying down the
based for a more realistic test as far webserver-like benchmarking.
I want to compare three solutions, that uses the same internal code (as
far as request parsing and content delivery).

1) A fully thread classical web "server". This does the trivial
accept+pthread_create and in there does open+fstat+sendhdrs+sendfile.
This is already done:

http://www.xmailserver.org/thrhttp.c

Do a `gcc -o thrhttp thrhttp.c -lpthread` and you're done.

2) A coronet (coroutine+epoll library) handling for network
events/dispatch, plus GUASI (Generic Userspace Asyncronous Syscall
Interface) to handle generic IO.

libpcl: http://www.xmailserver.org/libpcl.html
coronet: http://www.xmailserver.org/coronet-lib.html
GUASI: http://www.xmailserver.org/guasi-lib.html

cghttpd: http://www.xmailserver.org/cghttpd-home.html

All these are configure+make+make_install installable.
The cghttpd server has the same parsing/content-delivery features of
thrhttp, but it uses coroutine+epoll to handle network events, and
GUASI (an old pthread-based userspace implementation of async execution)
to give open/fstat/sendfile an async behaviour. It implements the
epoll_wait() (conet_events_wait() acutally, but that maps directly to
epoll_wait()) hosted inside an async GUASI request, and the GUASI
dispatch loop handle the case with:

if (cookie == conet_events_wait_cookie)
handle_network_events();

3) Finally, a very similar to cghttpd implementation, but this time using
the *syslets* to handle async requests. That should be a prety easy
change to cghttpd, by the mean of replacing the CGHTTPD_SYSCALL() macro
with proper syslet code. The big advantage of the syslets here, is that
they do have the cachehit optimization, while GUASI always have to
trigger a queueing.
Of course, since the only machine I have with enough RAM to keep many
thousands of session active is a dual Opteron, I'd need to have an
x86-64 version of the patch. That shouldn't be a big problem though.

This hopefully may prove two points. First, a fully threaded solution does
not scale well when dealing with thousands and thousands of sessions.
Second, the cachehit syslet trick is The Man in the syslet code, and kicks
userspace solution #2 ass.
In the meantime, I think Jens tests are more meaningfull, as far of usage
field goes, than any network based test.

- Davide