Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

Avi Kivity
2007-02-27T13:34:07Z

Ingo Molnar wrote:
> * Theodore Tso <tytso@mit.edu> wrote:
>
>
>> I think what you are not hearing, and what everyone else is saying
>> (INCLUDING Linus), is that for most programmers, state machines are
>> much, much harder to program, understand, and debug compared to
>> multi-threaded code. [...]
>>
>
> btw., another crutial thing that i think Evgeniy is missing is that
> threadlets /enable/ event loops to be used in practice! Right now the
> epoll/kevent programming model requires a total 100% avoidance of all
> context-switching in the 'main' event handler context while handling a
> request. If just 1% of all requests happen to block it might cause a
> /complete/ breakdown of an event loop's performance - it can easily
> cause a 10x drop in performance or worse!
>
> So context-switching has to be avoided in 100% of the code that runs
> while handling requests, file descriptors have to be set to nonblocking
> (causing extra system calls), and all the syscalls that might return
> incomplete with either -EINVAL or with a short read/write have to be
> converted into a state machine. (or in the alternative, user-space
> threading has to be used, which opens up another hornet's nest)
>
> /That/ is the main inhibiting factor of the measured use of event loops
> within Linux! It has zero integration capabilities with 'usual' coding
> techniques - driving the costs of its application up in the sky, and
> pushing event based servers into niches.
>
>

Having written such a niche event based server, I can 100% confirm what
Ingo is saying here. We had a single process drive I/O to the kernel
through an event model (based on kernel aio extended with IO_CMD_POLL),
and user level threads managed by a custom scheduler that managed I/O,
timeouts, and thread scheduling.

We once considered dropping from a user-level thread model to a state
machine model, but the effort was astronomical and we wouldn't see the
rewards until it was all done, so naturally we didn't do it.

> With threadlets the picture changes dramatically: all we have to
> concentrate on to get the performance of "100% event based servers" is
> to handle 'most' rescheduling events in the event loop. A 10-20% context
> switching ratio does not hurt at all. (it causes ~1% of throughput
> loss.)
>
> Furthermore, even if a particular configuration or module of the server
> (say Apache) happens to trigger a high rate of scheduling, the
> performance breakdown model of threadlets is /vastly/ superior to event
> based servers. The measurements so far have shown that the absolute
> worst-case threading server performance is at around 60% of that of
> non-context-switching servers - and even that level is reached
> gradually, leaving time for action for the server owner. While with
> fully event based servers there are mostly only two modes of
> performance: 100% performance and near-0% performance: total breakdown.
>

Yes. Threadlets as the default aio solution (easy to use, acceptable
performance even in worst cases), with specialized solutions where
applicable (epoll for networking, aio for O_DIRECT disk) look like a
good mix of performance and sanity.

--
error compiling committee.c: too many arguments to function