Re: [LSF/MM TOPIC] epoll topics

[Jason Baron <jbaron@akamai.com>]

On 01/20/2015 02:58 PM, Eric Wong wrote:
> Jason Baron <jbaron@akamai.com> wrote:
>> On 01/18/2015 10:50 PM, Eric Wong wrote:
>>> Jason Baron <jbaron@akamai.com> wrote:
>>>> In addition, we've observed some short comings from our production systems.
>>>> Because wait queues are woken up in order, we can get unfair loading
>>>> across threads,
>>> Are you referring to LIFO scheduling of epoll_wait calling
>>> __add_wait_queue_exclusive?  I always thought LIFO was preferable
>>> since it kept the same threads active to reduce cache misses.
>> So the specific case I had in mind was where you have an epfd
>> per-thread that is attached to a single listen socket. When a
>> POLLIN occurs on the listen socket, all threads in epoll_wait will
>> be woken up in the order they were added. Then, network
>> traffic ends up being processed on the thread which does the
>> accept(). This tends to result in an unbalanced load across the
>> threads.
> Ah, yes.  Non-blocking listen sockets is a problem with epoll.
>
> Have you tried using a dedicated thread for blocking accept() and
> EPOLL_CTL_ADD?

That's a possibility for us - I am also wondering if the idea of rotating
the wait queue will work well in practice, since epoll will naturally tend
to wake up the thread that is least busy at the moment (since any
threads sitting in epoll_wait will tend to process the wakeup first).

Thanks,

-Jason