Re: [LSF/MM TOPIC] epoll topics

[Jason Baron <jbaron@akamai.com>]

On 01/18/2015 10:50 PM, Eric Wong wrote:
> Jason Baron <jbaron@akamai.com> wrote:
>> In addition, we've observed some short comings from our production systems.
>> Because wait queues are woken up in order, we can get unfair loading
>> across threads,
> Are you referring to LIFO scheduling of epoll_wait calling
> __add_wait_queue_exclusive?  I always thought LIFO was preferable
> since it kept the same threads active to reduce cache misses.

So the specific case I had in mind was where you have an epfd
per-thread that is attached to a single listen socket. When a
POLLIN occurs on the listen socket, all threads in epoll_wait will
be woken up in the order they were added. Then, network
traffic ends up being processed on the thread which does the
accept(). This tends to result in an unbalanced load across the
threads.

I was thinking that if the wait queues rotated the wake up list
after each wake up it might help things in this case. Another option
is to employ SO_REUSEPORT, so I was wondering if this was
an issue pretty specific to network sockets or perhaps more
general...

Thanks,

-Jason