[LSF/MM TOPIC] epoll topics

[LSF/MM TOPIC] epoll topics

[Jason Baron]

Hi,

I think it would be interesting to discuss recent topics that have come
up around epoll.

I recently proposed a new locking scheme that removes the global locking
using 'connected components', see: https://lkml.org/lkml/2015/1/15/667.
There is a new syscall to batch epoll_ctl() operations and improve the
timer precision that Fam Zheng is proposing: https://lkml.org/lkml/2015/1/8/70.
In addition, Eric Wong has proposed a scheme using 'wait free queues' to
improve lock contention between wakeups and event gathering:
https://lkml.org/lkml/2013/3/14/14.

In addition, we've observed some short comings from our production systems.
Because wait queues are woken up in order, we can get unfair loading
across threads, and we can also receive unecessary wakeups. It would be
great to discuss these topics further.

Thanks,

-Jason

Re: [LSF/MM TOPIC] epoll topics

[Eric Wong]

Jason Baron <jbaron@akamai.com> wrote:
> In addition, Eric Wong has proposed a scheme using 'wait free queues' to
> improve lock contention between wakeups and event gathering:
> https://lkml.org/lkml/2013/3/14/14.

Thanks for bringing this up again.  wfcqueue for epoll is still on hold
until I take care of other problems in userspace.  I'd welcome other
interested folks to help get it into shape again.

> In addition, we've observed some short comings from our production systems.
> Because wait queues are woken up in order, we can get unfair loading
> across threads,

Are you referring to LIFO scheduling of epoll_wait calling
__add_wait_queue_exclusive?  I always thought LIFO was preferable
since it kept the same threads active to reduce cache misses.

> and we can also receive unecessary wakeups. It would be
> great to discuss these topics further.

I'm happy to discuss via email; but I don't travel well.

Re: [LSF/MM TOPIC] epoll topics

[Jason Baron]

On 01/18/2015 10:50 PM, Eric Wong wrote:
> Jason Baron <jbaron@akamai.com> wrote:
>> In addition, we've observed some short comings from our production systems.
>> Because wait queues are woken up in order, we can get unfair loading
>> across threads,
> Are you referring to LIFO scheduling of epoll_wait calling
> __add_wait_queue_exclusive?  I always thought LIFO was preferable
> since it kept the same threads active to reduce cache misses.

So the specific case I had in mind was where you have an epfd
per-thread that is attached to a single listen socket. When a
POLLIN occurs on the listen socket, all threads in epoll_wait will
be woken up in the order they were added. Then, network
traffic ends up being processed on the thread which does the
accept(). This tends to result in an unbalanced load across the
threads.

I was thinking that if the wait queues rotated the wake up list
after each wake up it might help things in this case. Another option
is to employ SO_REUSEPORT, so I was wondering if this was
an issue pretty specific to network sockets or perhaps more
general...

Thanks,

-Jason

Re: [LSF/MM TOPIC] epoll topics

[Eric Wong]

Jason Baron <jbaron@akamai.com> wrote:
> On 01/18/2015 10:50 PM, Eric Wong wrote:
> > Jason Baron <jbaron@akamai.com> wrote:
> >> In addition, we've observed some short comings from our production systems.
> >> Because wait queues are woken up in order, we can get unfair loading
> >> across threads,
> > Are you referring to LIFO scheduling of epoll_wait calling
> > __add_wait_queue_exclusive?  I always thought LIFO was preferable
> > since it kept the same threads active to reduce cache misses.
> 
> So the specific case I had in mind was where you have an epfd
> per-thread that is attached to a single listen socket. When a
> POLLIN occurs on the listen socket, all threads in epoll_wait will
> be woken up in the order they were added. Then, network
> traffic ends up being processed on the thread which does the
> accept(). This tends to result in an unbalanced load across the
> threads.

Ah, yes.  Non-blocking listen sockets is a problem with epoll.

Have you tried using a dedicated thread for blocking accept() and
EPOLL_CTL_ADD?
Other threads will work off entries via epoll_wait and do non-blocking
I/O like a normal epoll server.  I do this for cmogstored:

     git clone git://bogomips.org/cmogstored
     (see accept_loop.c, works best w/o TCP_DEFER_ACCEPT)

One extra side effect of blocking accept() is it ends up balancing
fairly across multiple processes, too, so cmogstored's undocumented
multi-process option can distribute FDs more evenly.

Re: [LSF/MM TOPIC] epoll topics

[Jason Baron]

On 01/20/2015 02:58 PM, Eric Wong wrote:
> Jason Baron <jbaron@akamai.com> wrote:
>> On 01/18/2015 10:50 PM, Eric Wong wrote:
>>> Jason Baron <jbaron@akamai.com> wrote:
>>>> In addition, we've observed some short comings from our production systems.
>>>> Because wait queues are woken up in order, we can get unfair loading
>>>> across threads,
>>> Are you referring to LIFO scheduling of epoll_wait calling
>>> __add_wait_queue_exclusive?  I always thought LIFO was preferable
>>> since it kept the same threads active to reduce cache misses.
>> So the specific case I had in mind was where you have an epfd
>> per-thread that is attached to a single listen socket. When a
>> POLLIN occurs on the listen socket, all threads in epoll_wait will
>> be woken up in the order they were added. Then, network
>> traffic ends up being processed on the thread which does the
>> accept(). This tends to result in an unbalanced load across the
>> threads.
> Ah, yes.  Non-blocking listen sockets is a problem with epoll.
>
> Have you tried using a dedicated thread for blocking accept() and
> EPOLL_CTL_ADD?

That's a possibility for us - I am also wondering if the idea of rotating
the wait queue will work well in practice, since epoll will naturally tend
to wake up the thread that is least busy at the moment (since any
threads sitting in epoll_wait will tend to process the wakeup first).

Thanks,

-Jason