Re: [RFC v2] epoll: avoid spinlock contention with wfcqueue

[Eric Wong <normalperson@yhbt.net>]

Eric Wong <normalperson@yhbt.net> wrote:
> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> > I'm also not entirely sure why you need to add enum epoll_item_state
> > along with expensive atomic ops to compute the state.  Wouldn't it be
> > enough to know in which queue the nodes are located ? If need be, you
> > could add new queues, e.g. one per state. So instead of marking states,
> > you would simply re-enqueue the nodes into per-state queues. This would
> > simplify zombie management and save a couple of brains in the process. ;-)
> 
> Is there a quick way to know which queue the node is located?
> 
> ep_enqueue may fire from several places at once (ep_poll_callback,
> ep_insert/ep_modify) so I think guarding it with something (currently
> ep_mark_ready) is required.  We used to use ep->lock to protect all the
> "if (!ep_is_linked) list_add_tail" calls, too.

I now think the EP_STATE_DEQUEUE state (and associated loop) is
unnecessary.  I only had it for the EPOLLET case.

Safe usage of EPOLLET requires hitting EAGAIN before expecting an event
anyways (as documented by epoll(7)).  So the potential of "losing" an
event while it is being sent to userspace existed before, and it still
exists now.  For an app which uses EPOLLET improperly (by not relying on
EAGAIN), the window for "losing" an event is slightly larger, but
probably still too small to matter...

This change wouldn't affect level-trigger (most epoll users, I think)
nor EPOLLONESHOT users.  Hopefully I'll be able to test more, soon.