Re: [PATCH] epoll: Improved support for multi-threaded clients

[Andrew Morton <akpm@linux-foundation.org>]

Let's cc Davide.

On Mon, 11 Jun 2012 15:34:49 -0700
Paton Lewis <palewis@adobe.com> wrote:

> 
> It is not currently possible to reliably delete epoll items when using the
> same epoll set from multiple threads. After calling epoll_ctl with
> EPOLL_CTL_DEL, another thread might still be executing code related to an
> event for that epoll item (in response to epoll_wait). Therefore the deleting
> thread does not know when it is safe to delete resources pertaining to the
> associated epoll item because another thread might be using those resources.

We often solve this sort of thing with refcounting - the EPOLL_CTL_DEL
will drop the refcount and, if that fell to zero, remove the object. 
So if the object is in use by another thread, that other thread will
hold a refcount and that other thread will do the object removal when
dropping its refcount.

I don't know if that model can be used here?

> The deleting thread could wait an arbitrary amount of time after calling
> epoll_ctl with EPOLL_CTL_DEL and before deleting the item, but this is
> inefficient and could result in the destruction of resources before another
> thread is done handling an event returned by epoll_wait.
> 
> This patch introduces the new epoll_ctl command EPOLL_CTL_DISABLE, which
> disables the associated epoll item and returns -EBUSY if the epoll item is not
> currently in the epoll ready queue. This allows multiple threads to use a
> mutex to determine when it is safe to delete an epoll item and its associated
> resources. This allows epoll items to be deleted and closed efficiently and
> without error.
> 
> This patch has been tested against the following checklist:
> http://kernelnewbies.org/UpstreamMerge/SubmitChecklist
> 
> The following psuedocode attempts to illustrate the problem as well as the
> solution provided by this patch.
> 
> 
> Pseudocode for the deleting thread:

It would be nice to start accumulating epoll test code in
tools/testing/selftests/epoll.  If you have something which can be used
to kick that off, please do consider preparing it.

Also, a user-visible feature wuch as this should be documented in Linux
manpages.  So please do cc Michael Kerrisk <mtk.manpages@gmail.com> as
we work on this.