Re: select() efficiency / epoll

[Davy Durham <pubaddr2@davyandbeth.com>]

Davide Libenzi wrote:

>
> There is no known problem in using epoll_ctl() in one thread while 
> another does epoll_wait().
> I suggest you to ask Valgrind to take a look at you binary. Since I 
> have no clue of what your software does, please create the *minimal* 
> code snippet that exploit the eventual problem, and post it.
>
Yes, I have pretty much confirmed this. And unfortunately I tried to 
make a minimal code snippet which demonstrates the problem, but wasn't 
able to do that before I figured out a work-around.  I may still try to 
create something for you to test against so you can fix it.  But I'm 
going to have to continue to work with the existing implementation since 
I'm going to be running this code on some production servers where 
updating the kernel might not be an option.

The work-around is as follows:

1) I create a queue that can hold operations to perform on the epoll 
structure and I protect it with a mutex.

2) Other threads (when needing to modify the epoll) lock the mutex and 
enque the operation into the operation queue instead of calling 
epoll_ctl itself (i.e. add this socket for reading.. add this socket for 
writing, remove this socket.. etc) *and* then cancel the epoll_wait() 
   I implemented the cancel by having a pipe() always being watched for 
read, and write a byte to it when I want to cancel (is there a better way?)
   There are several operations that could be supported 
(add/remove/modify/change userdata/etc), but I only need two myself.

3) There's only one thread that actually does the epoll_wait().  When 
epoll_wait() returns, (I first drain the cancel pipe so it never fills 
up) I handle what events need handling, and then lock the operations 
queue mutex, perform all the operations in the queue then clear the queue



So, this works for me now.

Thanks for all your guys' info.

-- Davy

P.S.   Davide, I still might get you that snipped, but it's not a 
trivial snippet as you can imagine... and timing is everything to the 
problem :( .. and also the question of WHERE it corrupts memory.. it 
seemed to be unpredictable so far.