Re: select() efficiency / epoll

Davy Durham
2005-08-24T07:50:35Z

Davide Libenzi wrote:

>
> There is no known problem in using epoll_ctl() in one thread while
> another does epoll_wait().
> I suggest you to ask Valgrind to take a look at you binary. Since I
> have no clue of what your software does, please create the *minimal*
> code snippet that exploit the eventual problem, and post it.
>
Yes, I have pretty much confirmed this. And unfortunately I tried to
make a minimal code snippet which demonstrates the problem, but wasn't
able to do that before I figured out a work-around. I may still try to
create something for you to test against so you can fix it. But I'm
going to have to continue to work with the existing implementation since
I'm going to be running this code on some production servers where
updating the kernel might not be an option.

The work-around is as follows:

1) I create a queue that can hold operations to perform on the epoll
structure and I protect it with a mutex.

2) Other threads (when needing to modify the epoll) lock the mutex and
enque the operation into the operation queue instead of calling
epoll_ctl itself (i.e. add this socket for reading.. add this socket for
writing, remove this socket.. etc) *and* then cancel the epoll_wait()
I implemented the cancel by having a pipe() always being watched for
read, and write a byte to it when I want to cancel (is there a better way?)
There are several operations that could be supported
(add/remove/modify/change userdata/etc), but I only need two myself.

3) There's only one thread that actually does the epoll_wait(). When
epoll_wait() returns, (I first drain the cancel pipe so it never fills
up) I handle what events need handling, and then lock the operations
queue mutex, perform all the operations in the queue then clear the queue

So, this works for me now.

Thanks for all your guys' info.

-- Davy

P.S. Davide, I still might get you that snipped, but it's not a
trivial snippet as you can imagine... and timing is everything to the
problem :( .. and also the question of WHERE it corrupts memory.. it
seemed to be unpredictable so far.