Thread safety for epoll/libaio

Thread safety for epoll/libaio

[Li, Peng]

I apologize if I should not post this on LKML, but there seems to be
some lack of documentation for using epoll/AIO with threads.  Are
these interfaces thread-safe?  Can I use them safely in the following
way:

Thread A:  while(1) { io_getevents();  ... }
// wait forever until an event occurs, then handles the event and loop

Thread B:  while(1) { epoll_wait();  ... }
// same as thread A

Thread C:  ... io_submit(); ...

Thread D:  ... epoll_ctl(); ....

Suppose thread B calls epoll_wait and blocks before thread D calls
epoll_ctl.  Is it safe to do so? Will thread B be notified for the
event submitted by thread D?  Thread A and C pose the same question
for AIO.

I wrote a simple program to test these interfaces and they seem to
work without problems, but I am not sure if it is really safe to do so
in general.  If all of them works, it seems easy to use epoll and AIO
together as I can simply use another thread to harvest events from
thread A and B and make it look like a unified event notification
interface.

Peng

Re: Thread safety for epoll/libaio

[Anton Titov]

On Tue, 2006-02-28 at 15:36 -0500, Li, Peng wrote:

> Thread B:  while(1) { epoll_wait();  ... }
> // same as thread A
> Thread D:  ... epoll_ctl(); ....
> 
> Suppose thread B calls epoll_wait and blocks before thread D calls
> epoll_ctl.  Is it safe to do so? Will thread B be notified for the
> event submitted by thread D? 

Hello,

I have some (more) expirience with epoll and threads and it seem to work
well. If you have epoll_wait() in one thread and another thread do
epoll_ctl to add a handle, epoll_wait will wake up as soon as the handle
is ready for operation (most of the time instantly, when operation is
write). 

epoll man page states:

              Q6     Will the close of an fd cause it to be removed from
all epoll sets automatically?

              A6     Yes.
but I was experiencing some (rare) segfaults with my application while
benchmarking it when I just closing my descriptors. Debugging showed,
that I'm getting events for destroyed objects (with closed descriptors).
Adding epoll_ctl(..., EPOLL_CTL_DEL, ...) fixed this.

Re: Thread safety for epoll/libaio

[Benjamin LaHaise]

On Tue, Feb 28, 2006 at 03:36:11PM -0500, Li, Peng wrote:
> I apologize if I should not post this on LKML, but there seems to be
> some lack of documentation for using epoll/AIO with threads.  Are
> these interfaces thread-safe?  Can I use them safely in the following
> way:

I can only speak for libaio, which is completely thread safe.  Having a 
single thread read events and dispatch is likely to work quite well given 
the way the kernel interface is structured internally.  There is still 
room for improving the event mechanism to use a futex for waking so that 
the library can parse multiple events from userspace, but that is pending 
a heavier user like networking.

		-ben
-- 
"Ladies and gentlemen, I'm sorry to interrupt, but the police are here 
and they've asked us to stop the party."  Don't Email: <dont@kvack.org>.

Re: Thread safety for epoll/libaio

[Phillip Susi]

You don't mix epoll with io_submit; they are two completely different 
methods for doing IO.  The former is for use with bsd style non blocking 
IO, and the latter is truly async io.  epoll will signal you whenever a 
file can be read ( the read ahead buffer is non empty ) or written ( 
there's memory available to buffer writes ).  io_getevents() won't 
notify you of anything until you actually request a read or write, in 
which case, it lets you know when that has completed.

The two systems are completely non interchangeable.  With the former you 
wait first, then request some IO, which completes immediately.  With the 
latter, you request some IO ( possibly many ), then wait for some to 
finish.

Li, Peng wrote:
> I apologize if I should not post this on LKML, but there seems to be
> some lack of documentation for using epoll/AIO with threads.  Are
> these interfaces thread-safe?  Can I use them safely in the following
> way:
> 
> Thread A:  while(1) { io_getevents();  ... }
> // wait forever until an event occurs, then handles the event and loop
> 
> Thread B:  while(1) { epoll_wait();  ... }
> // same as thread A
> 
> Thread C:  ... io_submit(); ...
> 
> Thread D:  ... epoll_ctl(); ....
> 
> Suppose thread B calls epoll_wait and blocks before thread D calls
> epoll_ctl.  Is it safe to do so? Will thread B be notified for the
> event submitted by thread D?  Thread A and C pose the same question
> for AIO.
> 
> I wrote a simple program to test these interfaces and they seem to
> work without problems, but I am not sure if it is really safe to do so
> in general.  If all of them works, it seems easy to use epoll and AIO
> together as I can simply use another thread to harvest events from
> thread A and B and make it look like a unified event notification
> interface.
> 
> Peng

RE: Thread safety for epoll/libaio

[David Schwartz]

> I apologize if I should not post this on LKML, but there seems to be
> some lack of documentation for using epoll/AIO with threads.  Are
> these interfaces thread-safe?  Can I use them safely in the following
> way:

	They are thread safe.

> Thread A:  while(1) { io_getevents();  ... }
> // wait forever until an event occurs, then handles the event and loop
>
> Thread B:  while(1) { epoll_wait();  ... }
> // same as thread A
>
> Thread C:  ... io_submit(); ...
>
> Thread D:  ... epoll_ctl(); ....
>
> Suppose thread B calls epoll_wait and blocks before thread D calls
> epoll_ctl.  Is it safe to do so? Will thread B be notified for the
> event submitted by thread D?  Thread A and C pose the same question
> for AIO.

	Using the interfaces this way is pretty much their entire point. They'd be
almost useless if you couldn't use them in this way.

	DS