(more) epoll troubles

(more) epoll troubles

[Michael Noisternig]

Hello,

and sorry again if this is the wrong place to ask (again, please hint to 
me to an appropriate place to ask in that case).

After experimenting with epoll edge-triggered mode I am clueless why on 
a few occassions I seem to not get any input notification despite data 
is available.

In detail: I have set up sockets with epoll events 
EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() as 
long as I get what I asked for, i.e. whenever read() returns either 
EAGAIN or less data than I asked for I take this as indication that I 
must wait for another EPOLLIN notification. However, this does not seem 
to work always.

Here is some log from my program:

0x9e6b8a8: read not avail (1460/2048 read)
i.e. tried to read 2048 bytes, got 1460 -> assume must wait for EPOLLIN 
for more data to read
(note that the fd is always in the epoll set with 
EPOLLET|EPOLLRDHUP|EPOLLIN)

0x9e6b8a8: read not avail (1460/2048 read)
got EPOLLIN notification, read 1460 bytes again -> wait for another 
notification

...after a few minutes without any notification...

0x9e6b8a8: GOT RDHUP (2001:0/80002001:0)
i.e. got notification for EPOLLRDHUP|EPOLLIN (2001), was waiting for 
EPOLLET|EPOLLRDHUP|EPOLLIN (80002001)

0x9e6b8a8: read not avail (342/2048 read)
then on trying to read we find data in the buffer (???)

0x9e6b8a8: stored 16384 bytes
this is the amount of data we expected to get /immediately/, i.e. 
without a minutes delay and rdhup

Of course you might think that the other side sent the data only just 
before closing its side, but it is unlikely since we expected to get the 
16384 bytes in one go (this is how that L7 protocol works), and I see 
this odd behaviour of missing input notifications now and then.

Does anybody know/have an idea why I am missing EPOLLIN notifications?

Any help is appreciated!!! Thanks in advance,
Michael
(please CC me when replying)

Re: (more) epoll troubles

[Robert Hancock]

Michael Noisternig wrote:
> Hello,
> 
> and sorry again if this is the wrong place to ask (again, please hint to 
> me to an appropriate place to ask in that case).
> 
> After experimenting with epoll edge-triggered mode I am clueless why on 
> a few occassions I seem to not get any input notification despite data 
> is available.
> 
> In detail: I have set up sockets with epoll events 
> EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() as 
> long as I get what I asked for, i.e. whenever read() returns either 
> EAGAIN or less data than I asked for I take this as indication that I 
> must wait for another EPOLLIN notification. However, this does not seem 
> to work always.
> 
> Here is some log from my program:
> 
> 0x9e6b8a8: read not avail (1460/2048 read)
> i.e. tried to read 2048 bytes, got 1460 -> assume must wait for EPOLLIN 
> for more data to read
> (note that the fd is always in the epoll set with 
> EPOLLET|EPOLLRDHUP|EPOLLIN)

It would likely be better to always continue trying to read until EAGAIN 
is returned, even if the read returned less than the requested amount, 
as implied here:

http://linux.die.net/man/7/epoll

"The function do_use_fd() uses the new ready file descriptor until 
EAGAIN is returned by either read(2) or write(2). An event driven state 
machine application should, after having received EAGAIN, record its 
current state so that at the next call to do_use_fd() it will continue 
to read(2) or write(2) from where it stopped before. "

> 
> 0x9e6b8a8: read not avail (1460/2048 read)
> got EPOLLIN notification, read 1460 bytes again -> wait for another 
> notification
> 
> ..after a few minutes without any notification...
> 
> 0x9e6b8a8: GOT RDHUP (2001:0/80002001:0)
> i.e. got notification for EPOLLRDHUP|EPOLLIN (2001), was waiting for 
> EPOLLET|EPOLLRDHUP|EPOLLIN (80002001)
> 
> 0x9e6b8a8: read not avail (342/2048 read)
> then on trying to read we find data in the buffer (???)
> 
> 0x9e6b8a8: stored 16384 bytes
> this is the amount of data we expected to get /immediately/, i.e. 
> without a minutes delay and rdhup
> 
> Of course you might think that the other side sent the data only just 
> before closing its side, but it is unlikely since we expected to get the 
> 16384 bytes in one go (this is how that L7 protocol works), and I see 
> this odd behaviour of missing input notifications now and then.
> 
> Does anybody know/have an idea why I am missing EPOLLIN notifications?
> 
> Any help is appreciated!!! Thanks in advance,
> Michael
> (please CC me when replying)
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: (more) epoll troubles

[Robert Hancock]

Robert Hancock wrote:
> Michael Noisternig wrote:
>> Hello,
>>
>> and sorry again if this is the wrong place to ask (again, please hint 
>> to me to an appropriate place to ask in that case).
>>
>> After experimenting with epoll edge-triggered mode I am clueless why 
>> on a few occassions I seem to not get any input notification despite 
>> data is available.
>>
>> In detail: I have set up sockets with epoll events 
>> EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() 
>> as long as I get what I asked for, i.e. whenever read() returns either 
>> EAGAIN or less data than I asked for I take this as indication that I 
>> must wait for another EPOLLIN notification. However, this does not 
>> seem to work always.
>>
>> Here is some log from my program:
>>
>> 0x9e6b8a8: read not avail (1460/2048 read)
>> i.e. tried to read 2048 bytes, got 1460 -> assume must wait for 
>> EPOLLIN for more data to read
>> (note that the fd is always in the epoll set with 
>> EPOLLET|EPOLLRDHUP|EPOLLIN)
> 
> It would likely be better to always continue trying to read until EAGAIN 
> is returned, even if the read returned less than the requested amount, 
> as implied here:
> 
> http://linux.die.net/man/7/epoll
> 
> "The function do_use_fd() uses the new ready file descriptor until 
> EAGAIN is returned by either read(2) or write(2). An event driven state 
> machine application should, after having received EAGAIN, record its 
> current state so that at the next call to do_use_fd() it will continue 
> to read(2) or write(2) from where it stopped before. "

Though, this is somewhat contradicted by the FAQ section:

"the condition that the read/write I/O space is exhausted can be 
detected by checking the amount of data read/write from/to the target 
file descriptor. For example, if you call read(2) by asking to read a 
certain amount of data and read(2) returns a lower number of bytes, you 
can be sure to have exhausted the read I/O space for such file descriptor."

Re: (more) epoll troubles

[Michael Noisternig]

Robert Hancock schrieb:
> Robert Hancock wrote:
>> Michael Noisternig wrote:
>>> Hello,
>>>
>>> and sorry again if this is the wrong place to ask (again, please hint 
>>> to me to an appropriate place to ask in that case).
>>>
>>> After experimenting with epoll edge-triggered mode I am clueless why 
>>> on a few occassions I seem to not get any input notification despite 
>>> data is available.
>>>
>>> In detail: I have set up sockets with epoll events 
>>> EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() 
>>> as long as I get what I asked for, i.e. whenever read() returns 
>>> either EAGAIN or less data than I asked for I take this as indication 
>>> that I must wait for another EPOLLIN notification. However, this does 
>>> not seem to work always.
>>>
>>> Here is some log from my program:
>>>
>>> 0x9e6b8a8: read not avail (1460/2048 read)
>>> i.e. tried to read 2048 bytes, got 1460 -> assume must wait for 
>>> EPOLLIN for more data to read
>>> (note that the fd is always in the epoll set with 
>>> EPOLLET|EPOLLRDHUP|EPOLLIN)
>>
>> It would likely be better to always continue trying to read until 
>> EAGAIN is returned, even if the read returned less than the requested 
>> amount, as implied here:
>>
>> http://linux.die.net/man/7/epoll
>>
>> "The function do_use_fd() uses the new ready file descriptor until 
>> EAGAIN is returned by either read(2) or write(2). An event driven 
>> state machine application should, after having received EAGAIN, record 
>> its current state so that at the next call to do_use_fd() it will 
>> continue to read(2) or write(2) from where it stopped before. "
> 
> Though, this is somewhat contradicted by the FAQ section:
> 
> "the condition that the read/write I/O space is exhausted can be 
> detected by checking the amount of data read/write from/to the target 
> file descriptor. For example, if you call read(2) by asking to read a 
> certain amount of data and read(2) returns a lower number of bytes, you 
> can be sure to have exhausted the read I/O space for such file descriptor."

Yes, exactly. I don't know what is causing the problem I'm experiencing. 
Especially as it happens rather infrequently.

Re: (more) epoll troubles

[Davide Libenzi]

On Mon, 1 Sep 2008, Michael Noisternig wrote:

> Robert Hancock schrieb:
> > Robert Hancock wrote:
> > > Michael Noisternig wrote:
> > > > Hello,
> > > > 
> > > > and sorry again if this is the wrong place to ask (again, please hint to
> > > > me to an appropriate place to ask in that case).
> > > > 
> > > > After experimenting with epoll edge-triggered mode I am clueless why on
> > > > a few occassions I seem to not get any input notification despite data
> > > > is available.
> > > > 
> > > > In detail: I have set up sockets with epoll events
> > > > EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() as
> > > > long as I get what I asked for, i.e. whenever read() returns either
> > > > EAGAIN or less data than I asked for I take this as indication that I
> > > > must wait for another EPOLLIN notification. However, this does not seem
> > > > to work always.
> > > > 
> > > > Here is some log from my program:
> > > > 
> > > > 0x9e6b8a8: read not avail (1460/2048 read)
> > > > i.e. tried to read 2048 bytes, got 1460 -> assume must wait for EPOLLIN
> > > > for more data to read
> > > > (note that the fd is always in the epoll set with
> > > > EPOLLET|EPOLLRDHUP|EPOLLIN)
> > > 
> > > It would likely be better to always continue trying to read until EAGAIN
> > > is returned, even if the read returned less than the requested amount, as
> > > implied here:
> > > 
> > > http://linux.die.net/man/7/epoll
> > > 
> > > "The function do_use_fd() uses the new ready file descriptor until EAGAIN
> > > is returned by either read(2) or write(2). An event driven state machine
> > > application should, after having received EAGAIN, record its current state
> > > so that at the next call to do_use_fd() it will continue to read(2) or
> > > write(2) from where it stopped before. "
> > 
> > Though, this is somewhat contradicted by the FAQ section:
> > 
> > "the condition that the read/write I/O space is exhausted can be detected by
> > checking the amount of data read/write from/to the target file descriptor.
> > For example, if you call read(2) by asking to read a certain amount of data
> > and read(2) returns a lower number of bytes, you can be sure to have
> > exhausted the read I/O space for such file descriptor."
> 
> Yes, exactly. I don't know what is causing the problem I'm experiencing.
> Especially as it happens rather infrequently.

The man page has been recently edited to avoid confusion:


Q9  Do I need to continuously read/write a file descriptor until 
    EAGAIN when using the EPOLLET  flag  (edge-triggered behavior) ?

A9  Receiving  an  event from epoll_wait(2) should suggest to you 
    that such file descriptor is ready for the requested I/O operation.  
    You must consider it ready until the next  (non-blocking)  read/write  
    yields EAGAIN.  When and how you will use the file descriptor is 
    entirely up to you.
    For  packet/token-oriented  files  (e.g.,  datagram socket, terminal 
    in canonical mode), the only way to detect the end of the read/write 
    I/O space is to continue to read/write until EAGAIN.
    For stream-oriented files (e.g., pipe, FIFO, stream socket), the 
    condition that the read/write I/O space is  exhausted can also be 
    detected by checking the amount of data read from / written to the 
    target file descriptor.  For example, if you call read(2) by asking to 
    read a certain amount  of  data  and  read(2) returns  a  lower number  
    of bytes, you can be sure of having exhausted the read I/O space for 
    the file descriptor.  The same is true when writing using write(2).  
    (Avoid this latter technique if  you  cannot guarantee that the 
    monitored file descriptor always refers to a stream-oriented file.)



- Davide

Re: (more) epoll troubles

[Andrew Morton]

On Sun, 31 Aug 2008 15:38:01 +0200 Michael Noisternig <mnoist@cosy.sbg.ac.at> wrote:

> Hello,
> 
> and sorry again if this is the wrong place to ask (again, please hint to 
> me to an appropriate place to ask in that case).
> 
> After experimenting with epoll edge-triggered mode I am clueless why on 
> a few occassions I seem to not get any input notification despite data 
> is available.
> 
> In detail: I have set up sockets with epoll events 
> EPOLLET|EPOLLRDHUP|EPOLLIN. When I get EPOLLIN for a socket, I read() as 
> long as I get what I asked for, i.e. whenever read() returns either 
> EAGAIN or less data than I asked for I take this as indication that I 
> must wait for another EPOLLIN notification. However, this does not seem 
> to work always.
> 
> Here is some log from my program:
> 
> 0x9e6b8a8: read not avail (1460/2048 read)
> i.e. tried to read 2048 bytes, got 1460 -> assume must wait for EPOLLIN 
> for more data to read
> (note that the fd is always in the epoll set with 
> EPOLLET|EPOLLRDHUP|EPOLLIN)
> 
> 0x9e6b8a8: read not avail (1460/2048 read)
> got EPOLLIN notification, read 1460 bytes again -> wait for another 
> notification
> 
> ...after a few minutes without any notification...
> 
> 0x9e6b8a8: GOT RDHUP (2001:0/80002001:0)
> i.e. got notification for EPOLLRDHUP|EPOLLIN (2001), was waiting for 
> EPOLLET|EPOLLRDHUP|EPOLLIN (80002001)
> 
> 0x9e6b8a8: read not avail (342/2048 read)
> then on trying to read we find data in the buffer (???)
> 
> 0x9e6b8a8: stored 16384 bytes
> this is the amount of data we expected to get /immediately/, i.e. 
> without a minutes delay and rdhup
> 
> Of course you might think that the other side sent the data only just 
> before closing its side, but it is unlikely since we expected to get the 
> 16384 bytes in one go (this is how that L7 protocol works), and I see 
> this odd behaviour of missing input notifications now and then.
> 
> Does anybody know/have an idea why I am missing EPOLLIN notifications?
> 

If you can put together a little test case which others can run then we
should be able to get this resolved super-quick.

Thanks.