[Xenomai-help] xnpipe EPOLLIN starvation

[Xenomai-help] xnpipe EPOLLIN starvation

[Filip Van Rillaer]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

Hi,
 
I am transferring data from the Xenomai domain to the 2nd domain with
the xenomai pipe.  In the 2nd domain I am using the epoll interface to
wait for the arrival of new data.  The first time that data are written
in the pipe at the Xenomai side, the 2nd domain receives the EPOLLIN (an
that's OK).  But the 2nd time data are written in the pipe at the
Xenomai side, the 2nd domain is NOT receiving EPOLLIN and therefore dies
forever.
 
I am not familiar at all with the Xenomai internals, but I tried to
gather some information: it look like when the first characters are
written at the Xenomai side, the xnpipe_wakeup_proc calls
xnpipe_dequeue_wait and this way it is removed forever from the
waiters-queue.
When I remove the "else"-instruction in xnpipe_poll so that
xnpipe_wakeup_proc is called un-conditionally, then it works as expected
... however I have no idea if this patch is correct.  So please comment.
 
FYI:
URL: http://svn.gna.org/svn/xenomai/trunk
Repository Root: http://svn.gna.org/svn/xenomai
Repository UUID: c6d672ea-8702-0410-b560-f74c916a59fe
Revision: 4047

 
Best regards,
Filip
 
 
 
 

[-- Attachment #2: Type: text/html, Size: 2956 bytes --]

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Gilles Chanteperdrix]

Filip Van Rillaer wrote:
> Hi,
>  
> I am transferring data from the Xenomai domain to the 2nd domain with
> the xenomai pipe.  In the 2nd domain I am using the epoll interface to
> wait for the arrival of new data.  The first time that data are written
> in the pipe at the Xenomai side, the 2nd domain receives the EPOLLIN (an
> that's OK).  But the 2nd time data are written in the pipe at the
> Xenomai side, the 2nd domain is NOT receiving EPOLLIN and therefore dies
> forever.
>  
> I am not familiar at all with the Xenomai internals, but I tried to
> gather some information: it look like when the first characters are
> written at the Xenomai side, the xnpipe_wakeup_proc calls
> xnpipe_dequeue_wait and this way it is removed forever from the
> waiters-queue.
> When I remove the "else"-instruction in xnpipe_poll so that
> xnpipe_wakeup_proc is called un-conditionally, then it works as expected
> ... however I have no idea if this patch is correct.  So please comment.
>  
> FYI:
> URL: http://svn.gna.org/svn/xenomai/trunk
> Repository Root: http://svn.gna.org/svn/xenomai
> Repository UUID: c6d672ea-8702-0410-b560-f74c916a59fe
> Revision: 4047

Are you sure you are not having a problem with "edge-triggered" versus
"level-triggered" notification ?
http://www.kegel.com/c10k.html#nb.edge
http://www.kegel.com/c10k.html#nb

-- 
                                                 Gilles.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Filip Van Rillaer]

Hello Gilles,

EPOLLET is indeed enabled.
The first time EPOLLIN is received in the 2nd domain, I do a read (it
receives 1 byte in a buffer of 256).  So the read is exhausted and I
expect that the next time data arrives I should get the EPOLLIN (as it
BECOMES readable).  But even adding another read (which returns -EAGAIN)
did not make it possible to receive the EPOLLIN when a little bit later
data is written at the Xenomai side of the xnpipe.

Best regards,
Filip

-----Original Message-----
From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org
Sent: 31 July 2008 12:12
To: Filip Van Rillaer
Cc: xenomai@xenomai.org
Subject: Re: [Xenomai-help] xnpipe EPOLLIN starvation

Filip Van Rillaer wrote:
> Hi,
>  
> I am transferring data from the Xenomai domain to the 2nd domain with 
> the xenomai pipe.  In the 2nd domain I am using the epoll interface to

> wait for the arrival of new data.  The first time that data are 
> written in the pipe at the Xenomai side, the 2nd domain receives the 
> EPOLLIN (an that's OK).  But the 2nd time data are written in the pipe

> at the Xenomai side, the 2nd domain is NOT receiving EPOLLIN and 
> therefore dies forever.
>  
> I am not familiar at all with the Xenomai internals, but I tried to 
> gather some information: it look like when the first characters are 
> written at the Xenomai side, the xnpipe_wakeup_proc calls 
> xnpipe_dequeue_wait and this way it is removed forever from the 
> waiters-queue.
> When I remove the "else"-instruction in xnpipe_poll so that 
> xnpipe_wakeup_proc is called un-conditionally, then it works as 
> expected ... however I have no idea if this patch is correct.  So
please comment.
>  
> FYI:
> URL: http://svn.gna.org/svn/xenomai/trunk
> Repository Root: http://svn.gna.org/svn/xenomai Repository UUID: 
> c6d672ea-8702-0410-b560-f74c916a59fe
> Revision: 4047

Are you sure you are not having a problem with "edge-triggered" versus
"level-triggered" notification ?
http://www.kegel.com/c10k.html#nb.edge
http://www.kegel.com/c10k.html#nb

-- 
                                                 Gilles.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Philippe Gerum]

Filip Van Rillaer wrote:
> Hello Gilles,
> 
> EPOLLET is indeed enabled.
> The first time EPOLLIN is received in the 2nd domain, I do a read (it
> receives 1 byte in a buffer of 256).  So the read is exhausted and I
> expect that the next time data arrives I should get the EPOLLIN (as it
> BECOMES readable).  But even adding another read (which returns -EAGAIN)
> did not make it possible to receive the EPOLLIN when a little bit later
> data is written at the Xenomai side of the xnpipe.
> 

Does this help?

--- ksrc/nucleus/pipe.c	(revision 4044)
+++ ksrc/nucleus/pipe.c	(working copy)
@@ -1029,9 +1029,12 @@
 {
 	xnpipe_state_t *state = (xnpipe_state_t *)file->private_data;
 	unsigned r_mask = 0, w_mask = 0;
+	spl_t s;

 	poll_wait(file, &state->readq, pt);

+	xnlock_get_irqsave(&nklock, s);
+
 	if (testbits(state->status, XNPIPE_KERN_CONN))
 		w_mask |= (POLLOUT | POLLWRNORM);

@@ -1046,6 +1049,8 @@
 		 */
 		xnpipe_enqueue_wait(state, XNPIPE_USER_WREAD);

+	xnlock_put_irqrestore(&nklock, s);
+
 	/*
 	 * A descriptor is always ready for writing with the current
 	 * implementation, so there is no need to have/handle the

-- 
Philippe.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Filip Van Rillaer]

Hello Gilles,

Thanks, but unfortunately it doesn't cure the problem.

Best regards,
Filip 

-----Original Message-----
From: Philippe Gerum [mailto:rpm@xenomai.org
Sent: 31 July 2008 12:34
To: Filip Van Rillaer
Cc: Gilles Chanteperdrix; xenomai@xenomai.org
Subject: Re: [Xenomai-help] xnpipe EPOLLIN starvation

Filip Van Rillaer wrote:
> Hello Gilles,
> 
> EPOLLET is indeed enabled.
> The first time EPOLLIN is received in the 2nd domain, I do a read (it 
> receives 1 byte in a buffer of 256).  So the read is exhausted and I 
> expect that the next time data arrives I should get the EPOLLIN (as it

> BECOMES readable).  But even adding another read (which returns 
> -EAGAIN) did not make it possible to receive the EPOLLIN when a little

> bit later data is written at the Xenomai side of the xnpipe.
> 

Does this help?

--- ksrc/nucleus/pipe.c	(revision 4044)
+++ ksrc/nucleus/pipe.c	(working copy)
@@ -1029,9 +1029,12 @@
 {
 	xnpipe_state_t *state = (xnpipe_state_t *)file->private_data;
 	unsigned r_mask = 0, w_mask = 0;
+	spl_t s;

 	poll_wait(file, &state->readq, pt);

+	xnlock_get_irqsave(&nklock, s);
+
 	if (testbits(state->status, XNPIPE_KERN_CONN))
 		w_mask |= (POLLOUT | POLLWRNORM);

@@ -1046,6 +1049,8 @@
 		 */
 		xnpipe_enqueue_wait(state, XNPIPE_USER_WREAD);

+	xnlock_put_irqrestore(&nklock, s);
+
 	/*
 	 * A descriptor is always ready for writing with the current
 	 * implementation, so there is no need to have/handle the

--
Philippe.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> Filip Van Rillaer wrote:
>> Hello Gilles,
>>
>> EPOLLET is indeed enabled.
>> The first time EPOLLIN is received in the 2nd domain, I do a read (it
>> receives 1 byte in a buffer of 256).  So the read is exhausted and I
>> expect that the next time data arrives I should get the EPOLLIN (as it
>> BECOMES readable). 

My point is that, when edge-triggered, you will not get notified if you
receive data while the reading thread has not yet managed to handle the
first notification.

-- 
                                                 Gilles.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Filip Van Rillaer]

Hello Gilles,
 
The 2nd write at the Xenomai side is coming a lot later.  In fact the
complete flow is a lot more complicated, I'll try to explain it. 

A) User program in 2nd domain is called xenopipe. This program is using
the epoll interface to synchronize with tty1 and the pipe
B  A driver called rtvcom (emulating a uart) in Xenomai connected to the
other side of the pipe and notified by rt_pipe_monitor of activity on
the pipe.
C) A modem program in Xenomai that is talking to rtvcom

Xenomai modem program <-> rtvcom driver <-> pipe <-> xenopipe user
program <-> tty1 <-> keyboard
                                   

1) I press 'A' on my keyboard that is connected to a terminal tty1
2) EPOLLIN and EPOLLOUT from tty1 are received by xenopipe (EPOLLOUT
ignore as there is nottying to write now)
3) xenopipe reads from tty1
4) xenopipe writes to /dev/rtp1
5) rtvcom receives P_EVENT_INPUT (rt_pipe_monitor)
6) rtvcom notifies the modemprogram that new data has arrived and the
modemprogram reads from rtvcom (rt_dev_read), which results in rtvcom
calling rt_pipe_read. So now the 'A' has arrived in the modem program.
7) the modemprogram echo's back the received 'A' so it writes to rtvcom
(rt_dev_write), which results in rtvcom calling rt_pipe_write
8) xenopipe receives EPOLLIN and EPOLLOUT from pipe (EPOLLOUT is ignored
as we have nothing to write now)
9) xenopipe reads the echo-back character from the pipe
10) rtvcom receives P_EVENT_OUTPUT (ignored as it has currently nothing
to do write)
11) xenopipe writes the echo-back character to tty1
12) xenopipe receives EPOLLOUT of tty1 when the HW has sent out the
echo-back character

When I then type a little bit later the character 'T' on my keyboard the
sequence starts again from 2) but stops at 7).

Best regards,
Filip

-----Original Message-----
From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org
Sent: 31 July 2008 14:14
To: rpm@xenomai.org
Cc: Filip Van Rillaer; xenomai@xenomai.org
Subject: Re: [Xenomai-help] xnpipe EPOLLIN starvation

Philippe Gerum wrote:
> Filip Van Rillaer wrote:
>> Hello Gilles,
>>
>> EPOLLET is indeed enabled.
>> The first time EPOLLIN is received in the 2nd domain, I do a read (it

>> receives 1 byte in a buffer of 256).  So the read is exhausted and I 
>> expect that the next time data arrives I should get the EPOLLIN (as 
>> it BECOMES readable).

My point is that, when edge-triggered, you will not get notified if you
receive data while the reading thread has not yet managed to handle the
first notification.

-- 
                                                 Gilles.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Philippe Gerum]

Filip Van Rillaer wrote:
> Hi,
>  
> I am transferring data from the Xenomai domain to the 2nd domain with
> the xenomai pipe.  In the 2nd domain I am using the epoll interface to
> wait for the arrival of new data.  The first time that data are written
> in the pipe at the Xenomai side, the 2nd domain receives the EPOLLIN (an
> that's OK).  But the 2nd time data are written in the pipe at the
> Xenomai side, the 2nd domain is NOT receiving EPOLLIN and therefore dies
> forever.
>

Actually, the xnpipe_poll logic is flawed. More later,

> I am not familiar at all with the Xenomai internals, but I tried to
> gather some information: it look like when the first characters are
> written at the Xenomai side, the xnpipe_wakeup_proc calls
> xnpipe_dequeue_wait and this way it is removed forever from the
> waiters-queue.
> When I remove the "else"-instruction in xnpipe_poll so that
> xnpipe_wakeup_proc is called un-conditionally, then it works as expected
> ... however I have no idea if this patch is correct.  So please comment.
>  
> FYI:
> URL: http://svn.gna.org/svn/xenomai/trunk
> Repository Root: http://svn.gna.org/svn/xenomai
> Repository UUID: c6d672ea-8702-0410-b560-f74c916a59fe
> Revision: 4047
>  
> Best regards,
> Filip
>  
>  
>  
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help


-- 
Philippe.

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Filip Van Rillaer]

Hello Philippe,

FYI: As I was not 100% sure that the EPOLLET of a read on a fd didn't
impact the write on the same fd, I just made a new version of xenopipe
(still with epoll) with separate filedescriptors for read and write in
the epoll_ctl.   So epoll_wait has 4 entries : fd_tty1_read,
fd_tty1_write, fd_pipe_read, fd_pipe_write ... but the problem remains.

To be able to open a pipe (e.g. /dev/rtp1) once as O_WRONLY and once as
O_RDONLY, I had to hack the pipe-code.  It would be nice if this would
be possible without hacking the code.

Best regards,
Filip

Re: [Xenomai-help] xnpipe EPOLLIN starvation

[Philippe Gerum]

Filip Van Rillaer wrote:
> Hello Philippe,
> 
> FYI: As I was not 100% sure that the EPOLLET of a read on a fd didn't
> impact the write on the same fd, I just made a new version of xenopipe
> (still with epoll) with separate filedescriptors for read and write in
> the epoll_ctl.   So epoll_wait has 4 entries : fd_tty1_read,
> fd_tty1_write, fd_pipe_read, fd_pipe_write ... but the problem remains.
> 
> To be able to open a pipe (e.g. /dev/rtp1) once as O_WRONLY and once as
> O_RDONLY, I had to hack the pipe-code.  It would be nice if this would
> be possible without hacking the code.
>

You may want to retry using epoll() with the latest updates committed.

> Best regards,
> Filip
> 
> 


-- 
Philippe.