[Xenomai-help] [RT_PIPE] - User space read() not waking up

[Xenomai-help] [RT_PIPE] - User space read() not waking up

[Viktor Stark]

Hello,

I'm having a little issue with RT pipes and I was wondering if anyone could give me some pointers.

I wrote a RT kernel module for a DAQ card. The DAQ card gets interrupts every 200 us (5 kHz) and I've got a RT interrupt handler which wakes up a RT function (by sending an event to it). Inside that function I use rt_pripe_write to send some data (about 300 bytes) down to user space.

On the user space side, I've got a real small executable which basically opens the pipe via the Xenomai registry and then reads data from it in a tight loop. Inside that loop I read the TSC after each write and I compute a minimum / maximum for a 10 seconds cycle. The problem is that I get a maximum of about 395 us (which is 2 times 200 us approximately). On the kernel side I also do timings and I am 100% sure that my function using rt_pipe_write() is woken up every 200 us (+- 10 us jitter), that is no interrupts are missed.

After further reasearch I realised that I get a maximum value every 10000 cycles, that is every two seconds. I don't have anything in my system (to my knowledge) that has such a period so I'm really confused about what could be causing this.

I tried to have a look at pipe.c in src/nucleus and saw that the user space side is supposed to get woken up on each xnpipe_send() but I admit that I didn't quite catch all the details.

So basically, my question is:
If I have rt_pipe_write() writing to a pipe from a kernel module with a 200 us period and I have a read() in a tight loop reading from that same pipe is user space, what could be the origin of the fact that sometimes my read() doesn't get woken up?

It's worth to mention too, that I'm not losing any data. That is, after the read which eats up 2 cycles, the next read finishes nearly instantaneously (within 5 us) as if only the second rt_pipe_write() woke up the user space read().
Timing illustration (time between read()'s in us): 195 | 200 | 205 | 395 | 5 | 195 | 200 | 200

My setup:
Xenomai 2.2-rc3
Kernel 2.6.17 (with Xenomai compiled in - ie. not compiled as modules)
P4 3GHz - 1Gb RAM
No APM/ACPI/CpuFreq - SMI workaround enabled.
Tried with 2.6.19 / Xenomai 2.3 - same results.

Thanks in advance for any pointers.

Keep up the good work! Xenomai just *rocks*.

Viktor STARK 
-- 
"Feel free" - 5 GB Mailbox, 50 FreeSMS/Monat ...
Jetzt GMX ProMail testen: www.gmx.net/de/go/mailfooter/promail-out

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]

Viktor Stark wrote:
> Hello,
> 
> I'm having a little issue with RT pipes and I was wondering if anyone could give me some pointers.
> 
> I wrote a RT kernel module for a DAQ card. The DAQ card gets interrupts every 200 us (5 kHz) and I've got a RT interrupt handler which wakes up a RT function (by sending an event to it). Inside that function I use rt_pripe_write to send some data (about 300 bytes) down to user space.
> 
> On the user space side, I've got a real small executable which basically opens the pipe via the Xenomai registry and then reads data from it in a tight loop. Inside that loop I read the TSC after each write and I compute a minimum / maximum for a 10 seconds cycle. The problem is that I get a maximum of about 395 us (which is 2 times 200 us approximately). On the kernel side I also do timings and I am 100% sure that my function using rt_pipe_write() is woken up every 200 us (+- 10 us jitter), that is no interrupts are missed.
> 
> After further reasearch I realised that I get a maximum value every 10000 cycles, that is every two seconds. I don't have anything in my system (to my knowledge) that has such a period so I'm really confused about what could be causing this.
> 
> I tried to have a look at pipe.c in src/nucleus and saw that the user space side is supposed to get woken up on each xnpipe_send() but I admit that I didn't quite catch all the details.
> 
> So basically, my question is:
> If I have rt_pipe_write() writing to a pipe from a kernel module with a 200 us period and I have a read() in a tight loop reading from that same pipe is user space, what could be the origin of the fact that sometimes my read() doesn't get woken up?
> 
> It's worth to mention too, that I'm not losing any data. That is, after the read which eats up 2 cycles, the next read finishes nearly instantaneously (within 5 us) as if only the second rt_pipe_write() woke up the user space read().
> Timing illustration (time between read()'s in us): 195 | 200 | 205 | 395 | 5 | 195 | 200 | 200
> 
> My setup:
> Xenomai 2.2-rc3
> Kernel 2.6.17 (with Xenomai compiled in - ie. not compiled as modules)
> P4 3GHz - 1Gb RAM
> No APM/ACPI/CpuFreq - SMI workaround enabled.
> Tried with 2.6.19 / Xenomai 2.3 - same results.
> 
> Thanks in advance for any pointers.

Have a look at the ipipe tracer:

http://www.xenomai.org/index.php/I-pipe:Tracer

Put a xntrace_user_freeze at the spot in your application where you
detect the misbehaviour and check the back-trace of the last few hundred
microseconds. If you don't understand what you see (the tracer's
information can be overwhelming...), feel free to post the trace.

> 
> Keep up the good work! Xenomai just *rocks*.

Nice to hear about yet another happy user. :)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Dmitry Adamushko]

[-- Attachment #1: Type: text/plain, Size: 1706 bytes --]

Hello,


I wrote a RT kernel module for a DAQ card. The DAQ card gets interrupts
> every 200 us (5 kHz) and I've got a RT interrupt handler which wakes up a RT
> function


I guess, you mean a RT task. Is it a kernel or user space task?


[...]
> So basically, my question is:
> If I have rt_pipe_write() writing to a pipe from a kernel module with a
> 200 us period and I have a read() in a tight loop reading from that same
> pipe is user space, what could be the origin of the fact that sometimes my
> read() doesn't get woken up?


Xenomai does provide a Linux system with hard real-time support but it
doesn't make the Linux kernel itself hard real-time, nor user-space
applications which relies merely on the standard kernel services.

Your task (here it even doesn't matter whether it's a linux or xenomai task)
relies on read() so it's under control of the linux scheduler, hence it
depends on the /scheduling latency/ provided by Linux, not Xenomai.

Roughly speaking, with Xenomai you have 2 domains: primary (all
time-critical activities must stay here) and secondary (Linux standard
drivers, applications, xenomai tasks while executing linux system calls).

So hard real-time support is provided only as long as your tasks/drivers
work in the primary domain. To this goal, (1) drivers are developed with
RTDM (preferable way); (2) applications (both kernel and user-space) are
developed using one of the Xenomai skins (native, posix, ...).

And as I mentioned above, it's not a case for your task with a read() loop.

So you should probably reconsider your application design to clearly
separate the parts that require hard real-time support from the rest.


-- 
Best regards,
Dmitry Adamushko

[-- Attachment #2: Type: text/html, Size: 2144 bytes --]

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Dmitry Adamushko wrote:
> [...]
>> So basically, my question is:
>> If I have rt_pipe_write() writing to a pipe from a kernel module with a
>> 200 us period and I have a read() in a tight loop reading from that same
>> pipe is user space, what could be the origin of the fact that
>> sometimes my
>> read() doesn't get woken up?
> 
> 
> Xenomai does provide a Linux system with hard real-time support but it
> doesn't make the Linux kernel itself hard real-time, nor user-space
> applications which relies merely on the standard kernel services.

Good that you read the mail more thoroughly than I did... :-/

Yes, that's obviously an application design issue.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Viktor STARK]

Dmitry Adamushko wrote:
>
> Hello,
>  
>
>     I wrote a RT kernel module for a DAQ card. The DAQ card gets
>     interrupts every 200 us (5 kHz) and I've got a RT interrupt
>     handler which wakes up a RT function
>
>
> I guess, you mean a RT task. Is it a kernel or user space task?
>  
It's a kernel space RT task, yes.
>
>     [...]
>     So basically, my question is:
>     If I have rt_pipe_write() writing to a pipe from a kernel module
>     with a 200 us period and I have a read() in a tight loop reading
>     from that same pipe is user space, what could be the origin of the
>     fact that sometimes my read() doesn't get woken up?
>
>
> Xenomai does provide a Linux system with hard real-time support but it 
> doesn't make the Linux kernel itself hard real-time, nor user-space 
> applications which relies merely on the standard kernel services.
>
> Your task (here it even doesn't matter whether it's a linux or xenomai 
> task) relies on read() so it's under control of the linux scheduler, 
> hence it depends on the /scheduling latency/ provided by Linux, not 
> Xenomai.
>
> Roughly speaking, with Xenomai you have 2 domains: primary (all 
> time-critical activities must stay here) and secondary (Linux standard 
> drivers, applications, xenomai tasks while executing linux system calls).
>
> So hard real-time support is provided only as long as your 
> tasks/drivers work in the primary domain. To this goal, (1) drivers 
> are developed with RTDM (preferable way); (2) applications (both 
> kernel and user-space) are developed using one of the Xenomai skins 
> (native, posix, ...).
>
> And as I mentioned above, it's not a case for your task with a read() 
> loop.
>
> So you should probably reconsider your application design to clearly 
> separate the parts that require hard real-time support from the rest.
I do understand the domain separation, but if I got things right, pipes 
can only be read using read() in user space. That's also what the API 
documentation suggests.

Of course, a read() can be scheduled by Linux, but:
1) Why does it happen exactly every 10000 cycles (2 seconds)? The fact 
that this erratic behaviour is so deterministic makes me think that it 
might not be standard Linux related. I mean the fact that it happens 
every 2 seconds (and I mean 2e6 us, not just approximately 2 seconds) 
just makes it look like it's provoked by something which is RT.
2) Even if my read() is scheduled, how comes I only get past the read() 
after the second rt_pipe_write()? One would think that if my read() is 
scheduled I could be waking up after 250 us or 300 us, no? But I'm 
*always* woken up after 2 cycles (that is 400 us).
3) Pipes seemed to be a good way to transfer data from kernel space. 
What other (maybe more deterministic) way would you suggest? I'm not 
asking for a complete program architecture here just a half dozen words 
describing how you would have done it.

Thanks for your help.

Best regards,

Viktor STARK
>
>
> -- 
> Best regards,
> Dmitry Adamushko 

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Dmitry Adamushko]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

Hello Viktor,


I do understand the domain separation, but if I got things right, pipes
> can only be read using read() in user space. That's also what the API
> documentation suggests.


Pipes are designed to connect actors from /different/ domains, normally a
flow of data is from primary --> secondary (linux). The actor in the
secondary domain must not be hard real-time latency dependent, for the
obvious reasons I described before.

Take a look at the queue interface (rt_queue_* for the native skin).


1) Why does it happen every 10000 cycles (2 seconds)? The fact that this
> erratic behaviour is so deterministic makes me think that it might not
> be standard Linux related. I mean the fact that it happens every 2
> seconds (and I mean 2e6 us, not just approximately 2 seconds) just makes
> it look like it's provoked by something which is RT.


Maybe yes, maybe no. Some periodic linux activity (ISR, long
interrupt/preemption disabled section, hight priority task) may be a reason
as well.

2) Even if my read() is scheduled, how comes I only get past the read()
> after the second rt_pipe_write()? One would think that if my read() is
> scheduled I could be waking up after 250 us or 300 us, no? But I'm
> *always* woken up after 2 cycles (that is 400 us).
>

Maybe something has to do with your part (driver + rt task) in the primary
domain..

Could you make a small sample program which still shows this "problem" and
post it here?

Additional option is to use the I-pipe tracer as Jan suggested.


3) Pipes seemed to be a good way to transfer data from kernel space.
> What other (maybe more deterministic) way would you suggest? I'm not
> asking for a complete program architecture here just a half dozen words
> describing how you would have done it.


rt_queue_*

Well, you may post your design briefly as well and ask for comments.


Thanks in advance.
>
> Best regards,
>
> Viktor STARK
>

-- 
Best regards,
Dmitry Adamushko

[-- Attachment #2: Type: text/html, Size: 2883 bytes --]

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Philippe Gerum]

On Sat, 2007-03-24 at 11:20 +0100, Dmitry Adamushko wrote:

> 
>         3) Pipes seemed to be a good way to transfer data from kernel
>         space.
>         What other (maybe more deterministic) way would you suggest?
>         I'm not 
>         asking for a complete program architecture here just a half
>         dozen words 
>         describing how you would have done it.
> 
> rt_queue_*
>  

I would add that if you really want to keep the pipe semantics for
exchanging data without crossing domains between real-time tasks, then
you may access the real-time API of message pipes from user-space too.
i.e.
rt_pipe_create/bind/delete/read/write/stream() are all available from a
user-space application, in which case data exchange will happen in
primary mode.

To sum up, given that you want to create a pipe in kernel space, to
exchange data with a user-space program, the options with Xenomai's
message pipes are:

1) Implementing a real-time communication path between two Xenomai
tasks, crossing the kernel/user-space boundary without leaving the
deterministic primary domain: you have to create the kernel-side
endpoint in your module, then bind the user-space endpoint to the latter
using the proper interface, as shown below in the REALTIME_TO_REALTIME
illustration.

2) Implementing a communication path between a real-time Xenomai task in
kernel space, and a regular user-space program: you still have to create
the kernel-side endpoint in your module, then use the standard
open/read/write routines to access to the pipe channel. This is what you
are doing right now.

kernel-side:

void foo_task(void)
{
	for (;;) {
		rt_pipe_read/write(&pipe_desc, ...);
	}
}

int init_module(void)
{
	rt_pipe_create(&pipe_desc, "channel", ...);
}

user-space side:

int main(int argc, char *argv[])
{
#ifdef REALTIME_TO_REALTIME
	/* promote main() to the Xenomai real-time space */
	rt_task_shadow(&task_desc, __FUNCTION__, ...);
	/* bind to the pipe created in kernel space */
	rt_pipe_bind(&pipe_desc, "channel", TM_INFINITE);
	for (;;) {
		rt_pipe_read/write(&pipe_desc, "channel", ...);
	}
#else /* REALTIME_TO_REGULAR */
	fd = open("/proc/xenomai/registry/native/pipes/channel",O_RDWR);
	for (;;) {
		read/write(fd, ...);
	}
#endif
}
-- 
Philippe.

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Viktor STARK]

Philippe Gerum wrote:
> On Sat, 2007-03-24 at 11:20 +0100, Dmitry Adamushko wrote:
>
>   
>>         3) Pipes seemed to be a good way to transfer data from kernel
>>         space.
>>         What other (maybe more deterministic) way would you suggest?
>>         I'm not 
>>         asking for a complete program architecture here just a half
>>         dozen words 
>>         describing how you would have done it.
>>
>> rt_queue_*
>>  
>>     
>
> I would add that if you really want to keep the pipe semantics for
> exchanging data without crossing domains between real-time tasks, then
> you may access the real-time API of message pipes from user-space too.
> i.e.
> rt_pipe_create/bind/delete/read/write/stream() are all available from a
> user-space application, in which case data exchange will happen in
> primary mode.
>   
Now, that is interesting. I was left under the impression that in user 
space, the only option to access pipes was using regular 
open()/read()/write().
I guess I overlooked something in the documentation. Thank you for 
pointing this out.

> To sum up, given that you want to create a pipe in kernel space, to
> exchange data with a user-space program, the options with Xenomai's
> message pipes are:
>
> 1) Implementing a real-time communication path between two Xenomai
> tasks, crossing the kernel/user-space boundary without leaving the
> deterministic primary domain: you have to create the kernel-side
> endpoint in your module, then bind the user-space endpoint to the latter
> using the proper interface, as shown below in the REALTIME_TO_REALTIME
> illustration.
>
> 2) Implementing a communication path between a real-time Xenomai task in
> kernel space, and a regular user-space program: you still have to create
> the kernel-side endpoint in your module, then use the standard
> open/read/write routines to access to the pipe channel. This is what you
> are doing right now.
>
> kernel-side:
>
> void foo_task(void)
> {
> 	for (;;) {
> 		rt_pipe_read/write(&pipe_desc, ...);
> 	}
> }
>
> int init_module(void)
> {
> 	rt_pipe_create(&pipe_desc, "channel", ...);
> }
>
> user-space side:
>
> int main(int argc, char *argv[])
> {
> #ifdef REALTIME_TO_REALTIME
> 	/* promote main() to the Xenomai real-time space */
> 	rt_task_shadow(&task_desc, __FUNCTION__, ...);
> 	/* bind to the pipe created in kernel space */
> 	rt_pipe_bind(&pipe_desc, "channel", TM_INFINITE);
> 	for (;;) {
> 		rt_pipe_read/write(&pipe_desc, "channel", ...);
> 	}
> #else /* REALTIME_TO_REGULAR */
> 	fd = open("/proc/xenomai/registry/native/pipes/channel",O_RDWR);
> 	for (;;) {
> 		read/write(fd, ...);
> 	}
> #endif
> }
>   
That looks like a pretty good solution. I'll be able to test this out on 
Monday. I'll keep the list posted about my results.

Thanks a lot for your enlightening comments / example. It's very much 
appreciated.


Viktor STARK

[off-list] Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Dmitry Adamushko]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

On 24/03/07, Philippe Gerum <rpm@xenomai.org> wrote:
>
> [...]
> I would add that if you really want to keep the pipe semantics for
> exchanging data without crossing domains between real-time tasks, then
> you may access the real-time API of message pipes from user-space too.
> i.e.
> rt_pipe_create/bind/delete/read/write/stream() are all available from a
> user-space application, in which case data exchange will happen in
> primary mode.


rt_pipe_read -> rt_pipe_receive -> xnpipe_recv -> take from state->inq

rt_pipe_write -> rt_pipe_send -> xnpipe_send -> put into state->outq

So there are 2 different queues.

If a user space side makes use of rt_pipe_read(), then a kernel side
(driver) should use
xnpipe_write(), not rt_pipe_*


For data transfer in the opposite direction, [user space] rt_pipe_write()
---> [kernel space] xnpipe_read()


kernel-side:
>
> void foo_task(void)
> {
>         for (;;) {
>                 rt_pipe_read/write(&pipe_desc, ...);
>         }
> }
>
> int init_module(void)
> {
>         rt_pipe_create(&pipe_desc, "channel", ...);
> }
>
> user-space side:
>
> int main(int argc, char *argv[])
> {
> #ifdef REALTIME_TO_REALTIME
>         /* promote main() to the Xenomai real-time space */
>         rt_task_shadow(&task_desc, __FUNCTION__, ...);
>         /* bind to the pipe created in kernel space */
>         rt_pipe_bind(&pipe_desc, "channel", TM_INFINITE);
>         for (;;) {
>                 rt_pipe_read/write(&pipe_desc, "channel", ...);
>         }
> #else /* REALTIME_TO_REGULAR */
>         fd = open("/proc/xenomai/registry/native/pipes/channel",O_RDWR);
>         for (;;) {
>                 read/write(fd, ...);
>         }
> #endif
> }
> --
> Philippe.
>
>
>


-- 
Best regards,
Dmitry Adamushko

[-- Attachment #2: Type: text/html, Size: 3187 bytes --]

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Philippe Gerum]

On Sat, 2007-03-24 at 13:07 +0100, Viktor STARK wrote:
> >   
> That looks like a pretty good solution. I'll be able to test this out on 
> Monday. I'll keep the list posted about my results.

Yeah, well, except that my last post was utter crap actually. I should
really consider having sleep because I don't even remember how my code
works, it seems. Ok, here is a fixed version: to sum up, you need to use
rt_queues to do what you want to do, i.e. exchanging data in primary
mode between real-time tasks. Performance-wise, there is no downside in
using queues instead of pipes anyway, except when rt_pipe_stream() is
actually appropriate for the sending side (i.e. when data is produced
incrementally by small chunks, and the reader has lower priority than
the sender).

What I sketched would not work because rt_pipe_stream/read/write used
from user-space would still feed the real-time endpoint of a pipe, which
could only be read from a non real-time endpoint. IOW, there is no way
to use pipes for RT to RT communication, because data sent using
rt_pipe_write() is only readable by read(), and data wrote by write() is
only readable from rt_pipe_read(). The rt_pipe API is available from
user-space in order to feed regular Linux tasks with real-time data
produced from a user-space context, that's all.

Sorry for the noise. Yes Dmitry, I'm going to take a nap right now, in
order to fix this annoying -ENOBRAIN error that surges from time to
time. Gasp...

-- 
Philippe.

Re: [Xenomai-help] [RT_PIPE] - User space read() not waking up

[Viktor STARK]

Philippe Gerum wrote:
> On Sat, 2007-03-24 at 13:07 +0100, Viktor STARK wrote:
>   
>>>   
>>>       
>> That looks like a pretty good solution. I'll be able to test this out on 
>> Monday. I'll keep the list posted about my results.
>>     
>
> Yeah, well, except that my last post was utter crap actually. I should
> really consider having sleep because I don't even remember how my code
> works, it seems. Ok, here is a fixed version: to sum up, you need to use
> rt_queues to do what you want to do, i.e. exchanging data in primary
> mode between real-time tasks. Performance-wise, there is no downside in
> using queues instead of pipes anyway, except when rt_pipe_stream() is
> actually appropriate for the sending side (i.e. when data is produced
> incrementally by small chunks, and the reader has lower priority than
> the sender).
>
>   
I knew it was too good to be true. ;)
> What I sketched would not work because rt_pipe_stream/read/write used
> >from user-space would still feed the real-time endpoint of a pipe, which
> could only be read from a non real-time endpoint. 
Right. I guess feeding the same endpoint all the time won't really make 
both sides communicate.
I knew that there was some reason why I hadn't even tried rt_pipe_* in 
user space. Thanks for reminding me what that reason was. :)
> IOW, there is no way
> to use pipes for RT to RT communication, because data sent using
> rt_pipe_write() is only readable by read(), and data wrote by write() is
> only readable from rt_pipe_read(). The rt_pipe API is available from
> user-space in order to feed regular Linux tasks with real-time data
> produced from a user-space context, that's all.
>
> Sorry for the noise. Yes Dmitry, I'm going to take a nap right now, in
> order to fix this annoying -ENOBRAIN error that surges from time to
> time. Gasp...
>
>   

OK. So off I go with rt_queue_*. I'll keep the list posted.

Thanks again for all the support.

Viktor STARK