[Xenomai-help] Modeswitch using message queues

[Xenomai-help] Modeswitch using message queues

[Thomas.Debes]

Hello,

I am writing a communication daemon based on Xenomai message queues
(using native skin). This daemon creates a queue und starts listening
for incoming client messages. A client can open a "connection" by
binding to the daemons queue and sending a special message containing
ist own queuename. Thereafter the daemon binds to the client queue and
sends back a response message. "Disconnecting" works the same way.
Everything works as expected, the only thing is that the mode switch
counter in /proc/xenomai/stat increases on each
rt_bind_queue/rt_unbind_queue call (for the daemon and the client). I
tracked this back by adding a signal handler for SIGXCPU and setting the
T_WARNSW bit in my realtime task. The gdb backtrace output shows the
nunmap(..) as last call before the mode switch occurs.

Program received signal SIGXCPU, CPU time limit exceeded.
[Switching to Thread 16386 (LWP 520)]
0x0fddb7d8 in munmap () from /lib/libc.so.6
(gdb) bt
#0  0x0fddb7d8 in munmap () from /lib/libc.so.6
#1  0x0fe829d8 in rt_queue_unbind (q=0x1003a1b6) at queue.c:100
#2  0x100084d4 in remove_connection (client_address=17) at
src/proto_mqc.c:1610
#3  0x10006230 in eval_disconnect_msg (connection=0x30036240,
    in_msg=0x300362b0) at src/proto_mqc.c:793
#4  0x10005cd4 in eval_msg (connection=0x30036240, in_msg=0x300362b0)
    at src/proto_mqc.c:634
#5  0x10005ba8 in proto_mqc_handler (msg=0x300362b0) at
src/proto_mqc.c:598
#6  0x1000399c in dispatch_message (msg=0x300362b0) at
src/dispatch.c:236
#7  0x10003c38 in dispatch (cookie=0x0) at src/dispatch.c:309
#8  0x0fe835f0 in rt_task_trampoline (cookie=0x0) at task.c:95
#9  0x0fba8920 in pthread_start_thread_event (arg=0x30036be0) at
manager.c:300
#10 0x0fddedc4 in clone () from /lib/libc.so.6
(gdb) c


All queue related functions (except create/delete) are called from
realtime tasks. The daemon itself and all other applications are user
space programs. So from my point of view, mode switches shouldn't occur
here.

Thanks for your help

Thomas 

MAN Roland Druckmaschinen AG
Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle    
Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592    
USt-Ident-Nr. DE 250200933

Re: [Xenomai-help] Modeswitch using message queues

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3100 bytes --]

Thomas.Debes@domain.hid wrote:
> Hello,
> 
> I am writing a communication daemon based on Xenomai message queues
> (using native skin). This daemon creates a queue und starts listening
> for incoming client messages. A client can open a "connection" by
> binding to the daemons queue and sending a special message containing
> ist own queuename. Thereafter the daemon binds to the client queue and
> sends back a response message. "Disconnecting" works the same way.
> Everything works as expected, the only thing is that the mode switch
> counter in /proc/xenomai/stat increases on each
> rt_bind_queue/rt_unbind_queue call (for the daemon and the client). I
> tracked this back by adding a signal handler for SIGXCPU and setting the
> T_WARNSW bit in my realtime task. The gdb backtrace output shows the
> nunmap(..) as last call before the mode switch occurs.
> 
> Program received signal SIGXCPU, CPU time limit exceeded.
> [Switching to Thread 16386 (LWP 520)]
> 0x0fddb7d8 in munmap () from /lib/libc.so.6
> (gdb) bt
> #0  0x0fddb7d8 in munmap () from /lib/libc.so.6
> #1  0x0fe829d8 in rt_queue_unbind (q=0x1003a1b6) at queue.c:100
> #2  0x100084d4 in remove_connection (client_address=17) at
> src/proto_mqc.c:1610
> #3  0x10006230 in eval_disconnect_msg (connection=0x30036240,
>     in_msg=0x300362b0) at src/proto_mqc.c:793
> #4  0x10005cd4 in eval_msg (connection=0x30036240, in_msg=0x300362b0)
>     at src/proto_mqc.c:634
> #5  0x10005ba8 in proto_mqc_handler (msg=0x300362b0) at
> src/proto_mqc.c:598
> #6  0x1000399c in dispatch_message (msg=0x300362b0) at
> src/dispatch.c:236
> #7  0x10003c38 in dispatch (cookie=0x0) at src/dispatch.c:309
> #8  0x0fe835f0 in rt_task_trampoline (cookie=0x0) at task.c:95
> #9  0x0fba8920 in pthread_start_thread_event (arg=0x30036be0) at
> manager.c:300
> #10 0x0fddedc4 in clone () from /lib/libc.so.6
> (gdb) c
> 
> 
> All queue related functions (except create/delete) are called from
> realtime tasks. The daemon itself and all other applications are user
> space programs. So from my point of view, mode switches shouldn't occur
> here.

Native queues are designed for efficient data transfer also of larger 
blocks, i.e. without copying. Therefore, when you establish a 
communication channel based on a queue between two processes, both have 
to map the underlying buffer into their address spaces. This means 
leaving RT land.

I think you have to redesign your communication protocol with this fact 
in mind, so that you can avoiding continuous reconnections. Those which 
are still required for clients joining the party, could be handled by 
splitting up the daemon into two threads: one for time-critical 
communication, the other for low-prio connection setup/cleanup.

An alternative would be an IPC scheme that requires at least one 
copy-step (from the sender's into the receiver's buffer). We once did 
this for the RACK RT-middleware based on RTDM sockets [1] (it's a bit 
"specialised").

Jan

[1] http://svn.berlios.de/viewcvs/rack/trunk/rack/main/tims/driver/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Modeswitch using message queues

[Thomas.Debes]

> -----Ursprüngliche Nachricht-----
> Von: jan.kiszka@domain.hid [mailto:jan.kiszka@domain.hid
> Gesendet: Donnerstag, 17. Januar 2008 22:12
> An: Debes, Thomas RAEK3 MRA
> Cc: xenomai@xenomai.org
> Betreff: Re: [Xenomai-help] Modeswitch using message queues
> 
> Thomas.Debes@domain.hid wrote:
> > Hello,
> > 
> > I am writing a communication daemon based on Xenomai message queues 
> > (using native skin). This daemon creates a queue und starts 
> listening 
> > for incoming client messages. A client can open a "connection" by 
> > binding to the daemons queue and sending a special message 
> containing 
> > ist own queuename. Thereafter the daemon binds to the 
> client queue and 
> > sends back a response message. "Disconnecting" works the same way.
> > Everything works as expected, the only thing is that the 
> mode switch 
> > counter in /proc/xenomai/stat increases on each 
> > rt_bind_queue/rt_unbind_queue call (for the daemon and the 
> client). I 
> > tracked this back by adding a signal handler for SIGXCPU 
> and setting 
> > the T_WARNSW bit in my realtime task. The gdb backtrace 
> output shows 
> > the
> > nunmap(..) as last call before the mode switch occurs.
> > 
> > Program received signal SIGXCPU, CPU time limit exceeded.
> > [Switching to Thread 16386 (LWP 520)]
> > 0x0fddb7d8 in munmap () from /lib/libc.so.6
> > (gdb) bt
> > #0  0x0fddb7d8 in munmap () from /lib/libc.so.6
> > #1  0x0fe829d8 in rt_queue_unbind (q=0x1003a1b6) at queue.c:100
> > #2  0x100084d4 in remove_connection (client_address=17) at 
> > src/proto_mqc.c:1610
> > #3  0x10006230 in eval_disconnect_msg (connection=0x30036240,
> >     in_msg=0x300362b0) at src/proto_mqc.c:793
> > #4  0x10005cd4 in eval_msg (connection=0x30036240, 
> in_msg=0x300362b0)
> >     at src/proto_mqc.c:634
> > #5  0x10005ba8 in proto_mqc_handler (msg=0x300362b0) at
> > src/proto_mqc.c:598
> > #6  0x1000399c in dispatch_message (msg=0x300362b0) at
> > src/dispatch.c:236
> > #7  0x10003c38 in dispatch (cookie=0x0) at src/dispatch.c:309
> > #8  0x0fe835f0 in rt_task_trampoline (cookie=0x0) at task.c:95
> > #9  0x0fba8920 in pthread_start_thread_event (arg=0x30036be0) at 
> > manager.c:300 #10 0x0fddedc4 in clone () from /lib/libc.so.6
> > (gdb) c
> > 
> > 
> > All queue related functions (except create/delete) are called from 
> > realtime tasks. The daemon itself and all other 
> applications are user 
> > space programs. So from my point of view, mode switches shouldn't 
> > occur here.
> 
> Native queues are designed for efficient data transfer also 
> of larger blocks, i.e. without copying. Therefore, when you 
> establish a communication channel based on a queue between 
> two processes, both have to map the underlying buffer into 
> their address spaces. This means leaving RT land.
> 
> I think you have to redesign your communication protocol with 
> this fact in mind, so that you can avoiding continuous 
> reconnections. Those which are still required for clients 
> joining the party, could be handled by splitting up the 
> daemon into two threads: one for time-critical communication, 
> the other for low-prio connection setup/cleanup.
> 
> An alternative would be an IPC scheme that requires at least 
> one copy-step (from the sender's into the receiver's buffer). 
> We once did this for the RACK RT-middleware based on RTDM 
> sockets [1] (it's a bit "specialised").
> 
> Jan
> 
> [1] http://svn.berlios.de/viewcvs/rack/trunk/rack/main/tims/driver/
>

Thanks for your help, I'll rewrite the connection setup stuff by putting it into a separate task.

Regards
Thomas 

MAN Roland Druckmaschinen AG
Vorsitzender des Aufsichtsrates: Hanno C. Fiedler
Vorstand: Gerd Finkbeiner (Vorsitzender), Dr. Ingo Koch, Dr. Markus Rall, Paul Steidle    
Sitz der Gesellschaft: Offenbach am Main, Registergericht: Amtsgericht Offenbach HRB-Nr. 42592    
USt-Ident-Nr. DE 250200933