[Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Roberto Bielli]

Hi,

sometimes i have a strange behaviour on boot. The kernel hangs. I saw 
with an hardware debugger the backtrace when i was in that situation and 
i saw this:

probe_irq_off | devm_free_irq
probe_irq_on
probe_irq_on
register_handler_proc
timecounter_read
timer_list_show
posix_cpu_timer_set
stk5_esdhci_init
load_module
exit_irq_thread
__setup_str_setup_slub_nomerge
__setup_str_setup_slub_nomerge
probe_irq_on
probe_irq_on
register_handler_proc
__xnpod_schedule
xnpipe_recv
rt_task_set_priority

The system is in loop/hangs in rt_task_set_priority.

My system has xenomai 2.5.6 on freescale imx25 arm.

Any ideas ?

Thanks for all

-- 
+------------------------------------------------------------------------------------------------+
							
Roberto Bielli
Sviluppo Software					
Axel S.r.l.					
							
Via Del Cannino, 3 				
21020 Crosio Della Valle			
Varese - Italy					
							
Telefono: +39 0332 949600			
Fax:      +39 0332 969315			
							
E-mail:   roberto.bielli@axelsw.it	
Web Site: www.axelsw.it

+------------------------------------------------------------------------------------------------+

Si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo del destinatario.
Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a terzi,
dandocene gentilmente comunicazione. Grazie.
Informativa sul trattamento dei dati personali (D. Lgs. 196/2003).
I dati utilizzati per la spedizione del presente messaggio sono utilizzati da Axel S.r.l., titolare del trattamento,
per l'invio delle comunicazioni dei diversi settori aziendali, non essendo autorizzata la divulgazione a terzi.
Potrete rivolgere alla seguente mail richieste di verifica, rettifica o cancellazione dei Vostri dati: info@axelsw.it

This e-mail and any attachments is confidential and may contain privileged information
intended for the addressee(s) only. Dissemination, copying, printing or use by anybody
else is unauthorised. If you are not the intended recipient,
please delete this message and any attachments and advise the sender
by return e-mail.Thank you.  			
							
+------------------------------------------------------------------------------------------------+

Re: [Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Philippe Gerum]

On 12/06/2013 09:03 AM, Roberto Bielli wrote:
> Hi,
>
> sometimes i have a strange behaviour on boot. The kernel hangs. I saw
> with an hardware debugger the backtrace when i was in that situation and
> i saw this:
>
> probe_irq_off | devm_free_irq
> probe_irq_on
> probe_irq_on
> register_handler_proc
> timecounter_read
> timer_list_show
> posix_cpu_timer_set
> stk5_esdhci_init
> load_module
> exit_irq_thread
> __setup_str_setup_slub_nomerge
> __setup_str_setup_slub_nomerge
> probe_irq_on
> probe_irq_on
> register_handler_proc
> __xnpod_schedule
> xnpipe_recv
> rt_task_set_priority
>
> The system is in loop/hangs in rt_task_set_priority.
>
> My system has xenomai 2.5.6 on freescale imx25 arm.
>
> Any ideas ?
>

If changing the priority of a thread makes that thread preempt the 
caller of rt_task_set_priority(), before entering a tight loop 
indefinitely, you would see this behavior. Enabling 
CONFIG_XENO_OPT_WATCHDOG would likely detect such a case.

Typical bug: not properly handling return codes of Xenomai syscalls in 
real-time work loops, such as -EINVAL, -EIDRM, -EINTR or -ETIMEDOUT, 
causing the underlying thread to iterate indefinitely on the failing 
syscall.

-- 
Philippe.

Re: [Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Gilles Chanteperdrix]

On 12/06/2013 09:03 AM, Roberto Bielli wrote:
> Hi,
> 
> sometimes i have a strange behaviour on boot. The kernel hangs. I saw 
> with an hardware debugger the backtrace when i was in that situation and 
> i saw this:
> 
> probe_irq_off | devm_free_irq
> probe_irq_on
> probe_irq_on
> register_handler_proc
> timecounter_read
> timer_list_show
> posix_cpu_timer_set
> stk5_esdhci_init
> load_module
> exit_irq_thread
> __setup_str_setup_slub_nomerge
> __setup_str_setup_slub_nomerge
> probe_irq_on
> probe_irq_on
> register_handler_proc
> __xnpod_schedule
> xnpipe_recv
> rt_task_set_priority

This backtrace does not seem to really make sense. Or maybe it is not a
backtrace?

> 
> The system is in loop/hangs in rt_task_set_priority.
> 
> My system has xenomai 2.5.6 on freescale imx25 arm.
> 
> Any ideas ?

Do you have the same issue with xenomai 2.6.3 ?


-- 
                                                                Gilles.

Re: [Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Roberto Bielli]

Hi Gilles,

i attached an image with the stall. (Excuse me for the image but it's 
impossibile in a fast way to convert into text ).
The old backtrace is invalid. The system cannot be updated.
The new backtrace says that the system loop in __ipipe_walk_pipeline.
It seems that irqpend_imask = 2 and irqpend_lowmask = (0, 
4194304,......) are the problem.
In a good start there arent that values.

Can you help me ?

Thanks a lot

Il 06/12/2013 13:20, Gilles Chanteperdrix ha scritto:
> On 12/06/2013 09:03 AM, Roberto Bielli wrote:
>> Hi,
>>
>> sometimes i have a strange behaviour on boot. The kernel hangs. I saw
>> with an hardware debugger the backtrace when i was in that situation and
>> i saw this:
>>
>> probe_irq_off | devm_free_irq
>> probe_irq_on
>> probe_irq_on
>> register_handler_proc
>> timecounter_read
>> timer_list_show
>> posix_cpu_timer_set
>> stk5_esdhci_init
>> load_module
>> exit_irq_thread
>> __setup_str_setup_slub_nomerge
>> __setup_str_setup_slub_nomerge
>> probe_irq_on
>> probe_irq_on
>> register_handler_proc
>> __xnpod_schedule
>> xnpipe_recv
>> rt_task_set_priority
> This backtrace does not seem to really make sense. Or maybe it is not a
> backtrace?
>
>> The system is in loop/hangs in rt_task_set_priority.
>>
>> My system has xenomai 2.5.6 on freescale imx25 arm.
>>
>> Any ideas ?
> Do you have the same issue with xenomai 2.6.3 ?
>
>


-- 
+------------------------------------------------------------------------------------------------+
							
Roberto Bielli
Sviluppo Software					
Axel S.r.l.					
							
Via Del Cannino, 3 				
21020 Crosio Della Valle			
Varese - Italy					
							
Telefono: +39 0332 949600			
Fax:      +39 0332 969315			
							
E-mail:   roberto.bielli@axelsw.it	
Web Site: www.axelsw.it

+------------------------------------------------------------------------------------------------+

Si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo del destinatario.
Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a terzi,
dandocene gentilmente comunicazione. Grazie.
Informativa sul trattamento dei dati personali (D. Lgs. 196/2003).
I dati utilizzati per la spedizione del presente messaggio sono utilizzati da Axel S.r.l., titolare del trattamento,
per l'invio delle comunicazioni dei diversi settori aziendali, non essendo autorizzata la divulgazione a terzi.
Potrete rivolgere alla seguente mail richieste di verifica, rettifica o cancellazione dei Vostri dati: info@axelsw.it

This e-mail and any attachments is confidential and may contain privileged information
intended for the addressee(s) only. Dissemination, copying, printing or use by anybody
else is unauthorised. If you are not the intended recipient,
please delete this message and any attachments and advise the sender
by return e-mail.Thank you.  			
							
+------------------------------------------------------------------------------------------------+

-------------- next part --------------
A non-text attachment was scrubbed...
Name: stall.jpeg
Type: image/jpeg
Size: 103164 bytes
Desc: not available
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20131220/e2111419/attachment.jpeg>

Re: [Xenomai] rt_task_set_priority hangs/loops sometimes on boot

[Gilles Chanteperdrix]

On 12/20/2013 09:22 AM, Roberto Bielli wrote:
> Hi Gilles,
> 
> i attached an image with the stall. (Excuse me for the image but it's 
> impossibile in a fast way to convert into text ).
> The old backtrace is invalid. The system cannot be updated.

Why is that, you do not have the root password on your test system? The
root filesystem is read-only?

I am just asking you to upgrade your test system, to see if you can
reproduce the bug with the latest version, as a first step, not to
upgrade the systems you have installed in the field. A lot of things
happened since 2.5.6 and whatever linux version you are using, which is
why I would like you to test xenomai 2.6.3 with linux 3.8.13. I do not
have time to spare on an issue which has already been fixed.

> The new backtrace says that the system loop in __ipipe_walk_pipeline.
> It seems that irqpend_imask = 2 and irqpend_lowmask = (0,
> 4194304,......) are the problem.
> In a good start there arent that values.

Using a symbolic debugguer shows you the things by the wrong end of the
problem. It is a bit like looking at the sun with a microscope.
ipipe_walk_pipeline is a function which is called all the time in the
I-pipe patched kernel.

If you feel like debugging the issue you have yourself, instead of
answering my request of upgrading before retesting, my advice would be
that there is no subtitute to really understanding what the code does,
on a macroscopic view, rather than on a miscroscopic view of "variable x
has value y".

The first thing I would try, is to add a printk in the interrupt
controller masking function, and the gpio demux function, if there is
one, to see if either one is not called in a loop.

-- 
                                                                Gilles.