Re: [Xenomai] resource leak and kernel panic with pSOS q_vcreate on arm (BeagleBone Black)

[Marcel van Mierlo <marcel.vanmierlo@marel.com>]

On 25/02/14 12:15, Marcel van Mierlo wrote:
> Hi,
>
> I posted this information a few weeks back but got no response...does 
> anyone have insight on this...I've been digging into the queue 
> management code in Xenomai but I'm not getting very far...any help on 
> this would be appreciated as I am not sure what to try next.
>
> This behaviour has been observed on BeagleBoard Black Linux arm 
> 3.8.13-bone26.1, Xenomai 2.6.3
>
> basically the resources created by pSOS q_vcreate do not appear to be 
> released when the application exits. q_create appears to be fine. When 
> the application starts up again after exit (without calling 
> q_vdelete), calling q_vcreate on the named queue returns (-17 File 
> Exists). If I instead use q_vident to get a handle on the queue, the 
> API appears to return with a valid handle, but when the program exits 
> the second time Xenomai appears to sit in a resource-release loop and 
> I end up with a kernel panic. This is a non-trivial application with 
> about 30 running tasks and lots of fixed size queues using CAN. The 
> system is stable except for v-queues so there certainly seems to be 
> something amiss here. I have included a trivial test case which 
> reproduces the problem.
>
> [  144.346739] Kernel panic - not syncing: softlockup: hung tasks
> [  144.352845] [<c00138dc>] (unwind_backtrace+0x0/0xe0) from 
> [<c06b0af0>] (panic+0x84/0x1e0)
> [  144.361426] [<c06b0af0>] (panic+0x84/0x1e0) from [<c0094724>] 
> (watchdog_timer_fn+0x120/0x164)
> [  144.370338] [<c0094724>] (watchdog_timer_fn+0x120/0x164) from 
> [<c005e1c4>] (__run_hrtimer+0xec/0x1e4)
> [  144.379976] [<c005e1c4>] (__run_hrtimer+0xec/0x1e4) from 
> [<c005ec00>] (hrtimer_interrupt+0x108/0x25c)
> [  144.389614] [<c005ec00>] (hrtimer_interrupt+0x108/0x25c) from 
> [<c0025f88>] (omap2_gp_timer_interrupt+0x44/0x54)
> [  144.400162] [<c0025f88>] (omap2_gp_timer_interrupt+0x44/0x54) from 
> [<c0095144>] (handle_irq_event_percpu+0x60/0x214)
> [  144.411156] [<c0095144>] (handle_irq_event_percpu+0x60/0x214) from 
> [<c0095334>] (handle_irq_event+0x3c/0x5c)
> [  144.421430] [<c0095334>] (handle_irq_event+0x3c/0x5c) from 
> [<c0098260>] (handle_level_irq+0x98/0xd0)
> [  144.430974] [<c0098260>] (handle_level_irq+0x98/0xd0) from 
> [<c0094b7c>] (generic_handle_irq+0x20/0x30)
> [  144.440700] [<c0094b7c>] (generic_handle_irq+0x20/0x30) from 
> [<c000e45c>] (handle_IRQ+0x64/0x8c)
> [  144.449886] [<c000e45c>] (handle_IRQ+0x64/0x8c) from [<c009e594>] 
> (__ipipe_do_sync_stage+0x1f4/0x220)
> [  144.459526] [<c009e594>] (__ipipe_do_sync_stage+0x1f4/0x220) from 
> [<c009e710>] (ipipe_unstall_root+0x30/0x3c)
> [  144.469887] [<c009e710>] (ipipe_unstall_root+0x30/0x3c) from 
> [<c003e908>] (vprintk_emit+0x444/0x4bc)
> [  144.479428] [<c003e908>] (vprintk_emit+0x444/0x4bc) from 
> [<c003eb24>] (vprintk+0x1c/0x20)
> [  144.487974] [<c003eb24>] (vprintk+0x1c/0x20) from [<c06b0d14>] 
> (printk+0xa4/0x144)
> [  144.495891] [<c06b0d14>] (printk+0xa4/0x144) from [<c0149638>] 
> (psos_shadow_eventcb+0x7f8/0x16b8)
> [  144.505169] [<c0149638>] (psos_shadow_eventcb+0x7f8/0x16b8) from 
> [<c00db35c>] (detach_ppd+0x28/0x44)
> [  144.514718] [<c00db35c>] (detach_ppd+0x28/0x44) from [<c00dce74>] 
> (taskexit_event+0x5ec/0x884)
> [  144.523724] [<c00dce74>] (taskexit_event+0x5ec/0x884) from 
> [<c009fb14>] (ipipe_kevent_hook+0x24/0x2c)
> [  144.533367] [<c009fb14>] (ipipe_kevent_hook+0x24/0x2c) from 
> [<c009e324>] (__ipipe_notify_kevent+0x3c/0x64)
> [  144.543457] [<c009e324>] (__ipipe_notify_kevent+0x3c/0x64) from 
> [<c004193c>] (do_exit+0x348/0x88c)
> [  144.552819] [<c004193c>] (do_exit+0x348/0x88c) from [<c0041f30>] 
> (do_group_exit+0x84/0xc0)
> [  144.561465] [<c0041f30>] (do_group_exit+0x84/0xc0) from 
> [<c004e938>] (get_signal_to_deliver+0x49c/0x508)
> [  144.571375] [<c004e938>] (get_signal_to_deliver+0x49c/0x508) from 
> [<c0010a7c>] (do_signal+0x90/0x478)
> [  144.581018] [<c0010a7c>] (do_signal+0x90/0x478) from [<c0011350>] 
> (do_work_pending+0x5c/0xa4)
> [  144.589966] [<c0011350>] (do_work_pending+0x5c/0xa4) from 
> [<c000dba4>] (work_pending+0xc/0x20)
> [  144.598961] drm_kms_helper: panic occurred, switching back to text 
> console
>
> In summary:
>
>   1. "USED main heap" under /proc/xenomai/heap does not return to 
> previous level after terminating an application which uses q_vcreate - 
> it is increased.
>   2. After (1) has occurred, calling q_vcreate in a separate 
> invocation of the application returns -17 (File Exists) - suggesting 
> the queue is still in existence. This happens with plenty of "main 
> heap" left. I would expect all resources associated with the queue to 
> be released on termination. "USED main heap" is increased after 
> termination for this scenario as well, which certainly seems wrong.
>
>
> [  677.528359] Xenomai: pSOS: cleaning up q "q002" (ret=58).
> whenever exiting without calling q_vdelete after q_vcreate.
>
> When using q_create, on exit I get:
> [ 445.419293] Xenomai: pSOS: cleaning up q "q001" (ret=0).
> when I do not call q_delete.
>
> I've included a simple test case which demonstrates the problem.
>
> Any assistance with this would be appreciated.
>
>
> Marcel.
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: qvcreatetest.tar.gz
> Type: application/x-gzip
> Size: 625 bytes
> Desc: not available
> URL: 
> <http://www.xenomai.org/pipermail/xenomai/attachments/20140225/8032fb74/attachment.bin>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
>

OK, digging a little further...If I check for the v-queues existence at 
start up (before attempting to create it) using q_vident and then *do 
not* call q_vdelete on it I seem to be able to avoid the kernel panic 
and resource leak...not sure whether the v-queue is actually usable the 
second time around yet....but it seems that it may get created ok, just 
messed up at point of deletion.

Marcel.