[Xenomai] issues with debugging enabled

[Xenomai] issues with debugging enabled

[Gilles Chanteperdrix]

Hi,

for a change, I ran the xeno-regression-test with a lot of debugging and
options known to having caused problems in the past and found two issues:

on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
series of warnings at the end of the mail.

on ARM, when a fault occurs, the fault ode is entered with hardware irqs
off (this is a recent change in the mainline kernel, this code used to
be executed with hardware irqs on), so I do:

ipipe_stall_root();
hard_local_irq_enabled();

But the context checking does not like that.

Any idea how to fix these?


------------[ cut here ]------------
WARNING: at kernel/rcutree.c:388 rcu_eqs_enter+0x4e/0x8d()
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffff88007ed13e78 ffffffff813f7730 ffff88007ed13eb8
 ffffffff8102c006 0000000000000184 0000000000000000 ffff88007f30d580
 ffff88007ed13f00 0000000000000000 0000000000000000 ffff88007ed13ec8
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81078f1c>] rcu_eqs_enter+0x4e/0x8d
 [<ffffffff81078f76>] rcu_idle_enter+0x1b/0x27
 [<ffffffff8105db0e>] cpu_startup_entry+0xcb/0x151
 [<ffffffff816d4766>] start_secondary+0x1a0/0x1a4
---[ end trace d49a723e441bc5a7 ]---
------------[ cut here ]------------
WARNING: at kernel/rcutree.c:363 rcu_eqs_enter_common.isra.42+0xc0/0xd3()
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffff88007ed13e38 ffffffff813f7730 ffff88007ed13e78
 ffffffff8102c006 000000000000016b 0000000000000000 ffff88007ecf37d0
 ffff88007f30d580 0000000000000000 0000000000000000 ffff88007ed13e88
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81078ebb>] rcu_eqs_enter_common.isra.42+0xc0/0xd3
 [<ffffffff81078e00>] ? rcu_eqs_enter_common.isra.42+0x5/0xd3
 [<ffffffff81078f56>] rcu_eqs_enter+0x88/0x8d
 [<ffffffff81078f76>] rcu_idle_enter+0x1b/0x27
 [<ffffffff8105db0e>] cpu_startup_entry+0xcb/0x151
 [<ffffffff816d4766>] start_secondary+0x1a0/0x1a4
---[ end trace d49a723e441bc5a8 ]---
------------[ cut here ]------------
WARNING: at kernel/rcutree.c:480 rcu_irq_exit+0x49/0x70()
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffff88007ed13e08 ffffffff813f7730 ffff88007ed13e48
 ffffffff8102c006 00000000000001e0 0000000000000000 ffff88007f30d580
 0000000000000000 0000000000001142 0000000000000000 ffff88007ed13e58
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81079f06>] rcu_irq_exit+0x49/0x70
 [<ffffffff81032c63>] irq_exit+0xb2/0xb6
 [<ffffffff8107e316>] __ipipe_do_sync_stage+0x11e/0x16c
 [<ffffffff8101c3f8>] __ipipe_halt_root+0x23/0x38
 [<ffffffff81009383>] default_idle+0x1f/0x30
 [<ffffffff81009a74>] arch_cpu_idle+0xf/0x11
 [<ffffffff8105db13>] cpu_startup_entry+0xd0/0x151
 [<ffffffff816d4766>] start_secondary+0x1a0/0x1a4
---[ end trace d49a723e441bc5a9 ]---
------------[ cut here ]------------
WARNING: at kernel/rcutree.c:528 rcu_eqs_exit+0x4b/0x91()
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffff88007ed13e68 ffffffff813f7730 ffff88007ed13ea8
 ffffffff8102c006 0000000000000210 0000000000000000 ffff88007f30d580
 ff00000000000000 0000000000000000 0000000000000000 ffff88007ed13eb8
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81078d8d>] rcu_eqs_exit+0x4b/0x91
 [<ffffffff81078dee>] rcu_idle_exit+0x1b/0x28
 [<ffffffff8105db62>] cpu_startup_entry+0x11f/0x151
 [<ffffffff816d4766>] start_secondary+0x1a0/0x1a4
---[ end trace d49a723e441bc5aa ]---
------------[ cut here ]------------
WARNING: at kernel/rcutree.c:502 rcu_eqs_exit_common.isra.40+0x41/0xce()
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffff88007ed13e38 ffffffff813f7730 ffff88007ed13e78
 ffffffff8102c006 00000000000001f6 0000000000000000 ffff88007f30d580
 0000000000000000 0000000000000000 0000000000000000 ffff88007ed13e88
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81078cb5>] rcu_eqs_exit_common.isra.40+0x41/0xce
 [<ffffffff81078dcb>] rcu_eqs_exit+0x89/0x91
 [<ffffffff81078dee>] rcu_idle_exit+0x1b/0x28
 [<ffffffff8105db62>] cpu_startup_entry+0x11f/0x151
 [<ffffffff816d4766>] start_secondary+0x1a0/0x1a4
---[ end trace d49a723e441bc5ab ]---
WARNING: at kernel/rcutree.c:388 rcu_eqs_enter+0x4e/0x8d()
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
 ffffffff8157e76d ffffffff81603e98 ffffffff813f7730 ffffffff81603ed8
 ffffffff8102c006 0000000000000184 0000000000000000 ffff88007f20d580
 ffffffff81603f00 ffffffff817230a0 ffff88007f64be80 ffffffff81603ee8
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff81078f1c>] rcu_eqs_enter+0x4e/0x8d
 [<ffffffff81078f76>] rcu_idle_enter+0x1b/0x27
 [<ffffffff8105db0e>] cpu_startup_entry+0xcb/0x151
 [<ffffffff813f123c>] rest_init+0x80/0x84
 [<ffffffff8169ec65>] start_kernel+0x322/0x330
 [<ffffffff8169e749>] ? repair_env_string+0x5a/0x5a
 [<ffffffff8169e481>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff8169e54b>] x86_64_start_kernel+0xc8/0xcc
---[ end trace d49a723e441bc5ac ]---
WARNING: at kernel/context_tracking.c:54 user_enter+0x5f/0xb2()
Modules linked in:
CPU: 1 PID: 356 Comm: kworker/1:1 Tainted: G        W    3.10.18 #3
Hardware name:  /, BIOS
Workqueue: events cache_reap
 ffffffff815818be ffff88007cb3bc68 ffffffff813f7730 ffff88007cb3bca8
 ffffffff8102c006 0000000000000036 0000000000000000 0000000000000000
 0000000000000287 ffffffff817bbe80 0000000000000000 ffff88007cb3bcb8
Call Trace:
 [<ffffffff813f7730>] dump_stack+0x19/0x1b
 [<ffffffff8102c006>] warn_slowpath_common+0x68/0x81
 [<ffffffff8102c039>] warn_slowpath_null+0x1a/0x1c
 [<ffffffff810a0a0d>] user_enter+0x5f/0xb2
 [<ffffffff813fad64>] preempt_schedule_irq+0x6c/0x71
 [<ffffffff813fb1e9>] __ipipe_preempt_schedule_irq+0x38/0x78
 [<ffffffff813fbef5>] retint_kernel+0x35/0x40
 [<ffffffff81213b4f>] ? debug_smp_processor_id+0x97/0x198
 [<ffffffff81118580>] cache_reap+0x3e/0x110
 [<ffffffff81045355>] process_one_work+0x188/0x29c
 [<ffffffff8104583a>] worker_thread+0x133/0x200
 [<ffffffff81045707>] ? rescuer_thread+0x26f/0x26f
 [<ffffffff8104a380>] kthread+0xa0/0xa8
 [<ffffffff8104a2e0>] ? __kthread_parkme+0x65/0x65
 [<ffffffff813fc6ad>] ret_from_fork+0x7d/0xb0
 [<ffffffff8104a2e0>] ? __kthread_parkme+0x65/0x65
---[ end trace d49a723e441bc5ad ]---


-- 
                                                                Gilles.

Re: [Xenomai] issues with debugging enabled

[Philippe Gerum]

On 11/12/2013 09:34 PM, Gilles Chanteperdrix wrote:
>
> Hi,
>
> for a change, I ran the xeno-regression-test with a lot of debugging and
> options known to having caused problems in the past and found two issues:
>
> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
> series of warnings at the end of the mail.

Our IRQ deferral might conflict with the RCU state logic (e.g. 
rcu-irq_enter/exit).

>
> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
> off (this is a recent change in the mainline kernel, this code used to
> be executed with hardware irqs on), so I do:
>
> ipipe_stall_root();
> hard_local_irq_enabled();
>
> But the context checking does not like that.
>

You mean ipipe_root_only() triggers over the stall point?

-- 
Philippe.

Re: [Xenomai] issues with debugging enabled

[Gilles Chanteperdrix]

On 11/13/2013 09:23 AM, Philippe Gerum wrote:
> On 11/12/2013 09:34 PM, Gilles Chanteperdrix wrote:
>>
>> Hi,
>>
>> for a change, I ran the xeno-regression-test with a lot of debugging and
>> options known to having caused problems in the past and found two issues:
>>
>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>> series of warnings at the end of the mail.
> 
> Our IRQ deferral might conflict with the RCU state logic (e.g. 
> rcu-irq_enter/exit).
> 
>>
>> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
>> off (this is a recent change in the mainline kernel, this code used to
>> be executed with hardware irqs on), so I do:
>>
>> ipipe_stall_root();
>> hard_local_irq_enabled();
>>
>> But the context checking does not like that.
>>
> 
> You mean ipipe_root_only() triggers over the stall point?
> 
Yes, because ipipe_stall_root() is called with hw irqs off if I
understand correctly.

-- 
                                                                Gilles.

Re: [Xenomai] issues with debugging enabled

[Philippe Gerum]

On 11/13/2013 09:37 AM, Gilles Chanteperdrix wrote:
> On 11/13/2013 09:23 AM, Philippe Gerum wrote:
>> On 11/12/2013 09:34 PM, Gilles Chanteperdrix wrote:
>>>
>>> Hi,
>>>
>>> for a change, I ran the xeno-regression-test with a lot of debugging and
>>> options known to having caused problems in the past and found two issues:
>>>
>>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>>> series of warnings at the end of the mail.
>>
>> Our IRQ deferral might conflict with the RCU state logic (e.g.
>> rcu-irq_enter/exit).
>>
>>>
>>> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
>>> off (this is a recent change in the mainline kernel, this code used to
>>> be executed with hardware irqs on), so I do:
>>>
>>> ipipe_stall_root();
>>> hard_local_irq_enabled();
>>>
>>> But the context checking does not like that.
>>>
>>
>> You mean ipipe_root_only() triggers over the stall point?
>>
> Yes, because ipipe_stall_root() is called with hw irqs off if I
> understand correctly.
>

This means that there is a discrepancy between the hw masking state and 
the logical one via the stall bit. This might also mean that we fault 
into a (root-)stalled area, or hopefully that some fixup already 
happened on the stall bit from the fault trampoline in the lowest core 
code. Could that be?

Also, we may not stall using the regular interface over non-root domains.

-- 
Philippe.

Re: [Xenomai] issues with debugging enabled

[Jan Kiszka]

On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
> 
> Hi,
> 
> for a change, I ran the xeno-regression-test with a lot of debugging and
> options known to having caused problems in the past and found two issues:
> 
> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
> series of warnings at the end of the mail.

We are currently facing crashes inside __switch_to, FPU related
(non-existing save area). Once that is sorted out, we can have a look at
that scenario as well. Do you have the .config online somewhere?

> 
> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
> off (this is a recent change in the mainline kernel, this code used to
> be executed with hardware irqs on), so I do:
> 
> ipipe_stall_root();
> hard_local_irq_enabled();
> 
> But the context checking does not like that.
> 
> Any idea how to fix these?

Only ARM's SMP version of ipipe_stall_root() contains
ipipe_stall_root(), UP and at least x86 do not have this. I understand
that it makes sense to deny this call over non-root, but as it is about
fixing up imbalances in the pipeline state, ipipe_stall_root() is likely
not the best choice. What about open-coding the relevant check for the
domain? Maybe it also works to push the check to the end of
ipipe_stall_root, i.e. after fixing up the pipeline state.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] issues with debugging enabled

[Gilles Chanteperdrix]

On 11/13/2013 02:25 PM, Jan Kiszka wrote:
> On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
>>
>> Hi,
>>
>> for a change, I ran the xeno-regression-test with a lot of debugging and
>> options known to having caused problems in the past and found two issues:
>>
>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>> series of warnings at the end of the mail.
>
> We are currently facing crashes inside __switch_to, FPU related
> (non-existing save area).

Probably the issue solved by this patch:
http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=9b3320eef67e3f118f96065c8c2584b715710d94

 > Once that is sorted out, we can have a look at
 > that scenario as well. Do you have the .config online somewhere?

http://sisyphus.hd.free.fr/~gilles/config-nestor

>> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
>> off (this is a recent change in the mainline kernel, this code used to
>> be executed with hardware irqs on), so I do:
>>
>> ipipe_stall_root();
>> hard_local_irq_enabled();
>>
>> But the context checking does not like that.
>>
>> Any idea how to fix these?
>
> Only ARM's SMP version of ipipe_stall_root() contains
> ipipe_stall_root(), UP and at least x86 do not have this. I understand
> that it makes sense to deny this call over non-root, but as it is about
> fixing up imbalances in the pipeline state, ipipe_stall_root() is likely
> not the best choice. What about open-coding the relevant check for the
> domain? Maybe it also works to push the check to the end of
> ipipe_stall_root, i.e. after fixing up the pipeline state.

I have to run again the test and check what really happens, because like 
Philippe said, it may be that I am using ipipe_stall_root() when 
entering this code over non root context, and that would be incorrect.
I have to report the trap to the pipeline first so as to give a chance 
to Xenomai to relax the thread, then enable irqs over root.

-- 
					    Gilles.

Re: [Xenomai] issues with debugging enabled

[Jan Kiszka]

On 2013-11-13 14:41, Gilles Chanteperdrix wrote:
> On 11/13/2013 02:25 PM, Jan Kiszka wrote:
>> On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
>>>
>>> Hi,
>>>
>>> for a change, I ran the xeno-regression-test with a lot of debugging and
>>> options known to having caused problems in the past and found two
>>> issues:
>>>
>>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>>> series of warnings at the end of the mail.
>>
>> We are currently facing crashes inside __switch_to, FPU related
>> (non-existing save area).
> 
> Probably the issue solved by this patch:
> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=9b3320eef67e3f118f96065c8c2584b715710d94

Thought I had everything, but I lost track, and it's missing here. Will
merge and give it a try.

> 
> 
>> Once that is sorted out, we can have a look at
>> that scenario as well. Do you have the .config online somewhere?
> 
> http://sisyphus.hd.free.fr/~gilles/config-nestor

OK, thanks.

> 
>>> on ARM, when a fault occurs, the fault ode is entered with hardware irqs
>>> off (this is a recent change in the mainline kernel, this code used to
>>> be executed with hardware irqs on), so I do:
>>>
>>> ipipe_stall_root();
>>> hard_local_irq_enabled();
>>>
>>> But the context checking does not like that.
>>>
>>> Any idea how to fix these?
>>
>> Only ARM's SMP version of ipipe_stall_root() contains
>> ipipe_stall_root(), UP and at least x86 do not have this. I understand
>> that it makes sense to deny this call over non-root, but as it is about
>> fixing up imbalances in the pipeline state, ipipe_stall_root() is likely
>> not the best choice. What about open-coding the relevant check for the
>> domain? Maybe it also works to push the check to the end of
>> ipipe_stall_root, i.e. after fixing up the pipeline state.
> 
> I have to run again the test and check what really happens, because like
> Philippe said, it may be that I am using ipipe_stall_root() when
> entering this code over non root context, and that would be incorrect.
> I have to report the trap to the pipeline first so as to give a chance
> to Xenomai to relax the thread, then enable irqs over root.
> 

OK.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] issues with debugging enabled

[Jan Kiszka]

On 2013-11-13 14:53, Jan Kiszka wrote:
> On 2013-11-13 14:41, Gilles Chanteperdrix wrote:
>> On 11/13/2013 02:25 PM, Jan Kiszka wrote:
>>> On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
>>>>
>>>> Hi,
>>>>
>>>> for a change, I ran the xeno-regression-test with a lot of debugging and
>>>> options known to having caused problems in the past and found two
>>>> issues:
>>>>
>>>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>>>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>>>> series of warnings at the end of the mail.
>>>
>>> We are currently facing crashes inside __switch_to, FPU related
>>> (non-existing save area).
>>
>> Probably the issue solved by this patch:
>> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=9b3320eef67e3f118f96065c8c2584b715710d94
> 
> Thought I had everything, but I lost track, and it's missing here. Will
> merge and give it a try.

OK, this fixes the crash in __switch_to, triggerable by terminating
xeno-test while running clocktest. But we still face a crash when
terminating switchtest early. Are you aware of this?

[   59.563411] [Xenomai] switching rtk1/0 to secondary mode after exception #14 in kernel-space at 0xffffffff815276f4 (pid 4371)
[   59.565408] BUG: unable to handle kernel paging request at ffffd2000235a704
[   59.566032] IP: [<ffffffff815276f4>] rtswitch_to_rt+0x384/0x540
[   59.566039] PGD 0 
[   59.566041] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[   59.566044] Modules linked in: 9p fscache xt_tcpudp xt_limit xt_pkttype ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_CT ipt_REJECT xt_state iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables ip6table_filter ip6_tables x_tables dm_crypt e1000 psmouse 9pnet_virtio 9pnet tpm_tis intel_agp i2c_piix4 tpm floppy serio_raw intel_gtt microcode tpm_bios pcspkr virtio_pci virtio_blk virtio virtio_ring ahci libahci
[   59.566109] CPU: 0 PID: 4371 Comm: rtk1/0 Not tainted 3.10.15+ #42
[   59.566111] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   59.566113] task: ffff88003c187500 ti: ffff88003c4bc000 task.ti: ffff88003c4bc000
[   59.566119] RIP: 0010:[<ffffffff815276f4>]  [<ffffffff815276f4>] rtswitch_to_rt+0x384/0x540
[   59.566121] RSP: 0000:ffff88003c4bfd98  EFLAGS: 00010206
[   59.566123] RAX: 000008fffffff700 RBX: ffff88003af14038 RCX: ffffc9000235b000
[   59.566125] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: ffff88003af14038
[   59.566127] RBP: ffff88003c4bfde8 R08: 0000000000000000 R09: 00000000002a1220
[   59.566129] R10: 0000000000000000 R11: ffff88003d400000 R12: ffffc9000235c200
[   59.566131] R13: ffffd2000235a700 R14: ffff88003af14038 R15: 0000000000000000
[   59.566134] FS:  0000000000000000(0000) GS:ffff88003d400000(0000) knlGS:0000000000000000
[   59.566136] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   59.566138] CR2: ffffd2000235a704 CR3: 000000003c6f2000 CR4: 00000000000006f0
[   59.566147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   59.566154] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   59.566155] I-pipe domain Linux
[   59.566157] Stack:
[   59.566162]  ffffc9000235c200 0000000000000002 ffff88003af14038 0000000000000000
[   59.566168]  ffff88003c4bfdc8 0000000000000000 ffffc9000235c200 0000000000000002
[   59.566173]  ffff88003af14038 0000000000000000 ffff88003c4bfe68 ffffffff81527a80
[   59.566175] Call Trace:
[   59.566181]  [<ffffffff81527a80>] rtswitch_ktask+0x70/0x220
[   59.566189]  [<ffffffff8111d733>] ? xnshadow_map_kernel+0x1d3/0x1e0
[   59.566194]  [<ffffffff8111b590>] ? xnshadow_call_mayday+0x90/0x90
[   59.566200]  [<ffffffff811251e0>] ? xnthread_cancel+0x3f0/0x3f0
[   59.566204]  [<ffffffff8112523d>] kthread_trampoline+0x5d/0xb0
[   59.566211]  [<ffffffff8106a48e>] kthread+0xde/0xf0
[   59.566217]  [<ffffffff8106a3b0>] ? flush_kthread_worker+0x100/0x100
[   59.566224]  [<ffffffff8164e1ad>] ret_from_fork+0x7d/0xb0
[   59.566229]  [<ffffffff8106a3b0>] ? flush_kthread_worker+0x100/0x100
[   59.566295] Code: b4 c0 ff 85 c0 0f 85 22 fe ff ff 0f ae f0 c7 05 3f 55 42 00 ff ff ff ff e9 10 fe ff ff 89 d2 48 8d 04 d2 48 c1 e0 08 4c 8d 2c 01 <41> 8b 45 04 83 e0 04 75 2b 4c 89 ab 18 01 00 00 8b bb 20 01 00 
[   59.566297] RIP  [<ffffffff815276f4>] rtswitch_to_rt+0x384/0x540
[   59.566298]  RSP <ffff88003c4bfd98>
[   59.566298] CR2: ffffd2000235a704
[   59.566301] ---[ end trace 251dca1b60fac852 ]---

Can debug this tomorrow - it nicely triggers in my vm.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] issues with debugging enabled

[Gilles Chanteperdrix]

On 11/14/2013 08:36 PM, Jan Kiszka wrote:
> On 2013-11-13 14:53, Jan Kiszka wrote:
>> On 2013-11-13 14:41, Gilles Chanteperdrix wrote:
>>> On 11/13/2013 02:25 PM, Jan Kiszka wrote:
>>>> On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> for a change, I ran the xeno-regression-test with a lot of debugging and
>>>>> options known to having caused problems in the past and found two
>>>>> issues:
>>>>>
>>>>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>>>>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>>>>> series of warnings at the end of the mail.
>>>>
>>>> We are currently facing crashes inside __switch_to, FPU related
>>>> (non-existing save area).
>>>
>>> Probably the issue solved by this patch:
>>> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=9b3320eef67e3f118f96065c8c2584b715710d94
>>
>> Thought I had everything, but I lost track, and it's missing here. Will
>> merge and give it a try.
> 
> OK, this fixes the crash in __switch_to, triggerable by terminating
> xeno-test while running clocktest. But we still face a crash when
> terminating switchtest early. Are you aware of this?

No, I was talking about Xenomai 2.6.3, not forge.


-- 
                                                                Gilles.

Re: [Xenomai] issues with debugging enabled

[Jan Kiszka]

On 2013-11-14 21:23, Gilles Chanteperdrix wrote:
> On 11/14/2013 08:36 PM, Jan Kiszka wrote:
>> On 2013-11-13 14:53, Jan Kiszka wrote:
>>> On 2013-11-13 14:41, Gilles Chanteperdrix wrote:
>>>> On 11/13/2013 02:25 PM, Jan Kiszka wrote:
>>>>> On 2013-11-12 21:34, Gilles Chanteperdrix wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> for a change, I ran the xeno-regression-test with a lot of debugging and
>>>>>> options known to having caused problems in the past and found two
>>>>>> issues:
>>>>>>
>>>>>> on x86 SMP, with full dynticks and debugging enabled (preemptible kernel
>>>>>> debugging, mutex, spinlocks, and sleep inside spinlocks), I get the
>>>>>> series of warnings at the end of the mail.
>>>>>
>>>>> We are currently facing crashes inside __switch_to, FPU related
>>>>> (non-existing save area).
>>>>
>>>> Probably the issue solved by this patch:
>>>> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=9b3320eef67e3f118f96065c8c2584b715710d94
>>>
>>> Thought I had everything, but I lost track, and it's missing here. Will
>>> merge and give it a try.
>>
>> OK, this fixes the crash in __switch_to, triggerable by terminating
>> xeno-test while running clocktest. But we still face a crash when
>> terminating switchtest early. Are you aware of this?
> 
> No, I was talking about Xenomai 2.6.3, not forge.

Fix for forge with this commit:

http://git.xenomai.org/?p=xenomai-jki.git;a=commitdiff;h=3e6d8ff9a99262e78655329dc043aacc607eb158

I suppose that error scenario does not occur with 2.6 due to the
different kernel thread model. Still, backporting may be worth considering.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux