Re: [Adeos-main] [RTnet-users] e1000 & MSI

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 12343 bytes --]

Bernhard Pfund wrote:
> Jan Kiszka wrote:
>> Please post the oops. Also include the Adeos patch version you are using
>> for this.
>>
>> Jan
>>
> 
> Hi Jan
> 
> Eventually I found the call that is responsible for the mess. It's
> basically not in the rt_e1000 driver when loaded, but the ioctl sent by
> rtifconfig (ifonfig of RTnet) when trying to activate the NIC. I enabled
> i-pipe debugging in the kernel and got the following. I also posted the
> trace to the RTAI list, but maybe you've seen something similar before?

That's good that you forwarded it as this is an Adeos issue, not an RTAI 
thing. Added the related list.

> 
> Bernhard
> 
> Adeos is RTAI's hal-linux-2.6.26-x86-2.0-09.patch

OK.

> 
> I-pipe: Domain RTAI registered.
> RTAI[hal]: <magma> mounted over IPIPE-NOTHREADS 2.0-09.
> RTAI[hal]: compiled with gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7).
> RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs VECTORED),
> ISOL_CPUS_MASK: e).
> PIPELINE layers:
> f8936600 9ac15d93 RTAI 200
> c0737540 0 Linux 100
> RTAI[malloc]: global heap size = 4194304 bytes, <BSD>.
> RTAI[sched]: IMMEDIATE, MP, USER/KERNEL SPACE: <with RTAI OWN KTASKs>,
> kstacks pool size = 1048576 bytes.
> RTAI[sched]: hard timer type/freq = APIC/16666663(Hz); default timing:
> periodic; linear timed lists.
> RTAI[sched]: Linux timer freq = 1000 (Hz), CPU freq = 2400046000 hz.
> RTAI[sched]: timer setup = 999 ns, resched latency = 0 ns.
> RTAI[usi]: enabled.
> RTAI[rtai_msgq]: loaded.
> RTAI[mq]: loaded.
> RTDM started.
> 
> *** RTnet 0.9.11 - built on Aug 13 2008 22:31:19 ***
> 
> RTnet: initialising real-time networking
> Intel(R) PRO/1000 Network Driver - version 7.1.9
> Copyright (c) 1999-2006 Intel Corporation.
> ACPI: PCI Interrupt Link [APC8] enabled at IRQ 16
> ACPI: PCI Interrupt 0000:03:00.0[A] -> Link [APC8] -> GSI 16 (level,
> low) -> IRQ 16
> PCI: Setting latency timer of device 0000:03:00.0 to 64
> e1000: 0000:03:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1)
> 00:1b:21:1e:56:64
> RTnet: registered rteth0
> e1000: rteth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> I-pipe: Detected illicit call from domain 'RTAI'
>         into a service reserved for domain 'Linux' and below.
> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #3
>  [<c0156866>] ipipe_check_context+0xd6/0xf0
>  [<c03e202e>] <6>e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps
> Full Duplex
> _spin_lock_irqsave+0x1e/0x80
>  [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>  [<c024c29e>] __pci_bus_find_cap_start+0x1e/0x50
>  [<c024c7f4>] pci_find_capability+0x24/0x50
>  [<c0254132>] msi_set_enable+0x22/0x80
>  [<c01176f3>] ? mcount+0x1f/0x23
>  [<c025444f>] msi_set_mask_bits+0xcf/0xe0
>  [<c01176f3>] ? mcount+0x1f/0x23
>  [<c02546f7>] unmask_msi_irq+0x17/0x30
>  [<c01542da>] default_enable+0x1a/0x30
>  [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>  [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>  [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>  [<c01021c5>] default_idle+0x45/0x60
>  [<c0102180>] default_idle+0x0/0x60
>  [<c0103cc7>] common_interrupt+0x2f/0x54
>  [<c0102180>] default_idle+0x0/0x60
>  [<c01500d8>] cgroup_file_write+0x118/0x140
>  [<c01021c5>] default_idle+0x45/0x60
>  [<c0101b76>] cpu_idle+0x86/0x140
>  [<c03dd7bd>] start_secondary+0x16d/0x210
>  [<c03d3a48>] initialize_secondary+0x8/0x20
>  =======================
> I-pipe tracer log (100 points):
>  |  +*func                    0 ipipe_trace_panic_freeze+0x9
> (ipipe_check_context+0x94)
>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>  |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>  |  +*func                   -1 __first_cpu+0x8 (ipipe_check_context+0x66)
>  |  +*func                   -1 ipipe_check_context+0x14
> (_spin_lock_irqsave+0x1e)
>  |  +*func                   -1 _spin_lock_irqsave+0x12
> (pci_bus_read_config_word+0x36)
>  |  +*func                   -1 pci_bus_read_config_word+0x14
> (__pci_bus_find_cap_start+0x1e)
>  |  +*func                   -1 __pci_bus_find_cap_start+0xc
> (pci_find_capability+0x24)
>  |  +*func                   -1 pci_find_capability+0x11
> (msi_set_enable+0x22)
>  |  +*func                   -1 msi_set_enable+0x14 (msi_set_mask_bits+0xcf)
>  |  +*func                   -1 msi_set_mask_bits+0xe (unmask_msi_irq+0x17)
>  |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)

I think to remember a similar issue biting the preempt-rt patch, and 
that resulted in some activity to cache that capability information. /me 
needs to dig into the archives, will let you know if I find something 
(tomorrow, likely).

>  |  +*func                   -2 default_enable+0x9 (rt_enable_irq+0xe
> [rtai_hal])
>  |  +*func                   -2 alloc_rtskb+0x14 [rtnet]
> (e1000_alloc_rx_buffers+0x147 [rt_e1000])
>  |  +*func                   -2 e1000_alloc_rx_buffers+0xe [rt_e1000]
> (e1000_intr+0x37d [rt_e1000])
>  |  +*func                   -5 e1000_intr+0x11 [rt_e1000]
> (xnintr_irq_handler+0x9e [rtai_rtdm])
>  |  +*func                   -5 xnintr_irq_handler+0xe [rtai_rtdm]
> (rtai_hirq_dispatcher+0xfb [rtai_hal])
>  |  +*func                   -5 ack_ioapic_irq+0x8
> (__ipipe_ack_edge_irq+0xe)
>  |  +*func                   -6 __ipipe_ack_edge_irq+0x8
> (__ipipe_ack_irq+0x19)
>  |  +*func                   -6 __ipipe_ack_irq+0x8
> (rtai_hirq_dispatcher+0x66 [rtai_hal])
>  |   +begin   0xffffff23     -6 common_interrupt+0x29 (default_idle+0x45)
>      +end     0x8000000e   -600 default_idle+0x43 (cpu_idle+0x86)
>      +func                 -600 default_idle+0x8 (cpu_idle+0x86)
>  |   +end     0x80000001   -600 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
>  |   #begin   0x80000001   -600 ipipe_suspend_domain+0xee (cpu_idle+0x84)
>      #func                 -600 ipipe_suspend_domain+0xe (cpu_idle+0x84)
>      +func                 -601 ipipe_check_context+0x14 (cpu_idle+0x5b)
>      +func                 -601 ipipe_check_context+0x14
> (_spin_unlock_irqrestore+0x23)
>  |   +end     0x80000000   -601 __ipipe_unstall_root+0x4a
> (__ipipe_restore_root+0x27)
>  |   #begin   0x80000000   -601 __ipipe_unstall_root+0x5b
> (__ipipe_restore_root+0x27)
>      #func                 -601 __ipipe_unstall_root+0x8
> (__ipipe_restore_root+0x27)
>      #func                 -601 __ipipe_restore_root+0x8
> (_spin_unlock_irqrestore+0x3d)
>      #func                 -601 _spin_unlock_irqrestore+0x8
> (rcu_check_callbacks+0x5c)
>      #func                 -601 __rcu_advance_callbacks+0x8
> (rcu_check_callbacks+0x35)
>      #func                 -601 ipipe_check_context+0x14
> (_spin_lock_irqsave+0x49)
>      +func                 -602 ipipe_check_context+0x14
> (_spin_lock_irqsave+0x1e)
>      +func                 -602 _spin_lock_irqsave+0x12
> (rcu_check_callbacks+0x2c)
>      +func                 -602 rcu_check_mb+0x8 (rcu_check_callbacks+0x1b)
>      +func                 -602 rcu_check_callbacks+0xa (cpu_idle+0xb2)
>      +func                 -602 rcu_pending+0x8 (cpu_idle+0xa5)
>  |   +end     0x8000000d   -602 __ipipe_unstall_iret_root+0x36
> (restore_nocheck_notrace+0x0)
>  |   #func                 -602 __ipipe_unstall_iret_root+0x9
> (restore_nocheck_notrace+0x0)
>  |   #end     0xffffff15   -602 ipipe_ipiX+0x3e (default_idle+0x45)
>  |   +end     0x8000000d   -603 __ipipe_unstall_iret_root+0x36
> (restore_nocheck_notrace+0x0)
>  |   #func                 -603 __ipipe_unstall_iret_root+0x9
> (restore_nocheck_notrace+0x0)
>      #func                 -605 __ipipe_do_critical_sync+0x9
> (__ipipe_sync_stage+0x27b)
>  |   #end     0x80000000   -606 __ipipe_sync_stage+0x21f
> (rtai_hirq_dispatcher+0x3a1 [rtai_hal])
>  |   +func                 -606 __ipipe_sync_stage+0xe
> (rtai_hirq_dispatcher+0x3a1 [rtai_hal])
>  |   #func                 -606 __ipipe_ack_apic+0x8
> (rtai_hirq_dispatcher+0x257 [rtai_hal])
>  |   +begin   0xffffff15   -606 ipipe_ipiX+0x2e (default_idle+0x45)
>      +end     0x8000000e  -113749 default_idle+0x43 (cpu_idle+0x86)
>      +func                -113749 default_idle+0x8 (cpu_idle+0x86)
>  |   +end     0x80000001  -113749 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
>  |   #begin   0x80000001  -113749 ipipe_suspend_domain+0xee (cpu_idle+0x84)
>      #func                -113749 ipipe_suspend_domain+0xe (cpu_idle+0x84)
>      +func                -113749 ipipe_check_context+0x14 (cpu_idle+0x5b)
>      +func                -113749 rcu_pending+0x8 (cpu_idle+0xa5)
>  |   +end     0x80000000  -113750 __ipipe_unstall_root+0x4a
> (__ipipe_restore_root+0x27)
>  |   #begin   0x80000000  -113750 __ipipe_unstall_root+0x5b
> (__ipipe_restore_root+0x27)
>      #func                -113750 __ipipe_unstall_root+0x8
> (__ipipe_restore_root+0x27)
>      #func                -113750 __ipipe_restore_root+0x8
> (tick_nohz_stop_sched_tick+0x23b)
>      #func                -113750 ipipe_check_context+0x14
> (hrtimer_start+0xe1)
>      #func                -113750 ipipe_check_context+0x14
> (_spin_unlock_irqrestore+0x23)
>      #func                -113750 __ipipe_restore_root+0x8
> (_spin_unlock_irqrestore+0x19)
>      #func                -113750 _spin_unlock_irqrestore+0x8
> (hrtimer_start+0xd2)
>      #func                -113750 ipipe_check_context+0x14
> (hrtimer_start+0xba)
>      #func                -113750 rb_insert_color+0xe (enqueue_hrtimer+0x7d)
>      #func                -113751 lapic_next_event+0x8
> (clockevents_program_event+0x9e)
>      #func                -113751 clockevents_program_event+0x14
> (tick_program_event+0x44)
>      #func                -113751 set_normalized_timespec+0x8
> (ktime_get_ts+0x43)
>      #func                -113751 native_read_tsc+0x8 (read_tsc+0xe)
>      #func                -113751 read_tsc+0x9 (getnstimeofday+0x48)
>      #func                -113751 getnstimeofday+0xe (ktime_get_ts+0x22)
>      #func                -113751 ktime_get_ts+0xa (ktime_get+0x1e)
>      #func                -113751 ktime_get+0x14 (tick_program_event+0x29)
>      #func                -113751 tick_program_event+0xe
> (hrtimer_reprogram+0x86)
>      #func                -113751 hrtimer_reprogram+0x14
> (enqueue_hrtimer+0xaa)
>      #func                -113751 enqueue_hrtimer+0xe (hrtimer_start+0xad)
>      #func                -113752 rb_erase+0xe (__remove_hrtimer+0x57)
>      #func                -113752 hrtimer_force_reprogram+0xa
> (__remove_hrtimer+0x83)
>      #func                -113752 rb_next+0x9 (__remove_hrtimer+0x5e)
>      #func                -113752 __remove_hrtimer+0x16
> (hrtimer_start+0x123)
>      #func                -113752 ipipe_check_context+0x14
> (_spin_lock_irqsave+0x49)
>      #func                -113752 ipipe_check_context+0x14
> (_spin_lock_irqsave+0x1e)
>      #func                -113752 _spin_lock_irqsave+0x12
> (lock_hrtimer_base+0x28)
>      #func                -113752 lock_hrtimer_base+0x16
> (hrtimer_start+0x1e)
>      #func                -113752 hrtimer_start+0xe
> (tick_nohz_stop_sched_tick+0x255)
>      #func                -113753 hweight32+0x8
> (select_nohz_load_balancer+0x58)
>      #func                -113753 hweight32+0x8
> (select_nohz_load_balancer+0x4a)
>      #func                -113753 select_nohz_load_balancer+0xa
> (tick_nohz_stop_sched_tick+0x287)
>      #func                -113753 rcu_needs_cpu+0x8
> (tick_nohz_stop_sched_tick+0x13c)
>      #func                -113753 ipipe_check_context+0x14
> (_spin_unlock_irqrestore+0x23)
>      #func                -113753 __ipipe_restore_root+0x8
> (_spin_unlock_irqrestore+0x19)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 5757 bytes --]

Jan Kiszka wrote:
> Bernhard Pfund wrote:
>> Jan Kiszka wrote:
>>> Please post the oops. Also include the Adeos patch version you are using
>>> for this.
>>>
>>> Jan
>>>
>>
>> Hi Jan
>>
>> Eventually I found the call that is responsible for the mess. It's
>> basically not in the rt_e1000 driver when loaded, but the ioctl sent by
>> rtifconfig (ifonfig of RTnet) when trying to activate the NIC. I enabled
>> i-pipe debugging in the kernel and got the following. I also posted the
>> trace to the RTAI list, but maybe you've seen something similar before?
> 
> That's good that you forwarded it as this is an Adeos issue, not an RTAI
> thing. Added the related list.
> 
>>
>> Bernhard
>>
>> Adeos is RTAI's hal-linux-2.6.26-x86-2.0-09.patch
> 
> OK.
> 
>>
>> I-pipe: Domain RTAI registered.
>> RTAI[hal]: <magma> mounted over IPIPE-NOTHREADS 2.0-09.
>> RTAI[hal]: compiled with gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7).
>> RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs VECTORED),
>> ISOL_CPUS_MASK: e).
>> PIPELINE layers:
>> f8936600 9ac15d93 RTAI 200
>> c0737540 0 Linux 100
>> RTAI[malloc]: global heap size = 4194304 bytes, <BSD>.
>> RTAI[sched]: IMMEDIATE, MP, USER/KERNEL SPACE: <with RTAI OWN KTASKs>,
>> kstacks pool size = 1048576 bytes.
>> RTAI[sched]: hard timer type/freq = APIC/16666663(Hz); default timing:
>> periodic; linear timed lists.
>> RTAI[sched]: Linux timer freq = 1000 (Hz), CPU freq = 2400046000 hz.
>> RTAI[sched]: timer setup = 999 ns, resched latency = 0 ns.
>> RTAI[usi]: enabled.
>> RTAI[rtai_msgq]: loaded.
>> RTAI[mq]: loaded.
>> RTDM started.
>>
>> *** RTnet 0.9.11 - built on Aug 13 2008 22:31:19 ***
>>
>> RTnet: initialising real-time networking
>> Intel(R) PRO/1000 Network Driver - version 7.1.9
>> Copyright (c) 1999-2006 Intel Corporation.
>> ACPI: PCI Interrupt Link [APC8] enabled at IRQ 16
>> ACPI: PCI Interrupt 0000:03:00.0[A] -> Link [APC8] -> GSI 16 (level,
>> low) -> IRQ 16
>> PCI: Setting latency timer of device 0000:03:00.0 to 64
>> e1000: 0000:03:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1)
>> 00:1b:21:1e:56:64
>> RTnet: registered rteth0
>> e1000: rteth0: e1000_probe: Intel(R) PRO/1000 Network Connection
>> I-pipe: Detected illicit call from domain 'RTAI'
>>         into a service reserved for domain 'Linux' and below.
>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #3
>>  [<c0156866>] ipipe_check_context+0xd6/0xf0
>>  [<c03e202e>] <6>e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps
>> Full Duplex
>> _spin_lock_irqsave+0x1e/0x80
>>  [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>  [<c024c29e>] __pci_bus_find_cap_start+0x1e/0x50
>>  [<c024c7f4>] pci_find_capability+0x24/0x50
>>  [<c0254132>] msi_set_enable+0x22/0x80
>>  [<c01176f3>] ? mcount+0x1f/0x23
>>  [<c025444f>] msi_set_mask_bits+0xcf/0xe0
>>  [<c01176f3>] ? mcount+0x1f/0x23
>>  [<c02546f7>] unmask_msi_irq+0x17/0x30
>>  [<c01542da>] default_enable+0x1a/0x30
>>  [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>  [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>  [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>  [<c01021c5>] default_idle+0x45/0x60
>>  [<c0102180>] default_idle+0x0/0x60
>>  [<c0103cc7>] common_interrupt+0x2f/0x54
>>  [<c0102180>] default_idle+0x0/0x60
>>  [<c01500d8>] cgroup_file_write+0x118/0x140
>>  [<c01021c5>] default_idle+0x45/0x60
>>  [<c0101b76>] cpu_idle+0x86/0x140
>>  [<c03dd7bd>] start_secondary+0x16d/0x210
>>  [<c03d3a48>] initialize_secondary+0x8/0x20
>>  =======================
>> I-pipe tracer log (100 points):
>>  |  +*func                    0 ipipe_trace_panic_freeze+0x9
>> (ipipe_check_context+0x94)
>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>  |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>>  |  +*func                   -1 __first_cpu+0x8
>> (ipipe_check_context+0x66)
>>  |  +*func                   -1 ipipe_check_context+0x14
>> (_spin_lock_irqsave+0x1e)
>>  |  +*func                   -1 _spin_lock_irqsave+0x12
>> (pci_bus_read_config_word+0x36)
>>  |  +*func                   -1 pci_bus_read_config_word+0x14
>> (__pci_bus_find_cap_start+0x1e)
>>  |  +*func                   -1 __pci_bus_find_cap_start+0xc
>> (pci_find_capability+0x24)
>>  |  +*func                   -1 pci_find_capability+0x11
>> (msi_set_enable+0x22)
>>  |  +*func                   -1 msi_set_enable+0x14
>> (msi_set_mask_bits+0xcf)
>>  |  +*func                   -1 msi_set_mask_bits+0xe
>> (unmask_msi_irq+0x17)
>>  |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
> 
> I think to remember a similar issue biting the preempt-rt patch, and
> that resulted in some activity to cache that capability information. /me
> needs to dig into the archives, will let you know if I find something
> (tomorrow, likely).

Found it. Could you give this patch a try and report the result?

http://permalink.gmane.org/gmane.linux.kernel/682362

If it's ok, I guess we should include it in ipipe until someone (From
-rt) manages to get it accepted upstream (I didn't recall much activity
in this direction yet, though).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 6272 bytes --]

Jan Kiszka wrote:
> Jan Kiszka wrote:
>> Bernhard Pfund wrote:
>>> Jan Kiszka wrote:
>>>> Please post the oops. Also include the Adeos patch version you are using
>>>> for this.
>>>>
>>>> Jan
>>>>
>>> Hi Jan
>>>
>>> Eventually I found the call that is responsible for the mess. It's
>>> basically not in the rt_e1000 driver when loaded, but the ioctl sent by
>>> rtifconfig (ifonfig of RTnet) when trying to activate the NIC. I enabled
>>> i-pipe debugging in the kernel and got the following. I also posted the
>>> trace to the RTAI list, but maybe you've seen something similar before?
>> That's good that you forwarded it as this is an Adeos issue, not an RTAI
>> thing. Added the related list.
>>
>>> Bernhard
>>>
>>> Adeos is RTAI's hal-linux-2.6.26-x86-2.0-09.patch
>> OK.
>>
>>> I-pipe: Domain RTAI registered.
>>> RTAI[hal]: <magma> mounted over IPIPE-NOTHREADS 2.0-09.
>>> RTAI[hal]: compiled with gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7).
>>> RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs VECTORED),
>>> ISOL_CPUS_MASK: e).
>>> PIPELINE layers:
>>> f8936600 9ac15d93 RTAI 200
>>> c0737540 0 Linux 100
>>> RTAI[malloc]: global heap size = 4194304 bytes, <BSD>.
>>> RTAI[sched]: IMMEDIATE, MP, USER/KERNEL SPACE: <with RTAI OWN KTASKs>,
>>> kstacks pool size = 1048576 bytes.
>>> RTAI[sched]: hard timer type/freq = APIC/16666663(Hz); default timing:
>>> periodic; linear timed lists.
>>> RTAI[sched]: Linux timer freq = 1000 (Hz), CPU freq = 2400046000 hz.
>>> RTAI[sched]: timer setup = 999 ns, resched latency = 0 ns.
>>> RTAI[usi]: enabled.
>>> RTAI[rtai_msgq]: loaded.
>>> RTAI[mq]: loaded.
>>> RTDM started.
>>>
>>> *** RTnet 0.9.11 - built on Aug 13 2008 22:31:19 ***
>>>
>>> RTnet: initialising real-time networking
>>> Intel(R) PRO/1000 Network Driver - version 7.1.9
>>> Copyright (c) 1999-2006 Intel Corporation.
>>> ACPI: PCI Interrupt Link [APC8] enabled at IRQ 16
>>> ACPI: PCI Interrupt 0000:03:00.0[A] -> Link [APC8] -> GSI 16 (level,
>>> low) -> IRQ 16
>>> PCI: Setting latency timer of device 0000:03:00.0 to 64
>>> e1000: 0000:03:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1)
>>> 00:1b:21:1e:56:64
>>> RTnet: registered rteth0
>>> e1000: rteth0: e1000_probe: Intel(R) PRO/1000 Network Connection
>>> I-pipe: Detected illicit call from domain 'RTAI'
>>>         into a service reserved for domain 'Linux' and below.
>>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #3
>>>  [<c0156866>] ipipe_check_context+0xd6/0xf0
>>>  [<c03e202e>] <6>e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps
>>> Full Duplex
>>> _spin_lock_irqsave+0x1e/0x80
>>>  [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>>  [<c024c29e>] __pci_bus_find_cap_start+0x1e/0x50
>>>  [<c024c7f4>] pci_find_capability+0x24/0x50
>>>  [<c0254132>] msi_set_enable+0x22/0x80
>>>  [<c01176f3>] ? mcount+0x1f/0x23
>>>  [<c025444f>] msi_set_mask_bits+0xcf/0xe0
>>>  [<c01176f3>] ? mcount+0x1f/0x23
>>>  [<c02546f7>] unmask_msi_irq+0x17/0x30
>>>  [<c01542da>] default_enable+0x1a/0x30
>>>  [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>>  [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>>  [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>>  [<c01021c5>] default_idle+0x45/0x60
>>>  [<c0102180>] default_idle+0x0/0x60
>>>  [<c0103cc7>] common_interrupt+0x2f/0x54
>>>  [<c0102180>] default_idle+0x0/0x60
>>>  [<c01500d8>] cgroup_file_write+0x118/0x140
>>>  [<c01021c5>] default_idle+0x45/0x60
>>>  [<c0101b76>] cpu_idle+0x86/0x140
>>>  [<c03dd7bd>] start_secondary+0x16d/0x210
>>>  [<c03d3a48>] initialize_secondary+0x8/0x20
>>>  =======================
>>> I-pipe tracer log (100 points):
>>>  |  +*func                    0 ipipe_trace_panic_freeze+0x9
>>> (ipipe_check_context+0x94)
>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>  |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>>>  |  +*func                   -1 __first_cpu+0x8
>>> (ipipe_check_context+0x66)
>>>  |  +*func                   -1 ipipe_check_context+0x14
>>> (_spin_lock_irqsave+0x1e)
>>>  |  +*func                   -1 _spin_lock_irqsave+0x12
>>> (pci_bus_read_config_word+0x36)
>>>  |  +*func                   -1 pci_bus_read_config_word+0x14
>>> (__pci_bus_find_cap_start+0x1e)
>>>  |  +*func                   -1 __pci_bus_find_cap_start+0xc
>>> (pci_find_capability+0x24)
>>>  |  +*func                   -1 pci_find_capability+0x11
>>> (msi_set_enable+0x22)
>>>  |  +*func                   -1 msi_set_enable+0x14
>>> (msi_set_mask_bits+0xcf)
>>>  |  +*func                   -1 msi_set_mask_bits+0xe
>>> (unmask_msi_irq+0x17)
>>>  |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
>> I think to remember a similar issue biting the preempt-rt patch, and
>> that resulted in some activity to cache that capability information. /me
>> needs to dig into the archives, will let you know if I find something
>> (tomorrow, likely).
> 
> Found it. Could you give this patch a try and report the result?
> 
> http://permalink.gmane.org/gmane.linux.kernel/682362
> 
> If it's ok, I guess we should include it in ipipe until someone (From
> -rt) manages to get it accepted upstream (I didn't recall much activity
> in this direction yet, though).

Not true, the patch is in 2.6.27-rcX.

But there is also
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ce6fce4295ba727b36fdc73040e444bd1aae64cd
which makes me wonder, for 2.6.27, if that may generate cases where
masking MSI interrupts will not work as expected for ipipe (Linux should
catch masked IRQs internally). However, future problems...

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Jan Kiszka wrote:
> Jan Kiszka wrote:
>> Jan Kiszka wrote:
>>> Bernhard Pfund wrote:
>>>> Jan Kiszka wrote:
>>>>> Please post the oops. Also include the Adeos patch version you are using
>>>>> for this.
>>>>>
>>>>> Jan
>>>>>
>>>> Hi Jan
>>>>
>>>> Eventually I found the call that is responsible for the mess. It's
>>>> basically not in the rt_e1000 driver when loaded, but the ioctl sent by
>>>> rtifconfig (ifonfig of RTnet) when trying to activate the NIC. I enabled
>>>> i-pipe debugging in the kernel and got the following. I also posted the
>>>> trace to the RTAI list, but maybe you've seen something similar before?
>>> That's good that you forwarded it as this is an Adeos issue, not an RTAI
>>> thing. Added the related list.
>>>
>>>> Bernhard
>>>>
>>>> Adeos is RTAI's hal-linux-2.6.26-x86-2.0-09.patch
>>> OK.
>>>
>>>> I-pipe: Domain RTAI registered.
>>>> RTAI[hal]: <magma> mounted over IPIPE-NOTHREADS 2.0-09.
>>>> RTAI[hal]: compiled with gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7).
>>>> RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs VECTORED),
>>>> ISOL_CPUS_MASK: e).
>>>> PIPELINE layers:
>>>> f8936600 9ac15d93 RTAI 200
>>>> c0737540 0 Linux 100
>>>> RTAI[malloc]: global heap size = 4194304 bytes, <BSD>.
>>>> RTAI[sched]: IMMEDIATE, MP, USER/KERNEL SPACE: <with RTAI OWN KTASKs>,
>>>> kstacks pool size = 1048576 bytes.
>>>> RTAI[sched]: hard timer type/freq = APIC/16666663(Hz); default timing:
>>>> periodic; linear timed lists.
>>>> RTAI[sched]: Linux timer freq = 1000 (Hz), CPU freq = 2400046000 hz.
>>>> RTAI[sched]: timer setup = 999 ns, resched latency = 0 ns.
>>>> RTAI[usi]: enabled.
>>>> RTAI[rtai_msgq]: loaded.
>>>> RTAI[mq]: loaded.
>>>> RTDM started.
>>>>
>>>> *** RTnet 0.9.11 - built on Aug 13 2008 22:31:19 ***
>>>>
>>>> RTnet: initialising real-time networking
>>>> Intel(R) PRO/1000 Network Driver - version 7.1.9
>>>> Copyright (c) 1999-2006 Intel Corporation.
>>>> ACPI: PCI Interrupt Link [APC8] enabled at IRQ 16
>>>> ACPI: PCI Interrupt 0000:03:00.0[A] -> Link [APC8] -> GSI 16 (level,
>>>> low) -> IRQ 16
>>>> PCI: Setting latency timer of device 0000:03:00.0 to 64
>>>> e1000: 0000:03:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1)
>>>> 00:1b:21:1e:56:64
>>>> RTnet: registered rteth0
>>>> e1000: rteth0: e1000_probe: Intel(R) PRO/1000 Network Connection
>>>> I-pipe: Detected illicit call from domain 'RTAI'
>>>>         into a service reserved for domain 'Linux' and below.
>>>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #3
>>>>  [<c0156866>] ipipe_check_context+0xd6/0xf0
>>>>  [<c03e202e>] <6>e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps
>>>> Full Duplex
>>>> _spin_lock_irqsave+0x1e/0x80
>>>>  [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>>>  [<c024c29e>] __pci_bus_find_cap_start+0x1e/0x50
>>>>  [<c024c7f4>] pci_find_capability+0x24/0x50
>>>>  [<c0254132>] msi_set_enable+0x22/0x80
>>>>  [<c01176f3>] ? mcount+0x1f/0x23
>>>>  [<c025444f>] msi_set_mask_bits+0xcf/0xe0
>>>>  [<c01176f3>] ? mcount+0x1f/0x23
>>>>  [<c02546f7>] unmask_msi_irq+0x17/0x30
>>>>  [<c01542da>] default_enable+0x1a/0x30
>>>>  [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>>>  [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>>>  [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>>>  [<c01021c5>] default_idle+0x45/0x60
>>>>  [<c0102180>] default_idle+0x0/0x60
>>>>  [<c0103cc7>] common_interrupt+0x2f/0x54
>>>>  [<c0102180>] default_idle+0x0/0x60
>>>>  [<c01500d8>] cgroup_file_write+0x118/0x140
>>>>  [<c01021c5>] default_idle+0x45/0x60
>>>>  [<c0101b76>] cpu_idle+0x86/0x140
>>>>  [<c03dd7bd>] start_secondary+0x16d/0x210
>>>>  [<c03d3a48>] initialize_secondary+0x8/0x20
>>>>  =======================
>>>> I-pipe tracer log (100 points):
>>>>  |  +*func                    0 ipipe_trace_panic_freeze+0x9
>>>> (ipipe_check_context+0x94)
>>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>>  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>>>  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>>>  |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>>>>  |  +*func                   -1 __first_cpu+0x8
>>>> (ipipe_check_context+0x66)
>>>>  |  +*func                   -1 ipipe_check_context+0x14
>>>> (_spin_lock_irqsave+0x1e)
>>>>  |  +*func                   -1 _spin_lock_irqsave+0x12
>>>> (pci_bus_read_config_word+0x36)
>>>>  |  +*func                   -1 pci_bus_read_config_word+0x14
>>>> (__pci_bus_find_cap_start+0x1e)
>>>>  |  +*func                   -1 __pci_bus_find_cap_start+0xc
>>>> (pci_find_capability+0x24)
>>>>  |  +*func                   -1 pci_find_capability+0x11
>>>> (msi_set_enable+0x22)
>>>>  |  +*func                   -1 msi_set_enable+0x14
>>>> (msi_set_mask_bits+0xcf)
>>>>  |  +*func                   -1 msi_set_mask_bits+0xe
>>>> (unmask_msi_irq+0x17)
>>>>  |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
>>> I think to remember a similar issue biting the preempt-rt patch, and
>>> that resulted in some activity to cache that capability information. /me
>>> needs to dig into the archives, will let you know if I find something
>>> (tomorrow, likely).
>> Found it. Could you give this patch a try and report the result?
>>
>> http://permalink.gmane.org/gmane.linux.kernel/682362
>>
>> If it's ok, I guess we should include it in ipipe until someone (From
>> -rt) manages to get it accepted upstream (I didn't recall much activity
>> in this direction yet, though).
> 
> Not true, the patch is in 2.6.27-rcX.
> 
> But there is also
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ce6fce4295ba727b36fdc73040e444bd1aae64cd
> which makes me wonder, for 2.6.27, if that may generate cases where
> masking MSI interrupts will not work as expected for ipipe (Linux should
> catch masked IRQs internally). However, future problems...
>

At the very least, this will affect the code relying on the I-pipe. The I-pipe
machinery itself should be fine as long as MSI interrupts are edge, which seems
to be the point in not relying on the MSI_ENABLE bit for devices that don't
support any INT disable bit. We have hw interrupts off from the MSI interrupt
receipt to the point where it has been marked as pending and possibly dispatched
to the domains, then the stall bit should handle any recursion properly.

However, having IRQ chip controller handlers not enforcing the requested
operation (i.e. masking) in the MSI case may clearly be a problem with respect
to explicit *_disable_irq() calls issued from Adeos clients, though. We should
probably emulate the IRQ_DISABLED flag with the I-pipe's per-IRQ lock bit for them.

> Jan
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[bernhard]

>>
>> Found it. Could you give this patch a try and report the result?
>>
>> http://permalink.gmane.org/gmane.linux.kernel/682362

Applied and tested, no luck...

I-pipe: Detected illicit call from domain 'RTAI'
         into a service reserved for domain 'Linux' and below.
Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #1
  [<c0156866>] ipipe_check_context+0xd6/0xf0
e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
  [<c03e206e>] _spin_lock_irqsave+0x1e/0x80
  [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
  [<c0254156>] __msi_set_enable+0x46/0x80
  [<c01176f3>] ? mcount+0x1f/0x23
  [<c0254498>] msi_set_mask_bits+0xd8/0xe0
  [<c01176f3>] ? mcount+0x1f/0x23
  [<c0254737>] unmask_msi_irq+0x17/0x30
  [<c01542da>] default_enable+0x1a/0x30
  [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
  [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
  [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
  [<c01021c5>] default_idle+0x45/0x60
  [<c0102180>] default_idle+0x0/0x60
  [<c0103cc7>] common_interrupt+0x2f/0x54
  [<c0102180>] default_idle+0x0/0x60
  [<c01500d8>] cgroup_file_write+0x118/0x140
  [<c01021c5>] default_idle+0x45/0x60
  [<c0101b76>] cpu_idle+0x86/0x140
  [<c03dd7fd>] start_secondary+0x16d/0x210
  [<c03d3a88>] initialize_secondary+0x8/0x20
  =======================
I-pipe tracer log (100 points):
  |  +*func                    0 ipipe_trace_panic_freeze+0x9  
(ipipe_check_context+0x94)
  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
  |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
  |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
  |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
  |  +*func                   -1 __first_cpu+0x8 (ipipe_check_context+0x66)
  |  +*func                   -1 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x1e)
  |  +*func                   -1 _spin_lock_irqsave+0x12  
(pci_bus_read_config_word+0x36)
  |  +*func                   -1 pci_bus_read_config_word+0x14  
(__msi_set_enable+0x46)
  |  +*func                   -1 __msi_set_enable+0x14 (msi_set_mask_bits+0xd8)
  |  +*func                   -1 msi_set_mask_bits+0xe (unmask_msi_irq+0x17)
  |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
  |  +*func                   -1 default_enable+0x9 (rt_enable_irq+0xe  
[rtai_hal])
  |  +*func                   -2 alloc_rtskb+0x14 [rtnet]  
(e1000_alloc_rx_buffers+0x147 [rt_e1000])
  |  +*func                   -2 e1000_alloc_rx_buffers+0xe [rt_e1000]  
(e1000_intr+0x37d [rt_e1000])
  |  +*func                   -4 e1000_intr+0x11 [rt_e1000]  
(xnintr_irq_handler+0x9e [rtai_rtdm])
  |  +*func                   -5 xnintr_irq_handler+0xe [rtai_rtdm]  
(rtai_hirq_dispatcher+0xfb [rtai_hal])
  |  +*func                   -5 ack_ioapic_irq+0x8 (__ipipe_ack_edge_irq+0xe)
  |  +*func                   -5 __ipipe_ack_edge_irq+0x8  
(__ipipe_ack_irq+0x19)
  |  +*func                   -5 __ipipe_ack_irq+0x8  
(rtai_hirq_dispatcher+0x66 [rtai_hal])
  |   +begin   0xffffff23     -5 common_interrupt+0x29 (default_idle+0x45)
      +end     0x8000000e   -580 default_idle+0x43 (cpu_idle+0x86)
      +func                 -580 default_idle+0x8 (cpu_idle+0x86)
  |   +end     0x80000001   -581 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
  |   #begin   0x80000001   -581 ipipe_suspend_domain+0xee (cpu_idle+0x84)
      #func                 -581 ipipe_suspend_domain+0xe (cpu_idle+0x84)
      +func                 -581 ipipe_check_context+0x14 (cpu_idle+0x5b)
      +func                 -581 ipipe_check_context+0x14  
(_spin_unlock_irqrestore+0x23)
  |   +end     0x80000000   -581 __ipipe_unstall_root+0x4a  
(__ipipe_restore_root+0x27)
  |   #begin   0x80000000   -581 __ipipe_unstall_root+0x5b  
(__ipipe_restore_root+0x27)
      #func                 -581 __ipipe_unstall_root+0x8  
(__ipipe_restore_root+0x27)
      #func                 -581 __ipipe_restore_root+0x8  
(_spin_unlock_irqrestore+0x3d)
      #func                 -581 _spin_unlock_irqrestore+0x8  
(rcu_check_callbacks+0x5c)
      #func                 -582 __rcu_advance_callbacks+0x8  
(rcu_check_callbacks+0x35)
      #func                 -582 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x49)
      +func                 -582 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x1e)
      +func                 -582 _spin_lock_irqsave+0x12  
(rcu_check_callbacks+0x2c)
      +func                 -582 rcu_check_mb+0x8 (rcu_check_callbacks+0x1b)
      +func                 -582 rcu_check_callbacks+0xa (cpu_idle+0xb2)
      +func                 -582 rcu_pending+0x8 (cpu_idle+0xa5)
  |   +end     0x8000000d   -583 __ipipe_unstall_iret_root+0x36  
(restore_nocheck_notrace+0x0)
  |   #func                 -583 __ipipe_unstall_iret_root+0x9  
(restore_nocheck_notrace+0x0)
  |   #end     0xffffff15   -583 ipipe_ipiX+0x3e (default_idle+0x45)
  |   +end     0x8000000d   -583 __ipipe_unstall_iret_root+0x36  
(restore_nocheck_notrace+0x0)
  |   #func                 -583 __ipipe_unstall_iret_root+0x9  
(restore_nocheck_notrace+0x0)
      #func                 -585 __ipipe_do_critical_sync+0x9  
(__ipipe_sync_stage+0x27b)
  |   #end     0x80000000   -586 __ipipe_sync_stage+0x21f  
(rtai_hirq_dispatcher+0x3a1 [rtai_hal])
  |   +func                 -586 __ipipe_sync_stage+0xe  
(rtai_hirq_dispatcher+0x3a1 [rtai_hal])
  |   #func                 -586 __ipipe_ack_apic+0x8  
(rtai_hirq_dispatcher+0x257 [rtai_hal])
  |   +begin   0xffffff15   -587 ipipe_ipiX+0x2e (default_idle+0x45)
      +end     0x8000000e  -318728 default_idle+0x43 (cpu_idle+0x86)
      +func                -318728 default_idle+0x8 (cpu_idle+0x86)
  |   +end     0x80000001  -318728 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
  |   #begin   0x80000001  -318728 ipipe_suspend_domain+0xee (cpu_idle+0x84)
      #func                -318728 ipipe_suspend_domain+0xe (cpu_idle+0x84)
      +func                -318729 ipipe_check_context+0x14 (cpu_idle+0x5b)
      +func                -318729 rcu_pending+0x8 (cpu_idle+0xa5)
  |   +end     0x80000000  -318729 __ipipe_unstall_root+0x4a  
(__ipipe_restore_root+0x27)
  |   #begin   0x80000000  -318729 __ipipe_unstall_root+0x5b  
(__ipipe_restore_root+0x27)
      #func                -318729 __ipipe_unstall_root+0x8  
(__ipipe_restore_root+0x27)
      #func                -318729 __ipipe_restore_root+0x8  
(tick_nohz_stop_sched_tick+0x23b)
      #func                -318729 ipipe_check_context+0x14  
(hrtimer_start+0xe1)
      #func                -318729 ipipe_check_context+0x14  
(_spin_unlock_irqrestore+0x23)
      #func                -318729 __ipipe_restore_root+0x8  
(_spin_unlock_irqrestore+0x19)
      #func                -318729 _spin_unlock_irqrestore+0x8  
(hrtimer_start+0xd2)
      #func                -318729 ipipe_check_context+0x14  
(hrtimer_start+0xba)
      #func                -318730 rb_insert_color+0xe (enqueue_hrtimer+0x7d)
      #func                -318730 lapic_next_event+0x8  
(clockevents_program_event+0x9e)
      #func                -318730 clockevents_program_event+0x14  
(tick_program_event+0x44)
      #func                -318730 set_normalized_timespec+0x8  
(ktime_get_ts+0x43)
      #func                -318730 native_read_tsc+0x8 (read_tsc+0xe)
      #func                -318730 read_tsc+0x9 (getnstimeofday+0x48)
      #func                -318730 getnstimeofday+0xe (ktime_get_ts+0x22)
      #func                -318730 ktime_get_ts+0xa (ktime_get+0x1e)
      #func                -318730 ktime_get+0x14 (tick_program_event+0x29)
      #func                -318730 tick_program_event+0xe  
(hrtimer_reprogram+0x86)
      #func                -318731 hrtimer_reprogram+0x14  
(enqueue_hrtimer+0xaa)
      #func                -318731 enqueue_hrtimer+0xe (hrtimer_start+0xad)
      #func                -318731 rb_erase+0xe (__remove_hrtimer+0x57)
      #func                -318731 hrtimer_force_reprogram+0xa  
(__remove_hrtimer+0x83)
      #func                -318731 rb_next+0x9 (__remove_hrtimer+0x5e)
      #func                -318731 __remove_hrtimer+0x16 (hrtimer_start+0x123)
      #func                -318731 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x49)
      #func                -318731 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x1e)
      #func                -318731 _spin_lock_irqsave+0x12  
(lock_hrtimer_base+0x28)
      #func                -318731 lock_hrtimer_base+0x16 (hrtimer_start+0x1e)
      #func                -318732 hrtimer_start+0xe  
(tick_nohz_stop_sched_tick+0x255)
      #func                -318732 hweight32+0x8  
(select_nohz_load_balancer+0x58)
      #func                -318732 hweight32+0x8  
(select_nohz_load_balancer+0x4a)
      #func                -318732 select_nohz_load_balancer+0xa  
(tick_nohz_stop_sched_tick+0x287)
      #func                -318732 rcu_needs_cpu+0x8  
(tick_nohz_stop_sched_tick+0x13c)
      #func                -318732 ipipe_check_context+0x14  
(_spin_unlock_irqrestore+0x23)
      #func                -318732 __ipipe_restore_root+0x8  
(_spin_unlock_irqrestore+0x19)
      #func                -318732 _spin_unlock_irqrestore+0x8  
(hrtimer_get_next_event+0xdb)
      #func                -318732 ipipe_check_context+0x14  
(_spin_lock_irqsave+0x49)


>> If it's ok, I guess we should include it in ipipe until someone (From
>> -rt) manages to get it accepted upstream (I didn't recall much activity
>> in this direction yet, though).
>
> Not true, the patch is in 2.6.27-rcX.
>
> But there is also
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ce6fce4295ba727b36fdc73040e444bd1aae64cd
> which makes me wonder, for 2.6.27, if that may generate cases where
> masking MSI interrupts will not work as expected for ipipe (Linux should
> catch masked IRQs internally). However, future problems...

Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

bernhard@domain.hid wrote:
>>> Found it. Could you give this patch a try and report the result?
>>>
>>> http://permalink.gmane.org/gmane.linux.kernel/682362
> 
> Applied and tested, no luck...
> 
> I-pipe: Detected illicit call from domain 'RTAI'
>          into a service reserved for domain 'Linux' and below.
> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #1
>   [<c0156866>] ipipe_check_context+0xd6/0xf0
> e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
>   [<c03e206e>] _spin_lock_irqsave+0x1e/0x80
>   [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>   [<c0254156>] __msi_set_enable+0x46/0x80
>   [<c01176f3>] ? mcount+0x1f/0x23
>   [<c0254498>] msi_set_mask_bits+0xd8/0xe0
>   [<c01176f3>] ? mcount+0x1f/0x23
>   [<c0254737>] unmask_msi_irq+0x17/0x30
>   [<c01542da>] default_enable+0x1a/0x30
>   [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>   [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>   [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>   [<c01021c5>] default_idle+0x45/0x60
>   [<c0102180>] default_idle+0x0/0x60
>   [<c0103cc7>] common_interrupt+0x2f/0x54
>   [<c0102180>] default_idle+0x0/0x60
>   [<c01500d8>] cgroup_file_write+0x118/0x140
>   [<c01021c5>] default_idle+0x45/0x60
>   [<c0101b76>] cpu_idle+0x86/0x140
>   [<c03dd7fd>] start_secondary+0x16d/0x210
>   [<c03d3a88>] initialize_secondary+0x8/0x20
>   =======================
> I-pipe tracer log (100 points):
>   |  +*func                    0 ipipe_trace_panic_freeze+0x9  

I see no option aside of ironing the inner code that reads/writes the PCI
config, so here is an ugly yet possible solution for x86, that might work
(totally untested):

diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
index 6e64aaf..7f32101 100644
--- a/arch/x86/pci/common.c
+++ b/arch/x86/pci/common.c
@@ -75,7 +75,7 @@ int pcibios_scanned;
  * This interrupt-safe spinlock protects all accesses to PCI
  * configuration space.
  */
-DEFINE_SPINLOCK(pci_config_lock);
+IPIPE_DEFINE_SPINLOCK(pci_config_lock);

 static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
 {
diff --git a/drivers/pci/access.c b/drivers/pci/access.c
index 39bb96b..9a74083 100644
--- a/drivers/pci/access.c
+++ b/drivers/pci/access.c
@@ -12,7 +12,7 @@
  * configuration space.
  */

-static DEFINE_SPINLOCK(pci_lock);
+static IPIPE_DEFINE_SPINLOCK(pci_lock);

 /*
  *  Wrappers for all PCI configuration access functions.  They just check

> (ipipe_check_context+0x94)
>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>   |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>   |  +*func                   -1 __first_cpu+0x8 (ipipe_check_context+0x66)
>   |  +*func                   -1 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x1e)
>   |  +*func                   -1 _spin_lock_irqsave+0x12  
> (pci_bus_read_config_word+0x36)
>   |  +*func                   -1 pci_bus_read_config_word+0x14  
> (__msi_set_enable+0x46)
>   |  +*func                   -1 __msi_set_enable+0x14 (msi_set_mask_bits+0xd8)
>   |  +*func                   -1 msi_set_mask_bits+0xe (unmask_msi_irq+0x17)
>   |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
>   |  +*func                   -1 default_enable+0x9 (rt_enable_irq+0xe  
> [rtai_hal])
>   |  +*func                   -2 alloc_rtskb+0x14 [rtnet]  
> (e1000_alloc_rx_buffers+0x147 [rt_e1000])
>   |  +*func                   -2 e1000_alloc_rx_buffers+0xe [rt_e1000]  
> (e1000_intr+0x37d [rt_e1000])
>   |  +*func                   -4 e1000_intr+0x11 [rt_e1000]  
> (xnintr_irq_handler+0x9e [rtai_rtdm])
>   |  +*func                   -5 xnintr_irq_handler+0xe [rtai_rtdm]  
> (rtai_hirq_dispatcher+0xfb [rtai_hal])
>   |  +*func                   -5 ack_ioapic_irq+0x8 (__ipipe_ack_edge_irq+0xe)
>   |  +*func                   -5 __ipipe_ack_edge_irq+0x8  
> (__ipipe_ack_irq+0x19)
>   |  +*func                   -5 __ipipe_ack_irq+0x8  
> (rtai_hirq_dispatcher+0x66 [rtai_hal])
>   |   +begin   0xffffff23     -5 common_interrupt+0x29 (default_idle+0x45)
>       +end     0x8000000e   -580 default_idle+0x43 (cpu_idle+0x86)
>       +func                 -580 default_idle+0x8 (cpu_idle+0x86)
>   |   +end     0x80000001   -581 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
>   |   #begin   0x80000001   -581 ipipe_suspend_domain+0xee (cpu_idle+0x84)
>       #func                 -581 ipipe_suspend_domain+0xe (cpu_idle+0x84)
>       +func                 -581 ipipe_check_context+0x14 (cpu_idle+0x5b)
>       +func                 -581 ipipe_check_context+0x14  
> (_spin_unlock_irqrestore+0x23)
>   |   +end     0x80000000   -581 __ipipe_unstall_root+0x4a  
> (__ipipe_restore_root+0x27)
>   |   #begin   0x80000000   -581 __ipipe_unstall_root+0x5b  
> (__ipipe_restore_root+0x27)
>       #func                 -581 __ipipe_unstall_root+0x8  
> (__ipipe_restore_root+0x27)
>       #func                 -581 __ipipe_restore_root+0x8  
> (_spin_unlock_irqrestore+0x3d)
>       #func                 -581 _spin_unlock_irqrestore+0x8  
> (rcu_check_callbacks+0x5c)
>       #func                 -582 __rcu_advance_callbacks+0x8  
> (rcu_check_callbacks+0x35)
>       #func                 -582 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x49)
>       +func                 -582 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x1e)
>       +func                 -582 _spin_lock_irqsave+0x12  
> (rcu_check_callbacks+0x2c)
>       +func                 -582 rcu_check_mb+0x8 (rcu_check_callbacks+0x1b)
>       +func                 -582 rcu_check_callbacks+0xa (cpu_idle+0xb2)
>       +func                 -582 rcu_pending+0x8 (cpu_idle+0xa5)
>   |   +end     0x8000000d   -583 __ipipe_unstall_iret_root+0x36  
> (restore_nocheck_notrace+0x0)
>   |   #func                 -583 __ipipe_unstall_iret_root+0x9  
> (restore_nocheck_notrace+0x0)
>   |   #end     0xffffff15   -583 ipipe_ipiX+0x3e (default_idle+0x45)
>   |   +end     0x8000000d   -583 __ipipe_unstall_iret_root+0x36  
> (restore_nocheck_notrace+0x0)
>   |   #func                 -583 __ipipe_unstall_iret_root+0x9  
> (restore_nocheck_notrace+0x0)
>       #func                 -585 __ipipe_do_critical_sync+0x9  
> (__ipipe_sync_stage+0x27b)
>   |   #end     0x80000000   -586 __ipipe_sync_stage+0x21f  
> (rtai_hirq_dispatcher+0x3a1 [rtai_hal])
>   |   +func                 -586 __ipipe_sync_stage+0xe  
> (rtai_hirq_dispatcher+0x3a1 [rtai_hal])
>   |   #func                 -586 __ipipe_ack_apic+0x8  
> (rtai_hirq_dispatcher+0x257 [rtai_hal])
>   |   +begin   0xffffff15   -587 ipipe_ipiX+0x2e (default_idle+0x45)
>       +end     0x8000000e  -318728 default_idle+0x43 (cpu_idle+0x86)
>       +func                -318728 default_idle+0x8 (cpu_idle+0x86)
>   |   +end     0x80000001  -318728 ipipe_suspend_domain+0xd7 (cpu_idle+0x84)
>   |   #begin   0x80000001  -318728 ipipe_suspend_domain+0xee (cpu_idle+0x84)
>       #func                -318728 ipipe_suspend_domain+0xe (cpu_idle+0x84)
>       +func                -318729 ipipe_check_context+0x14 (cpu_idle+0x5b)
>       +func                -318729 rcu_pending+0x8 (cpu_idle+0xa5)
>   |   +end     0x80000000  -318729 __ipipe_unstall_root+0x4a  
> (__ipipe_restore_root+0x27)
>   |   #begin   0x80000000  -318729 __ipipe_unstall_root+0x5b  
> (__ipipe_restore_root+0x27)
>       #func                -318729 __ipipe_unstall_root+0x8  
> (__ipipe_restore_root+0x27)
>       #func                -318729 __ipipe_restore_root+0x8  
> (tick_nohz_stop_sched_tick+0x23b)
>       #func                -318729 ipipe_check_context+0x14  
> (hrtimer_start+0xe1)
>       #func                -318729 ipipe_check_context+0x14  
> (_spin_unlock_irqrestore+0x23)
>       #func                -318729 __ipipe_restore_root+0x8  
> (_spin_unlock_irqrestore+0x19)
>       #func                -318729 _spin_unlock_irqrestore+0x8  
> (hrtimer_start+0xd2)
>       #func                -318729 ipipe_check_context+0x14  
> (hrtimer_start+0xba)
>       #func                -318730 rb_insert_color+0xe (enqueue_hrtimer+0x7d)
>       #func                -318730 lapic_next_event+0x8  
> (clockevents_program_event+0x9e)
>       #func                -318730 clockevents_program_event+0x14  
> (tick_program_event+0x44)
>       #func                -318730 set_normalized_timespec+0x8  
> (ktime_get_ts+0x43)
>       #func                -318730 native_read_tsc+0x8 (read_tsc+0xe)
>       #func                -318730 read_tsc+0x9 (getnstimeofday+0x48)
>       #func                -318730 getnstimeofday+0xe (ktime_get_ts+0x22)
>       #func                -318730 ktime_get_ts+0xa (ktime_get+0x1e)
>       #func                -318730 ktime_get+0x14 (tick_program_event+0x29)
>       #func                -318730 tick_program_event+0xe  
> (hrtimer_reprogram+0x86)
>       #func                -318731 hrtimer_reprogram+0x14  
> (enqueue_hrtimer+0xaa)
>       #func                -318731 enqueue_hrtimer+0xe (hrtimer_start+0xad)
>       #func                -318731 rb_erase+0xe (__remove_hrtimer+0x57)
>       #func                -318731 hrtimer_force_reprogram+0xa  
> (__remove_hrtimer+0x83)
>       #func                -318731 rb_next+0x9 (__remove_hrtimer+0x5e)
>       #func                -318731 __remove_hrtimer+0x16 (hrtimer_start+0x123)
>       #func                -318731 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x49)
>       #func                -318731 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x1e)
>       #func                -318731 _spin_lock_irqsave+0x12  
> (lock_hrtimer_base+0x28)
>       #func                -318731 lock_hrtimer_base+0x16 (hrtimer_start+0x1e)
>       #func                -318732 hrtimer_start+0xe  
> (tick_nohz_stop_sched_tick+0x255)
>       #func                -318732 hweight32+0x8  
> (select_nohz_load_balancer+0x58)
>       #func                -318732 hweight32+0x8  
> (select_nohz_load_balancer+0x4a)
>       #func                -318732 select_nohz_load_balancer+0xa  
> (tick_nohz_stop_sched_tick+0x287)
>       #func                -318732 rcu_needs_cpu+0x8  
> (tick_nohz_stop_sched_tick+0x13c)
>       #func                -318732 ipipe_check_context+0x14  
> (_spin_unlock_irqrestore+0x23)
>       #func                -318732 __ipipe_restore_root+0x8  
> (_spin_unlock_irqrestore+0x19)
>       #func                -318732 _spin_unlock_irqrestore+0x8  
> (hrtimer_get_next_event+0xdb)
>       #func                -318732 ipipe_check_context+0x14  
> (_spin_lock_irqsave+0x49)
> 
> 
>>> If it's ok, I guess we should include it in ipipe until someone (From
>>> -rt) manages to get it accepted upstream (I didn't recall much activity
>>> in this direction yet, though).
>> Not true, the patch is in 2.6.27-rcX.
>>
>> But there is also
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ce6fce4295ba727b36fdc73040e444bd1aae64cd
>> which makes me wonder, for 2.6.27, if that may generate cases where
>> masking MSI interrupts will not work as expected for ipipe (Linux should
>> catch masked IRQs internally). However, future problems...
> 
> Bernhard
> 
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main
> 


-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Gilles Chanteperdrix]

Philippe Gerum wrote:
>> (ipipe_check_context+0x94)
>>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>   |  +*func                    0 find_next_bit+0xa (__next_cpu+0x1a)
>>   |  +*func                    0 __next_cpu+0x9 (ipipe_check_context+0x88)
>>   |  +*func                    0 find_first_bit+0xa (__first_cpu+0x12)
>>   |  +*func                   -1 __first_cpu+0x8 (ipipe_check_context+0x66)
>>   |  +*func                   -1 ipipe_check_context+0x14  
>> (_spin_lock_irqsave+0x1e)
>>   |  +*func                   -1 _spin_lock_irqsave+0x12  
>> (pci_bus_read_config_word+0x36)
>>   |  +*func                   -1 pci_bus_read_config_word+0x14  
>> (__msi_set_enable+0x46)
>>   |  +*func                   -1 __msi_set_enable+0x14 (msi_set_mask_bits+0xd8)
>>   |  +*func                   -1 msi_set_mask_bits+0xe (unmask_msi_irq+0x17)
>>   |  +*func                   -1 unmask_msi_irq+0x9 (default_enable+0x1a)
>>   |  +*func                   -1 default_enable+0x9 (rt_enable_irq+0xe  

Excuse my naive question: but for every MSI interrupts, we need to issue
a PCI read and look for some bits ? Is not this a bit insane ?

-- 
                                                 Gilles.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Bernhard Pfund]

> I see no option aside of ironing the inner code that reads/writes the PCI
> config, so here is an ugly yet possible solution for x86, that might work
> (totally untested):
> 
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index 6e64aaf..7f32101 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -75,7 +75,7 @@ int pcibios_scanned;
>   * This interrupt-safe spinlock protects all accesses to PCI
>   * configuration space.
>   */
> -DEFINE_SPINLOCK(pci_config_lock);
> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
> 
>  static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
>  {
> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
> index 39bb96b..9a74083 100644
> --- a/drivers/pci/access.c
> +++ b/drivers/pci/access.c
> @@ -12,7 +12,7 @@
>   * configuration space.
>   */
> 
> -static DEFINE_SPINLOCK(pci_lock);
> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
> 
>  /*
>   *  Wrappers for all PCI configuration access functions.  They just check
> 

This results in:

arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
was here

Didn't look into it, just tried ;)

Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Gilles Chanteperdrix]

Bernhard Pfund wrote:
>> I see no option aside of ironing the inner code that reads/writes the PCI
>> config, so here is an ugly yet possible solution for x86, that might work
>> (totally untested):
>>
>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>> index 6e64aaf..7f32101 100644
>> --- a/arch/x86/pci/common.c
>> +++ b/arch/x86/pci/common.c
>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>   * This interrupt-safe spinlock protects all accesses to PCI
>>   * configuration space.
>>   */
>> -DEFINE_SPINLOCK(pci_config_lock);
>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>
>>  static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
>>  {
>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>> index 39bb96b..9a74083 100644
>> --- a/drivers/pci/access.c
>> +++ b/drivers/pci/access.c
>> @@ -12,7 +12,7 @@
>>   * configuration space.
>>   */
>>
>> -static DEFINE_SPINLOCK(pci_lock);
>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>
>>  /*
>>   *  Wrappers for all PCI configuration access functions.  They just check
>>
> 
> This results in:
> 
> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
> was here
> 
> Didn't look into it, just tried ;)

You probably need to replace the DECLARE_SPINLOCK in pci.h with
IPIPE_DECLARE_SPINLOCK...

-- 
                                                 Gilles.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Bernhard Pfund wrote:
>> I see no option aside of ironing the inner code that reads/writes the PCI
>> config, so here is an ugly yet possible solution for x86, that might work
>> (totally untested):
>>
>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>> index 6e64aaf..7f32101 100644
>> --- a/arch/x86/pci/common.c
>> +++ b/arch/x86/pci/common.c
>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>   * This interrupt-safe spinlock protects all accesses to PCI
>>   * configuration space.
>>   */
>> -DEFINE_SPINLOCK(pci_config_lock);
>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>
>>  static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
>>  {
>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>> index 39bb96b..9a74083 100644
>> --- a/drivers/pci/access.c
>> +++ b/drivers/pci/access.c
>> @@ -12,7 +12,7 @@
>>   * configuration space.
>>   */
>>
>> -static DEFINE_SPINLOCK(pci_lock);
>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>
>>  /*
>>   *  Wrappers for all PCI configuration access functions.  They just check
>>
> 
> This results in:
> 
> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
> was here
> 
> Didn't look into it, just tried ;)
>

Just change the declaration in pci.h the same way.

> Bernhard
> 


-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Bernhard Pfund]

Philippe Gerum wrote:
> Bernhard Pfund wrote:
>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>> config, so here is an ugly yet possible solution for x86, that might work
>>> (totally untested):
>>>
>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>> index 6e64aaf..7f32101 100644
>>> --- a/arch/x86/pci/common.c
>>> +++ b/arch/x86/pci/common.c
>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>   * configuration space.
>>>   */
>>> -DEFINE_SPINLOCK(pci_config_lock);
>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>
>>>  static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
>>>  {
>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>> index 39bb96b..9a74083 100644
>>> --- a/drivers/pci/access.c
>>> +++ b/drivers/pci/access.c
>>> @@ -12,7 +12,7 @@
>>>   * configuration space.
>>>   */
>>>
>>> -static DEFINE_SPINLOCK(pci_lock);
>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>
>>>  /*
>>>   *  Wrappers for all PCI configuration access functions.  They just check
>>>
>> This results in:
>>
>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>> was here
>>
>> Didn't look into it, just tried ;)
>>
> 
> Just change the declaration in pci.h the same way.
> 

Ok, thanx! Seems to work for now, no extensive testing done (yet)
though. What's the plan for the future? Will this change find its way
into the official patch?

Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Bernhard Pfund wrote:
> Philippe Gerum wrote:
>> Bernhard Pfund wrote:
>>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>>> config, so here is an ugly yet possible solution for x86, that might work
>>>> (totally untested):
>>>>
>>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>>> index 6e64aaf..7f32101 100644
>>>> --- a/arch/x86/pci/common.c
>>>> +++ b/arch/x86/pci/common.c
>>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>>   * configuration space.
>>>>   */
>>>> -DEFINE_SPINLOCK(pci_config_lock);
>>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>>
>>>>  static int __devinit can_skip_ioresource_align(const struct dmi_system_id *d)
>>>>  {
>>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>>> index 39bb96b..9a74083 100644
>>>> --- a/drivers/pci/access.c
>>>> +++ b/drivers/pci/access.c
>>>> @@ -12,7 +12,7 @@
>>>>   * configuration space.
>>>>   */
>>>>
>>>> -static DEFINE_SPINLOCK(pci_lock);
>>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>>
>>>>  /*
>>>>   *  Wrappers for all PCI configuration access functions.  They just check
>>>>
>>> This results in:
>>>
>>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>>> was here
>>>
>>> Didn't look into it, just tried ;)
>>>
>> Just change the declaration in pci.h the same way.
>>
> 
> Ok, thanx! Seems to work for now, no extensive testing done (yet)
> though. What's the plan for the future? Will this change find its way
> into the official patch?
> 

This change could be merged into the 2.6.26 patch provided it does not raise any
pathological latency when enabling MSI. I would rather ask people to refrain
from using MSI until it is fixed (once again) in later releases, than suffering
random latency peaks. 2.6.27 and beyond is another issue; this will need a
different approach than simply ironing the PCI lock in any case.

We need more test data for 2.6.26 + this patch.

-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[bernhard]

Zitat von Philippe Gerum <rpm@xenomai.org>:

> Bernhard Pfund wrote:
>> Philippe Gerum wrote:
>>> Bernhard Pfund wrote:
>>>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>>>> config, so here is an ugly yet possible solution for x86, that might work
>>>>> (totally untested):
>>>>>
>>>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>>>> index 6e64aaf..7f32101 100644
>>>>> --- a/arch/x86/pci/common.c
>>>>> +++ b/arch/x86/pci/common.c
>>>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>>>   * configuration space.
>>>>>   */
>>>>> -DEFINE_SPINLOCK(pci_config_lock);
>>>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>>>
>>>>>  static int __devinit can_skip_ioresource_align(const struct   
>>>>> dmi_system_id *d)
>>>>>  {
>>>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>>>> index 39bb96b..9a74083 100644
>>>>> --- a/drivers/pci/access.c
>>>>> +++ b/drivers/pci/access.c
>>>>> @@ -12,7 +12,7 @@
>>>>>   * configuration space.
>>>>>   */
>>>>>
>>>>> -static DEFINE_SPINLOCK(pci_lock);
>>>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>>>
>>>>>  /*
>>>>>   *  Wrappers for all PCI configuration access functions.  They   
>>>>> just check
>>>>>
>>>> This results in:
>>>>
>>>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>>>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>>>> was here
>>>>
>>>> Didn't look into it, just tried ;)
>>>>
>>> Just change the declaration in pci.h the same way.
>>>
>>
>> Ok, thanx! Seems to work for now, no extensive testing done (yet)
>> though. What's the plan for the future? Will this change find its way
>> into the official patch?
>>
>
> This change could be merged into the 2.6.26 patch provided it does   
> not raise any
> pathological latency when enabling MSI. I would rather ask people to refrain
> from using MSI until it is fixed (once again) in later releases,   
> than suffering
> random latency peaks. 2.6.27 and beyond is another issue; this will need a
> different approach than simply ironing the PCI lock in any case.
>
> We need more test data for 2.6.26 + this patch.
>

Let me know if I can be of any help. I'm in an early stage of the  
project, so there's some time available for MSI experiments...

Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

bernhard@domain.hid wrote:
> Zitat von Philippe Gerum <rpm@xenomai.org>:
> 
>> Bernhard Pfund wrote:
>>> Philippe Gerum wrote:
>>>> Bernhard Pfund wrote:
>>>>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>>>>> config, so here is an ugly yet possible solution for x86, that might work
>>>>>> (totally untested):
>>>>>>
>>>>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>>>>> index 6e64aaf..7f32101 100644
>>>>>> --- a/arch/x86/pci/common.c
>>>>>> +++ b/arch/x86/pci/common.c
>>>>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>>>>   * configuration space.
>>>>>>   */
>>>>>> -DEFINE_SPINLOCK(pci_config_lock);
>>>>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>>>>
>>>>>>  static int __devinit can_skip_ioresource_align(const struct   
>>>>>> dmi_system_id *d)
>>>>>>  {
>>>>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>>>>> index 39bb96b..9a74083 100644
>>>>>> --- a/drivers/pci/access.c
>>>>>> +++ b/drivers/pci/access.c
>>>>>> @@ -12,7 +12,7 @@
>>>>>>   * configuration space.
>>>>>>   */
>>>>>>
>>>>>> -static DEFINE_SPINLOCK(pci_lock);
>>>>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>>>>
>>>>>>  /*
>>>>>>   *  Wrappers for all PCI configuration access functions.  They   
>>>>>> just check
>>>>>>
>>>>> This results in:
>>>>>
>>>>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>>>>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>>>>> was here
>>>>>
>>>>> Didn't look into it, just tried ;)
>>>>>
>>>> Just change the declaration in pci.h the same way.
>>>>
>>> Ok, thanx! Seems to work for now, no extensive testing done (yet)
>>> though. What's the plan for the future? Will this change find its way
>>> into the official patch?
>>>
>> This change could be merged into the 2.6.26 patch provided it does   
>> not raise any
>> pathological latency when enabling MSI. I would rather ask people to refrain
>> from using MSI until it is fixed (once again) in later releases,   
>> than suffering
>> random latency peaks. 2.6.27 and beyond is another issue; this will need a
>> different approach than simply ironing the PCI lock in any case.
>>
>> We need more test data for 2.6.26 + this patch.
>>
> 
> Let me know if I can be of any help. I'm in an early stage of the  
> project, so there's some time available for MSI experiments...
> 

Thanks. Basically, we need to know the impact of this patch under high load for
at least four hours runtime, with significant interrupt traffic to/from MSI
devices in parallel.

-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Bernhard Pfund]

Philippe Gerum wrote:
> bernhard@domain.hid wrote:
>> Zitat von Philippe Gerum <rpm@xenomai.org>:
>>
>>> Bernhard Pfund wrote:
>>>> Philippe Gerum wrote:
>>>>> Bernhard Pfund wrote:
>>>>>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>>>>>> config, so here is an ugly yet possible solution for x86, that might work
>>>>>>> (totally untested):
>>>>>>>
>>>>>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>>>>>> index 6e64aaf..7f32101 100644
>>>>>>> --- a/arch/x86/pci/common.c
>>>>>>> +++ b/arch/x86/pci/common.c
>>>>>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>>>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>>>>>   * configuration space.
>>>>>>>   */
>>>>>>> -DEFINE_SPINLOCK(pci_config_lock);
>>>>>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>>>>>
>>>>>>>  static int __devinit can_skip_ioresource_align(const struct   
>>>>>>> dmi_system_id *d)
>>>>>>>  {
>>>>>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>>>>>> index 39bb96b..9a74083 100644
>>>>>>> --- a/drivers/pci/access.c
>>>>>>> +++ b/drivers/pci/access.c
>>>>>>> @@ -12,7 +12,7 @@
>>>>>>>   * configuration space.
>>>>>>>   */
>>>>>>>
>>>>>>> -static DEFINE_SPINLOCK(pci_lock);
>>>>>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>>>>>
>>>>>>>  /*
>>>>>>>   *  Wrappers for all PCI configuration access functions.  They   
>>>>>>> just check
>>>>>>>
>>>>>> This results in:
>>>>>>
>>>>>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>>>>>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>>>>>> was here
>>>>>>
>>>>>> Didn't look into it, just tried ;)
>>>>>>
>>>>> Just change the declaration in pci.h the same way.
>>>>>
>>>> Ok, thanx! Seems to work for now, no extensive testing done (yet)
>>>> though. What's the plan for the future? Will this change find its way
>>>> into the official patch?
>>>>
>>> This change could be merged into the 2.6.26 patch provided it does   
>>> not raise any
>>> pathological latency when enabling MSI. I would rather ask people to refrain
>>> from using MSI until it is fixed (once again) in later releases,   
>>> than suffering
>>> random latency peaks. 2.6.27 and beyond is another issue; this will need a
>>> different approach than simply ironing the PCI lock in any case.
>>>
>>> We need more test data for 2.6.26 + this patch.
>>>
>> Let me know if I can be of any help. I'm in an early stage of the  
>> project, so there's some time available for MSI experiments...
>>
> 
> Thanks. Basically, we need to know the impact of this patch under high load for
> at least four hours runtime, with significant interrupt traffic to/from MSI
> devices in parallel.
> 

Hmm, I think I actually have a setup that could do the trick. One GBit
NIC under control of RTnet and another in the Linux domain, both PCIe
devices. MSIs are enabled in the kernel (2.6.26.2), without MSI they
would share the same IRQ.

I could connect them to the same physical network and ping flood both
cards from a second system, for 24h if necessary.

How would you measure the effect of the patch?

HTH::Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Bernhard Pfund wrote:
> Philippe Gerum wrote:
>> bernhard@domain.hid wrote:
>>> Zitat von Philippe Gerum <rpm@xenomai.org>:
>>>
>>>> Bernhard Pfund wrote:
>>>>> Philippe Gerum wrote:
>>>>>> Bernhard Pfund wrote:
>>>>>>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>>>>>>> config, so here is an ugly yet possible solution for x86, that might work
>>>>>>>> (totally untested):
>>>>>>>>
>>>>>>>> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
>>>>>>>> index 6e64aaf..7f32101 100644
>>>>>>>> --- a/arch/x86/pci/common.c
>>>>>>>> +++ b/arch/x86/pci/common.c
>>>>>>>> @@ -75,7 +75,7 @@ int pcibios_scanned;
>>>>>>>>   * This interrupt-safe spinlock protects all accesses to PCI
>>>>>>>>   * configuration space.
>>>>>>>>   */
>>>>>>>> -DEFINE_SPINLOCK(pci_config_lock);
>>>>>>>> +IPIPE_DEFINE_SPINLOCK(pci_config_lock);
>>>>>>>>
>>>>>>>>  static int __devinit can_skip_ioresource_align(const struct   
>>>>>>>> dmi_system_id *d)
>>>>>>>>  {
>>>>>>>> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
>>>>>>>> index 39bb96b..9a74083 100644
>>>>>>>> --- a/drivers/pci/access.c
>>>>>>>> +++ b/drivers/pci/access.c
>>>>>>>> @@ -12,7 +12,7 @@
>>>>>>>>   * configuration space.
>>>>>>>>   */
>>>>>>>>
>>>>>>>> -static DEFINE_SPINLOCK(pci_lock);
>>>>>>>> +static IPIPE_DEFINE_SPINLOCK(pci_lock);
>>>>>>>>
>>>>>>>>  /*
>>>>>>>>   *  Wrappers for all PCI configuration access functions.  They   
>>>>>>>> just check
>>>>>>>>
>>>>>>> This results in:
>>>>>>>
>>>>>>> arch/x86/pci/common.c:78: error: conflicting types for ‘pci_config_lock’
>>>>>>> arch/x86/pci/pci.h:84: error: previous declaration of ‘pci_config_lock’
>>>>>>> was here
>>>>>>>
>>>>>>> Didn't look into it, just tried ;)
>>>>>>>
>>>>>> Just change the declaration in pci.h the same way.
>>>>>>
>>>>> Ok, thanx! Seems to work for now, no extensive testing done (yet)
>>>>> though. What's the plan for the future? Will this change find its way
>>>>> into the official patch?
>>>>>
>>>> This change could be merged into the 2.6.26 patch provided it does   
>>>> not raise any
>>>> pathological latency when enabling MSI. I would rather ask people to refrain
>>>> from using MSI until it is fixed (once again) in later releases,   
>>>> than suffering
>>>> random latency peaks. 2.6.27 and beyond is another issue; this will need a
>>>> different approach than simply ironing the PCI lock in any case.
>>>>
>>>> We need more test data for 2.6.26 + this patch.
>>>>
>>> Let me know if I can be of any help. I'm in an early stage of the  
>>> project, so there's some time available for MSI experiments...
>>>
>> Thanks. Basically, we need to know the impact of this patch under high load for
>> at least four hours runtime, with significant interrupt traffic to/from MSI
>> devices in parallel.
>>
> 
> Hmm, I think I actually have a setup that could do the trick. One GBit
> NIC under control of RTnet and another in the Linux domain, both PCIe
> devices. MSIs are enabled in the kernel (2.6.26.2), without MSI they
> would share the same IRQ.
> 
> I could connect them to the same physical network and ping flood both
> cards from a second system, for 24h if necessary.
> 
> How would you measure the effect of the patch?
>

The risk in ironing those PCI locks is to run with hw interrupts disabled for a
long time, inducing pathological latencies, so running RTAI's latency test in
the background should help detecting those peaks.

However, we may find nothing bad if the kernel uses the MMConfig access method
to the PCI space since this is basically fast mmio there. But since you seem to
be running on x86_32, we may want to check whether BIOS or direct access to the
PCI config does not raise the typical latency too much, as well (I'm unsure that
PCI_GOBIOS will give us decent results though).

To sum up: with different settings for the PCI config access method in "Bus
options" (by order of criticality, MMConfig then Direct, then maybe BIOS), does
the latency tool report pathological peaks?

TIA,

> HTH::Bernhard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[bernhard]

Zitat von Philippe Gerum <rpm@xenomai.org>:

> The risk in ironing those PCI locks is to run with hw interrupts   
> disabled for a
> long time, inducing pathological latencies, so running RTAI's latency test in
> the background should help detecting those peaks.
>
> However, we may find nothing bad if the kernel uses the MMConfig   
> access method
> to the PCI space since this is basically fast mmio there. But since   
> you seem to
> be running on x86_32, we may want to check whether BIOS or direct   
> access to the
> PCI config does not raise the typical latency too much, as well (I'm  
>  unsure that
> PCI_GOBIOS will give us decent results though).
>
> To sum up: with different settings for the PCI config access method in "Bus
> options" (by order of criticality, MMConfig then Direct, then maybe   
> BIOS), does
> the latency tool report pathological peaks?
>

Hi Philippe

I played with the different PCI configurations and the results are  
devastating. Latencies (and jitter) skyrocket after some minutes of  
testing and peak at several milliseconds. I didn't do the regression  
with 'normal' INTs though, but that's something up next. Additionally  
MMCONFIG produced some strange msg at boot.

I suspect I'm going to use INTs for my current project :(

Bernhard


PCI_GOMMCONFIG
--------------

I get the following dmesg:

ACPI: bus type pci registered
PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 31
PCI: Not using MMCONFIG.
PCI: Fatal: No config space access function found
Setting up standard PCI resources
ACPI: EC: Look up EC in DSDT
ACPI Exception (evregion-0420): AE_ERROR, Returned by Handler for
[PCI_Config] [20080321]
ACPI Error (psparse-0530): Method parse/execution failed
[\_SB_.PCI0.LNK1._STA] (Node f7835e3c), AE_ERROR
ACPI Error (uteval-0233): Method execution failed [\_SB_.PCI0.LNK1._STA]
(Node f7835e3c), AE_ERROR
ACPI Exception (evregion-0420): AE_ERROR, Returned by Handler for
[PCI_Config] [20080321]
ACPI Error (psparse-0530): Method parse/execution failed
[\_SB_.PCI0.LNK2._STA] (Node f7835d9c), AE_ERROR
ACPI Error (uteval-0233): Method execution failed [\_SB_.PCI0.LNK2._STA]
(Node f7835d9c), AE_ERROR
ACPI Exception (evregion-0420): AE_ERROR, Returned by Handler for
[PCI_Config] [20080321]
ACPI Error (psparse-0530): Method parse/execution failed
[\_SB_.PCI0.LNK3._STA] (Node f7835cfc), AE_ERROR
ACPI Error (uteval-0233): Method execution failed [\_SB_.PCI0.LNK3._STA]
(Node f7835cfc), AE_ERROR
ACPI Exception (evregion-0420): AE_ERROR, Returned by Handler for
[PCI_Config] [20080321]
ACPI Error (psparse-0530): Method parse/execution failed
[\_SB_.PCI0.LNK4._STA] (Node f7835c5c), AE_ERROR

[....]


PCI_GODIRECT
------------

Here desg looks better:

ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
Setting up standard PCI resources
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Transparent bridge - 0000:00:0a.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.HUB0._PRT]
ACPI: PCI Interrupt Link [LNK1] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNK2] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNK3] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNK4] (IRQs 5 7 9 *10 11 14 15)
ACPI: PCI Interrupt Link [LNK5] (IRQs 5 7 9 *10 11 14 15)
ACPI: PCI Interrupt Link [LNK6] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNK7] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNK8] (IRQs 5 7 9 10 *11 14 15)
ACPI: PCI Interrupt Link [LP2P] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LIGP] (IRQs *5 7 9 10 11 14 15)
ACPI: PCI Interrupt Link [LUBA] (IRQs 5 7 9 10 *11 14 15)
ACPI: PCI Interrupt Link [LUB2] (IRQs *5 7 9 10 11 14 15)
ACPI: PCI Interrupt Link [LU1B] (IRQs 5 7 9 10 11 14 15) *0
ACPI: PCI Interrupt Link [LU2B] (IRQs 5 7 9 10 11 14 15) *0
ACPI: PCI Interrupt Link [LMAC] (IRQs 5 7 9 10 11 14 *15)
ACPI: PCI Interrupt Link [LAZA] (IRQs 5 7 9 10 *11 14 15)
ACPI: PCI Interrupt Link [LPMU] (IRQs 5 7 9 *10 11 14 15)
ACPI: PCI Interrupt Link [LSMB] (IRQs 5 7 9 *10 11 14 15)
ACPI: PCI Interrupt Link [LIDE] (IRQs 5 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LSID] (IRQs 5 7 9 10 *11 14 15)
ACPI: PCI Interrupt Link [APC1] (IRQs 16) *0, disabled.
ACPI: PCI Interrupt Link [APC2] (IRQs 17) *0, disabled.
ACPI: PCI Interrupt Link [APC3] (IRQs 18) *0, disabled.
ACPI: PCI Interrupt Link [APC4] (IRQs 19) *0
ACPI: PCI Interrupt Link [APC5] (IRQs 16) *0
ACPI: PCI Interrupt Link [APC6] (IRQs 16) *0, disabled.
ACPI: PCI Interrupt Link [APC7] (IRQs 16) *0, disabled.
ACPI: PCI Interrupt Link [APC8] (IRQs 16) *0
ACPI: PCI Interrupt Link [AIGP] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [AUBA] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [AUB2] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [AU1B] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [AU2B] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [APCH] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [APMU] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [AAZA] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [APCS] (IRQs 20 21 22 23) *0
ACPI: PCI Interrupt Link [APCM] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [APCZ] (IRQs 20 21 22 23) *0, disabled.
ACPI: PCI Interrupt Link [APSI] (IRQs 20 21 22 23) *0

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Jan Kiszka]

bernhard@domain.hid wrote:
> Zitat von Philippe Gerum <rpm@xenomai.org>:
> 
>> The risk in ironing those PCI locks is to run with hw interrupts   
>> disabled for a
>> long time, inducing pathological latencies, so running RTAI's latency test in
>> the background should help detecting those peaks.
>>
>> However, we may find nothing bad if the kernel uses the MMConfig   
>> access method
>> to the PCI space since this is basically fast mmio there. But since   
>> you seem to
>> be running on x86_32, we may want to check whether BIOS or direct   
>> access to the
>> PCI config does not raise the typical latency too much, as well (I'm  
>>  unsure that
>> PCI_GOBIOS will give us decent results though).
>>
>> To sum up: with different settings for the PCI config access method in "Bus
>> options" (by order of criticality, MMConfig then Direct, then maybe   
>> BIOS), does
>> the latency tool report pathological peaks?
>>
> 
> Hi Philippe
> 
> I played with the different PCI configurations and the results are  
> devastating. Latencies (and jitter) skyrocket after some minutes of  
> testing and peak at several milliseconds. I didn't do the regression  
> with 'normal' INTs though, but that's something up next. Additionally  
> MMCONFIG produced some strange msg at boot.

Could you catch some path traces with the ipipe latency tracer when this
happens? See [1] for details. Dunno if RTAI's latency tool is prepared
to support you here (triggering a trace on new peaks), but
CONFIG_IPIPE_TRACE_IRQSOFF should already suffice in this case - if the
latency is due to IRQ disabling over PCI code.

Thanks,
Jan

[1] http://www.xenomai.org/index.php/I-pipe:Tracer

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[bernhard]

Zitat von Jan Kiszka <jan.kiszka@domain.hid>:

> bernhard@domain.hid wrote:
>> Zitat von Philippe Gerum <rpm@xenomai.org>:
>>
>>> The risk in ironing those PCI locks is to run with hw interrupts
>>> disabled for a
>>> long time, inducing pathological latencies, so running RTAI's   
>>> latency test in
>>> the background should help detecting those peaks.
>>>
>>> However, we may find nothing bad if the kernel uses the MMConfig
>>> access method
>>> to the PCI space since this is basically fast mmio there. But since
>>> you seem to
>>> be running on x86_32, we may want to check whether BIOS or direct
>>> access to the
>>> PCI config does not raise the typical latency too much, as well (I'm
>>>  unsure that
>>> PCI_GOBIOS will give us decent results though).
>>>
>>> To sum up: with different settings for the PCI config access method in "Bus
>>> options" (by order of criticality, MMConfig then Direct, then maybe
>>> BIOS), does
>>> the latency tool report pathological peaks?
>>>
>>
>> Hi Philippe
>>
>> I played with the different PCI configurations and the results are
>> devastating. Latencies (and jitter) skyrocket after some minutes of
>> testing and peak at several milliseconds. I didn't do the regression
>> with 'normal' INTs though, but that's something up next. Additionally
>> MMCONFIG produced some strange msg at boot.
>
> Could you catch some path traces with the ipipe latency tracer when this
> happens? See [1] for details. Dunno if RTAI's latency tool is prepared
> to support you here (triggering a trace on new peaks), but
> CONFIG_IPIPE_TRACE_IRQSOFF should already suffice in this case - if the
> latency is due to IRQ disabling over PCI code.
>
> Thanks,
> Jan
>
> [1] http://www.xenomai.org/index.php/I-pipe:Tracer

I'll have a look. Should be no problem to modify the RTAI testsuite to  
trigger traces in the event of a new peak. Just moved back to INTs, so  
I have to rebuild... Will test and report.

Bernhard

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

bernhard@domain.hid wrote:
> Zitat von Philippe Gerum <rpm@xenomai.org>:
> 
>> The risk in ironing those PCI locks is to run with hw interrupts   
>> disabled for a
>> long time, inducing pathological latencies, so running RTAI's latency test in
>> the background should help detecting those peaks.
>>
>> However, we may find nothing bad if the kernel uses the MMConfig   
>> access method
>> to the PCI space since this is basically fast mmio there. But since   
>> you seem to
>> be running on x86_32, we may want to check whether BIOS or direct   
>> access to the
>> PCI config does not raise the typical latency too much, as well (I'm  
>>  unsure that
>> PCI_GOBIOS will give us decent results though).
>>
>> To sum up: with different settings for the PCI config access method in "Bus
>> options" (by order of criticality, MMConfig then Direct, then maybe   
>> BIOS), does
>> the latency tool report pathological peaks?
>>
> 
> Hi Philippe
> 
> I played with the different PCI configurations and the results are  
> devastating. Latencies (and jitter) skyrocket after some minutes of  
> testing and peak at several milliseconds. I didn't do the regression  
> with 'normal' INTs though, but that's something up next. Additionally  
> MMCONFIG produced some strange msg at boot.
> 
> I suspect I'm going to use INTs for my current project :(
> 
> Bernhard
> 
> 
> PCI_GOMMCONFIG
> --------------
> 
> I get the following dmesg:
> 
> ACPI: bus type pci registered
> PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 31
> PCI: Not using MMCONFIG.
> PCI: Fatal: No config space access function found

Your hw does not seem to have PCI Express support, hence MMConfig access method
is not available. The kernel is expected to downgrade to direct access method.

> [....]
> 
> 
> PCI_GODIRECT
> ------------
> 
> Here desg looks better:
> 
> ACPI: bus type pci registered
> PCI: Using configuration type 1 for base access

Direct access type 1 is done through in/out{b, w, l} commands, which is not
cheap. Well, if it proves that we don't face any unknown issue with direct
access, only getting virtualized masking/unmasking of MSI driven interrupts
would save the day.

-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

Philippe Gerum wrote:
> bernhard@domain.hid wrote:
>>>> Found it. Could you give this patch a try and report the result?
>>>>
>>>> http://permalink.gmane.org/gmane.linux.kernel/682362
>> Applied and tested, no luck...
>>
>> I-pipe: Detected illicit call from domain 'RTAI'
>>          into a service reserved for domain 'Linux' and below.
>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #1
>>   [<c0156866>] ipipe_check_context+0xd6/0xf0
>> e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
>>   [<c03e206e>] _spin_lock_irqsave+0x1e/0x80
>>   [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>   [<c0254156>] __msi_set_enable+0x46/0x80
>>   [<c01176f3>] ? mcount+0x1f/0x23
>>   [<c0254498>] msi_set_mask_bits+0xd8/0xe0
>>   [<c01176f3>] ? mcount+0x1f/0x23
>>   [<c0254737>] unmask_msi_irq+0x17/0x30
>>   [<c01542da>] default_enable+0x1a/0x30
>>   [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>   [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>   [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>   [<c01021c5>] default_idle+0x45/0x60
>>   [<c0102180>] default_idle+0x0/0x60
>>   [<c0103cc7>] common_interrupt+0x2f/0x54
>>   [<c0102180>] default_idle+0x0/0x60
>>   [<c01500d8>] cgroup_file_write+0x118/0x140
>>   [<c01021c5>] default_idle+0x45/0x60
>>   [<c0101b76>] cpu_idle+0x86/0x140
>>   [<c03dd7fd>] start_secondary+0x16d/0x210
>>   [<c03d3a88>] initialize_secondary+0x8/0x20
>>   =======================
>> I-pipe tracer log (100 points):
>>   |  +*func                    0 ipipe_trace_panic_freeze+0x9  
> 
> I see no option aside of ironing the inner code that reads/writes the PCI
> config, so here is an ugly yet possible solution for x86, that might work
> (totally untested):

Very ugly. There is potentially some heavy code under this lock. 
Wouldn't it be better to switch to soft-disabling directly, also given 
that not all devices will support that method anyway?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Jan Kiszka wrote:
> Philippe Gerum wrote:
>> bernhard@domain.hid wrote:
>>>>> Found it. Could you give this patch a try and report the result?
>>>>>
>>>>> http://permalink.gmane.org/gmane.linux.kernel/682362
>>> Applied and tested, no luck...
>>>
>>> I-pipe: Detected illicit call from domain 'RTAI'
>>>          into a service reserved for domain 'Linux' and below.
>>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #1
>>>   [<c0156866>] ipipe_check_context+0xd6/0xf0
>>> e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
>>>   [<c03e206e>] _spin_lock_irqsave+0x1e/0x80
>>>   [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>>   [<c0254156>] __msi_set_enable+0x46/0x80
>>>   [<c01176f3>] ? mcount+0x1f/0x23
>>>   [<c0254498>] msi_set_mask_bits+0xd8/0xe0
>>>   [<c01176f3>] ? mcount+0x1f/0x23
>>>   [<c0254737>] unmask_msi_irq+0x17/0x30
>>>   [<c01542da>] default_enable+0x1a/0x30
>>>   [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>>   [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>>   [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>>   [<c01021c5>] default_idle+0x45/0x60
>>>   [<c0102180>] default_idle+0x0/0x60
>>>   [<c0103cc7>] common_interrupt+0x2f/0x54
>>>   [<c0102180>] default_idle+0x0/0x60
>>>   [<c01500d8>] cgroup_file_write+0x118/0x140
>>>   [<c01021c5>] default_idle+0x45/0x60
>>>   [<c0101b76>] cpu_idle+0x86/0x140
>>>   [<c03dd7fd>] start_secondary+0x16d/0x210
>>>   [<c03d3a88>] initialize_secondary+0x8/0x20
>>>   =======================
>>> I-pipe tracer log (100 points):
>>>   |  +*func                    0 ipipe_trace_panic_freeze+0x9  
>>
>> I see no option aside of ironing the inner code that reads/writes the PCI
>> config, so here is an ugly yet possible solution for x86, that might work
>> (totally untested):
> 
> Very ugly. There is potentially some heavy code under this lock.

No, actually, the worst-case code you could have is as usual x86-based, i.e.
BIOS calls for raw pci_read/writes. Since we cannot cache PCI config values,
there is no escape.

> Wouldn't it be better to switch to soft-disabling directly, also given
> that not all devices will support that method anyway?
> 

Then you would have to teach all your clients on top of the I-pipe about this.
Basically, the pipeline does not care that much here because soft disabling is
already in place: this is a client issue.

> Jan
> 


-- 
Philippe.

Re: [Adeos-main] [RTnet-users] e1000 & MSI

[Philippe Gerum]

Philippe Gerum wrote:
> Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> bernhard@domain.hid wrote:
>>>>>> Found it. Could you give this patch a try and report the result?
>>>>>>
>>>>>> http://permalink.gmane.org/gmane.linux.kernel/682362
>>>> Applied and tested, no luck...
>>>>
>>>> I-pipe: Detected illicit call from domain 'RTAI'
>>>>          into a service reserved for domain 'Linux' and below.
>>>> Pid: 0, comm: swapper Not tainted 2.6.26.2-FuCS #1
>>>>   [<c0156866>] ipipe_check_context+0xd6/0xf0
>>>> e1000: rteth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
>>>>   [<c03e206e>] _spin_lock_irqsave+0x1e/0x80
>>>>   [<c024a7a6>] pci_bus_read_config_word+0x36/0x80
>>>>   [<c0254156>] __msi_set_enable+0x46/0x80
>>>>   [<c01176f3>] ? mcount+0x1f/0x23
>>>>   [<c0254498>] msi_set_mask_bits+0xd8/0xe0
>>>>   [<c01176f3>] ? mcount+0x1f/0x23
>>>>   [<c0254737>] unmask_msi_irq+0x17/0x30
>>>>   [<c01542da>] default_enable+0x1a/0x30
>>>>   [<f892f1ee>] rt_enable_irq+0xe/0x10 [rtai_hal]
>>>>   [<f8dabd99>] ? xnintr_irq_handler+0x149/0x1f0 [rtai_rtdm]
>>>>   [<f893164b>] rtai_hirq_dispatcher+0xfb/0x430 [rtai_hal]
>>>>   [<c01021c5>] default_idle+0x45/0x60
>>>>   [<c0102180>] default_idle+0x0/0x60
>>>>   [<c0103cc7>] common_interrupt+0x2f/0x54
>>>>   [<c0102180>] default_idle+0x0/0x60
>>>>   [<c01500d8>] cgroup_file_write+0x118/0x140
>>>>   [<c01021c5>] default_idle+0x45/0x60
>>>>   [<c0101b76>] cpu_idle+0x86/0x140
>>>>   [<c03dd7fd>] start_secondary+0x16d/0x210
>>>>   [<c03d3a88>] initialize_secondary+0x8/0x20
>>>>   =======================
>>>> I-pipe tracer log (100 points):
>>>>   |  +*func                    0 ipipe_trace_panic_freeze+0x9  
>>> I see no option aside of ironing the inner code that reads/writes the PCI
>>> config, so here is an ugly yet possible solution for x86, that might work
>>> (totally untested):
>> Very ugly. There is potentially some heavy code under this lock.
> 
> No, actually, the worst-case code you could have is as usual x86-based, i.e.
> BIOS calls for raw pci_read/writes. Since we cannot cache PCI config values,
> there is no escape.
> 
>> Wouldn't it be better to switch to soft-disabling directly, also given
>> that not all devices will support that method anyway?
>>
> 
> Then you would have to teach all your clients on top of the I-pipe about this.
> Basically, the pipeline does not care that much here because soft disabling is
> already in place: this is a client issue.
> 

Said differently: soft disabling may be obtained using ipipe_lock_irq(), instead
of poking into the PCI config, but this interface is a per-domain one, so
clients need to be aware of the "depth" of their action when calling it, so that
they apply the right masking to the right domain. This is why I'm saying that
this is a client issue, above all, because we need to think about it. Granted,
we could emulate hard stalling by locking the interrupt at the highest I-pipe stage.

Aside of this, yes, it would be perfectly sensible to use soft disabling using
that interface. But well, let's wait for upstream to eventually pick an option
for handling interrupts, that will last at least for a couple of releases per
their standard, before jumping into such a change.

-- 
Philippe.