[Xenomai-core] Registering MSI interrupt with Xenomai fails

[Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

We have a driver that operates on a PCIe card. The card has IRQ17. If we use
it like that (IO-APIC-fasteoi), interrupt registration using
rtdm_irq_request works correctly. (We also use rtdm_irq_enable afterwards,
but it seems that the request already enables the interrupt.)

However, if we redefine our interrupt as MSI using pci_enable_msi(),
rtdm_irq_request freezes the machine. (After pci_enable_msi, the new
interrupt vector is 218 and /proc/interrupts correctly reports
PCI-MSI-edge.)

We have another MSI-enabled card in the system (network card controlled by
Linux) and this one works correctly. So I suspect that the Ipipe is clear
and the bug must reside in Xenomai.

I've been adding printk instrumentation throughout the Ipipe, Xenomai and
RTDM, but the problem is that possibly the contents of the kernel log do not
make it to the terminal upon the freeze (no oops, no panic). Is there any
way of efficiently debugging this ?

Linux 2.6.23
Adeos 1.10-11
Xenomai SVN Rev. 3146

Thanks,


Jeroen.

[-- Attachment #2: Type: text/html, Size: 1262 bytes --]

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> We have a driver that operates on a PCIe card. The card has IRQ17. If we
> use it like that (IO-APIC-fasteoi), interrupt registration using
> rtdm_irq_request works correctly. (We also use rtdm_irq_enable
> afterwards, but it seems that the request already enables the interrupt.)
>  

Yes, it does.

> However, if we redefine our interrupt as MSI using pci_enable_msi(),
> rtdm_irq_request freezes the machine. (After pci_enable_msi, the new
> interrupt vector is 218 and /proc/interrupts correctly reports
> PCI-MSI-edge.)
>  
> We have another MSI-enabled card in the system (network card controlled
> by Linux) and this one works correctly. So I suspect that the Ipipe is
> clear and the bug must reside in Xenomai.
>  
> I've been adding printk instrumentation throughout the Ipipe, Xenomai
> and RTDM, but the problem is that possibly the contents of the kernel
> log do not make it to the terminal upon the freeze (no oops, no panic).
> Is there any way of efficiently debugging this ?
>

First, you may want to try commenting out the call to
xnintr_irq_enable() from rtdm_irq_request() - and your own call if any -
to check whether the system remains functional without enabling the IRQ
line. If so, then this may be an acknowledge issue.

-- 
Philippe.

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

>
>
> Now looking into xnintr_attach().
>

In xnintr_attach(), the crash occurs in xnarch_set_irq_affinity(). If the
call is removed, the driver works as expected (gaining another 3 usec of
latency). Now investigating ipipe_set_irq_affinity.

Jeroen.


>
>
> Jeroen.
>

[-- Attachment #2: Type: text/html, Size: 663 bytes --]

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Philippe Gerum]

Jeroen Van den Keybus wrote:
>      
>     Now looking into xnintr_attach().
> 
>  
> In xnintr_attach(), the crash occurs in xnarch_set_irq_affinity(). If
> the call is removed, the driver works as expected (gaining another 3
> usec of latency). Now investigating ipipe_set_irq_affinity.
>  

arch_setup_msi_irq() creates an IRQ on-the-fly from the current
descriptor which is being converted to an MSI interrupt using
pci_msi_enable(). From that point, the I-pipe might have an obsolete
view of the interrupt map. I suspect an I-pipe issue here.

-- 
Philippe.

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

>
>
> arch_setup_msi_irq() creates an IRQ on-the-fly from the current
> descriptor which is being converted to an MSI interrupt using
> pci_msi_enable(). From that point, the I-pipe might have an obsolete
> view of the interrupt map. I suspect an I-pipe issue here.



I think the I-pipe is alright. It only cares for the actual interrupt
numbers and irq_desc[] should be current with these numbers upon the
ipipe_virtualize_irq call, which occurs only after enabling MSI, right ?

Currently it looks like every PCI config space access instruction in
read_msi_msg() (used to perform set_msi_irq_affinity) freezes the machine. I
have absolutely no clue yet why this happens.

Jeroen.


--
> Philippe.
>

[-- Attachment #2: Type: text/html, Size: 1046 bytes --]

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Philippe Gerum]

Jeroen Van den Keybus wrote:
> 
>     arch_setup_msi_irq() creates an IRQ on-the-fly from the current
>     descriptor which is being converted to an MSI interrupt using
>     pci_msi_enable(). From that point, the I-pipe might have an obsolete
>     view of the interrupt map. I suspect an I-pipe issue here.
> 
> 
> 
> I think the I-pipe is alright. It only cares for the actual interrupt
> numbers and irq_desc[] should be current with these numbers upon the
> ipipe_virtualize_irq call, which occurs only after enabling MSI, right ?
> 

The I-pipe virtualizes all IOAPIC and ISA interrupts upon startup. Then,
any code calling pci_msi_enable() would end up allocating a new MSI
interrupt vector.

> Currently it looks like every PCI config space access instruction in
> read_msi_msg() (used to perform set_msi_irq_affinity) freezes the
> machine. I have absolutely no clue yet why this happens.
> 

Wild trivial guess, is the irq parameter the expected one, since the
rest depends on it?

> Jeroen.
> 
> 
>     --
>     Philippe.
> 
> 


-- 
Philippe.

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

>
> The I-pipe virtualizes all IOAPIC and ISA interrupts upon startup. Then,
> any code calling pci_msi_enable() would end up allocating a new MSI
> interrupt vector.


I see. So in fact, at least for the Linux domain, which indeed registers all
interrupts upon initialization, every newly created MSI vector should be
revirtualized.



> > Currently it looks like every PCI config space access instruction in
> > read_msi_msg() (used to perform set_msi_irq_affinity) freezes the
> > machine. I have absolutely no clue yet why this happens.
> >
>
> Wild trivial guess, is the irq parameter the expected one, since the
> rest depends on it?


You mean the one passed into xnarch_set_irq_affinity ? Yes, it is
consistently 219 and the mask is 0x3. In set_msi_irq_affinity(), the
associated vector is calculated as 225. I'm not sure the latter one is
correct, but at least it shouldn't freeze the machine upon a PCI config
read.

Could there be any reason why e.g. a pci_config_read_dword() call would in
Xenomai context (because when still in the module probe code, I can perform
the call correctly - in both cases I checked the dev->bus->ops->read/write
pointers and they are identical) ?


Jeroen.

[-- Attachment #2: Type: text/html, Size: 1654 bytes --]

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

>
> Could there be any reason why e.g. a pci_config_read_dword() call would
> fail in Xenomai context (because when still in the module probe code, I
> can perform the call correctly - in both cases I checked the
> dev->bus->ops->read/write pointers and they are identical) ?
>

Today I moved the Xenomai ISR registration (rtdm_request_irq)) from the RTDM
open method (rtdm_dev_open) to the module probe code (dev_probe). Then the
registration works, but not always. Upon frequent insmod- and rmmodding, I
still can make the machine freeze.

I'm now more or less lost and open to any ideas...


Jeroen.


BTW: In the process of removing all instrumentation, I inadvertently
upgraded to SVN head (3149), after which no Xenomai process could be run
anymore (Binding failed. Bad address.). I solved that temporarily by
reverting to my original 3146.


>
>
> Jeroen.
>

[-- Attachment #2: Type: text/html, Size: 1425 bytes --]

Re: [Xenomai-core] Registering MSI interrupt with Xenomai fails

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

I must suspend the search for this problem for a while. The specific
hardware (computer) is unavailable to me for about a week.

Jeroen.




>  Could there be any reason why e.g. a pci_config_read_dword() call would
> > fail in Xenomai context (because when still in the module probe code, I
> > can perform the call correctly - in both cases I checked the
> > dev->bus->ops->read/write pointers and they are identical) ?
> >
>
> Today I moved the Xenomai ISR registration (rtdm_request_irq)) from the
> RTDM open method (rtdm_dev_open) to the module probe code (dev_probe). Then
> the registration works, but not always. Upon frequent insmod- and rmmodding,
> I still can make the machine freeze.
>
> I'm now more or less lost and open to any ideas..
>

[-- Attachment #2: Type: text/html, Size: 1258 bytes --]