INTR-REMAP error with UDD driver

INTR-REMAP error with UDD driver

[Jeff Webb]

I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program.  This pair of programs works as intended on one machine.  When I tried running the same programs on another machine, I got this error when the first interrupt occurred:

kernel: [   75.122305] DMAR: DRHD: handling fault status reg 2
kernel: [   75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request

I would guess that the [f0:1f.0] in the message is a PCI bus address, but I don't see this via lspci.  The closest match is:

00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC Controller [8086:1c4e] (rev 05)

After the first attempt, I see no more of these messages until I reboot, but the UDD driver never receives an interrupt in any case.

I tried moving my PCI card to five different slots, and finally found one where everything work properly on the second machine.  I can at continue to make progress using the working slot, but I would really like to resolve the issue with the other slots.  Does anyone have any idea what to try next?

I am using these sources:

linux-4.14.89.tar
ipipe-core-4.14.89-x86-2.patch
xenomai-3.0.8.tar

I have attached the boot log, config, and cpu information.

Thanks,

-Jeff Webb
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cpuinfo.txt
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/b0ab681a/attachment.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boot.log
Type: text/x-log
Size: 62797 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/b0ab681a/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-4.14.89-xenomai-3.0.8
Type: application/x-troff-man
Size: 213256 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/b0ab681a/attachment.man>

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 08.03.19 02:35, Jeff Webb via Xenomai wrote:
> I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program.  This pair of programs works as intended on one machine.  When I tried running the same programs on another machine, I got this error when the first interrupt occurred:
> 
> kernel: [   75.122305] DMAR: DRHD: handling fault status reg 2
> kernel: [   75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request
> 

This means the device is under IOMMU interrupt remapping regime, but it was 
programmed to use a standard MSI/MSI-X request, rather than requesting that 
parameters (address/data tuple) from the Linux interrupt management layer that 
will assign a remapping slot and hand out the right values. Now you interrupt is 
stuck in the filter.

Before suggesting the workaround/hack to disable interrupt remapping: How did 
you program MSI/-X? Via proper Linux pci_* helpers?

Jan

> I would guess that the [f0:1f.0] in the message is a PCI bus address, but I don't see this via lspci.  The closest match is:
> 
> 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC Controller [8086:1c4e] (rev 05)
> 
> After the first attempt, I see no more of these messages until I reboot, but the UDD driver never receives an interrupt in any case.
> 
> I tried moving my PCI card to five different slots, and finally found one where everything work properly on the second machine.  I can at continue to make progress using the working slot, but I would really like to resolve the issue with the other slots.  Does anyone have any idea what to try next?
> 
> I am using these sources:
> 
> linux-4.14.89.tar
> ipipe-core-4.14.89-x86-2.patch
> xenomai-3.0.8.tar
> 
> I have attached the boot log, config, and cpu information.
> 
> Thanks,
> 
> -Jeff Webb

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

On Friday, March 8, 2019 1:14 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 08.03.19 02:35, Jeff Webb via Xenomai wrote:
>
> > I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program. This pair of programs works as intended on one machine. When I tried running the same programs on another machine, I got this error when the first interrupt occurred:
> > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2
> > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request
>
> This means the device is under IOMMU interrupt remapping regime, but it was
> programmed to use a standard MSI/MSI-X request, rather than requesting that
> parameters (address/data tuple) from the Linux interrupt management layer that
> will assign a remapping slot and hand out the right values. Now you interrupt is
> stuck in the filter.
>
> Before suggesting the workaround/hack to disable interrupt remapping: How did
> you program MSI/-X? Via proper Linux pci_* helpers?

The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A.  It's confusing to me that this error is associated with MSI.  Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots.  There is only one slot where things work as expected.

Thanks for the response,

-Jeff

>
> Jan
>
> > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I don't see this via lspci. The closest match is:
> > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC Controller [8086:1c4e] (rev 05)
> > After the first attempt, I see no more of these messages until I reboot, but the UDD driver never receives an interrupt in any case.
> > I tried moving my PCI card to five different slots, and finally found one where everything work properly on the second machine. I can at continue to make progress using the working slot, but I would really like to resolve the issue with the other slots. Does anyone have any idea what to try next?
> > I am using these sources:
> > linux-4.14.89.tar
> > ipipe-core-4.14.89-x86-2.patch
> > xenomai-3.0.8.tar
> > I have attached the boot log, config, and cpu information.
> > Thanks,
> > -Jeff Webb
>
> --
>
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 08.03.19 15:07, Jeff Webb wrote:
> On Friday, March 8, 2019 1:14 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
>> On 08.03.19 02:35, Jeff Webb via Xenomai wrote:
>>
>>> I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program. This pair of programs works as intended on one machine. When I tried running the same programs on another machine, I got this error when the first interrupt occurred:
>>> kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2
>>> kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request
>>
>> This means the device is under IOMMU interrupt remapping regime, but it was
>> programmed to use a standard MSI/MSI-X request, rather than requesting that
>> parameters (address/data tuple) from the Linux interrupt management layer that
>> will assign a remapping slot and hand out the right values. Now you interrupt is
>> stuck in the filter.
>>
>> Before suggesting the workaround/hack to disable interrupt remapping: How did
>> you program MSI/-X? Via proper Linux pci_* helpers?
> 
> The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A.  It's confusing to me that this error is associated with MSI.  Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots.  There is only one slot where things work as expected.
> 

Ok, then we may have an issue with how we register legacy INTx at the IOAPIC 
when interrupt remapping is on. Basically, the IOAPIC sends MSI-like messages as 
well that are subject to remapping. So its registers need to be programmed with 
the right translatable values, just like the MSI registers of a device. Linux 
does that, it's in charge of the IOMMU. I don't recall how we (Xenomai/I-pipe) 
fill the IOAPIC if a line was not used before under Linux. Possibly, there is 
the wrong Linux function triggered. But maybe it's also something different.

I assume you tested that your card would work find if it used only regular Linux 
driver services, no RTDM, right? That would be good in order to exclude a 
general platform issue, maybe a BIOS (ACPI) bug in describing which PCI slot is 
under which IOMMU regime.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

On Friday, March 8, 2019 9:00 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 08.03.19 15:07, Jeff Webb wrote:
>
> > On Friday, March 8, 2019 1:14 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
> >
> > > On 08.03.19 02:35, Jeff Webb via Xenomai wrote:
> > >
> > > > I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program. This pair of programs works as intended on one machine. When I tried running the same programs on another machine, I got this error when the first interrupt occurred:
> > > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2
> > > > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request
> > >
> > > This means the device is under IOMMU interrupt remapping regime, but it was
> > > programmed to use a standard MSI/MSI-X request, rather than requesting that
> > > parameters (address/data tuple) from the Linux interrupt management layer that
> > > will assign a remapping slot and hand out the right values. Now you interrupt is
> > > stuck in the filter.
> > > Before suggesting the workaround/hack to disable interrupt remapping: How did
> > > you program MSI/-X? Via proper Linux pci_* helpers?
> >
> > The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A. It's confusing to me that this error is associated with MSI. Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots. There is only one slot where things work as expected.
>
> Ok, then we may have an issue with how we register legacy INTx at the IOAPIC
> when interrupt remapping is on. Basically, the IOAPIC sends MSI-like messages as
> well that are subject to remapping. So its registers need to be programmed with
> the right translatable values, just like the MSI registers of a device. Linux
> does that, it's in charge of the IOMMU. I don't recall how we (Xenomai/I-pipe)
> fill the IOAPIC if a line was not used before under Linux. Possibly, there is
> the wrong Linux function triggered. But maybe it's also something different.
>
> I assume you tested that your card would work find if it used only regular Linux
> driver services, no RTDM, right? That would be good in order to exclude a
> general platform issue, maybe a BIOS (ACPI) bug in describing which PCI slot is
> under which IOMMU regime.

I just wrote a similar UIO driver and user-space program.  The UIO version works when the card is in any slot.  The UDD version only works when the card is in one slot (out of 5).  The UDD version does not work in the "bad" slots even if the UIO version is run first.  All of the UIO testing is with the same kernel as the UDD testing.

Thanks,

-Jeff

Re: INTR-REMAP error with UDD driver

[Jim Elliott]

Hey guys, I was wondering if there is an update on the INTR-REMAP UDD driver issue.  We have not had any troubleshooting breakthroughs on our end.


Best Regards,

Jim Elliott

________________________________
From: Xenomai <xenomai-bounces@xenomai.org> on behalf of Jeff Webb via Xenomai <xenomai@xenomai.org>
Sent: Friday, March 8, 2019 8:07 AM
To: Jan Kiszka
Cc: xenomai@xenomai.org
Subject: Re: INTR-REMAP error with UDD driver

On Friday, March 8, 2019 1:14 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 08.03.19 02:35, Jeff Webb via Xenomai wrote:
>
> > I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program. This pair of programs works as intended on one machine. When I tried running the same programs on another machine, I got this error when the first interrupt occurred:
> > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2
> > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request
>
> This means the device is under IOMMU interrupt remapping regime, but it was
> programmed to use a standard MSI/MSI-X request, rather than requesting that
> parameters (address/data tuple) from the Linux interrupt management layer that
> will assign a remapping slot and hand out the right values. Now you interrupt is
> stuck in the filter.
>
> Before suggesting the workaround/hack to disable interrupt remapping: How did
> you program MSI/-X? Via proper Linux pci_* helpers?

The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A.  It's confusing to me that this error is associated with MSI.  Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots.  There is only one slot where things work as expected.

Thanks for the response,

-Jeff

>
> Jan
>
> > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I don't see this via lspci. The closest match is:
> > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC Controller [8086:1c4e] (rev 05)
> > After the first attempt, I see no more of these messages until I reboot, but the UDD driver never receives an interrupt in any case.
> > I tried moving my PCI card to five different slots, and finally found one where everything work properly on the second machine. I can at continue to make progress using the working slot, but I would really like to resolve the issue with the other slots. Does anyone have any idea what to try next?
> > I am using these sources:
> > linux-4.14.89.tar
> > ipipe-core-4.14.89-x86-2.patch
> > xenomai-3.0.8.tar
> > I have attached the boot log, config, and cpu information.
> > Thanks,
> > -Jeff Webb
>
> --
>
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux




________________________________

CONFIDENTIALITY NOTICE: The contents of this e-mail message may be privileged, confidential, proprietary, Controlled Unclassified Information (CUI) or otherwise protected against disclosure or dissemination. This e-mail message and any files transmitted with it are intended solely for the use of the individual/individuals or entity/entities to whom they are addressed. If you are not the intended recipient of this message, any dissemination, distribution, printing or copying is strictly prohibited. If you have received this message in error, please e-mail or call the sender (jim.elliott@nta-inc.net, ) and destroy all copies.

Re: INTR-REMAP error with UDD driver

[Per Oberg]

----- Den 28 mar 2019, på kl 18:10, xenomai xenomai@xenomai.org skrev:

> Hey guys, I was wondering if there is an update on the INTR-REMAP UDD driver
> issue. We have not had any troubleshooting breakthroughs on our end.

> Best Regards,

> Jim Elliott

> ________________________________
> From: Xenomai <xenomai-bounces@xenomai.org> on behalf of Jeff Webb via Xenomai
> <xenomai@xenomai.org>
> Sent: Friday, March 8, 2019 8:07 AM
> To: Jan Kiszka
> Cc: xenomai@xenomai.org
> Subject: Re: INTR-REMAP error with UDD driver

> On Friday, March 8, 2019 1:14 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:

> > On 08.03.19 02:35, Jeff Webb via Xenomai wrote:

>> > I have implemented a simple UDD mini-driver (interrupt handler) and an
>> > associated Xenomai userspace test program. This pair of programs works as
>> > intended on one machine. When I tried running the same programs on another
> > > machine, I got this error when the first interrupt occurred:
> > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2
>> > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index
> > > ee00 [fault reason 37] Blocked a compatibility format interrupt request

> > This means the device is under IOMMU interrupt remapping regime, but it was
> > programmed to use a standard MSI/MSI-X request, rather than requesting that
> > parameters (address/data tuple) from the Linux interrupt management layer that
> > will assign a remapping slot and hand out the right values. Now you interrupt is
> > stuck in the filter.

> > Before suggesting the workaround/hack to disable interrupt remapping: How did
> > you program MSI/-X? Via proper Linux pci_* helpers?

> The PCI card I am working with does not implement MSI interrupts, but just uses
> the legacy interrupt pin A. It's confusing to me that this error is associated
> with MSI. Since this behavior is slot-dependent, I figured at first that the
> error is from another device on the same interrupt line, but I get the same
> error on several different slots. There is only one slot where things work as
> expected.

When I read this I realize I have a similar or the same issue. When I moved from 4.9.38 to 4.9.90 the xenomai_peak_pci driver stopped getting interrupts and I got a note about io-remapping and a blocked compability interrupt. It worked on another motherboard. 

The kernel-configs were not equal, and I haven't had time to troubleshoot this. When I switched to the new Peak driver it worked so I focused on other things instead. 

I may be able to contribute in the future.


Best regards
Per Öberg 


> Thanks for the response,

> -Jeff


> > Jan

>> > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I
> > > don't see this via lspci. The closest match is:
>> > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC Controller
> > > [8086:1c4e] (rev 05)
>> > After the first attempt, I see no more of these messages until I reboot, but the
> > > UDD driver never receives an interrupt in any case.
>> > I tried moving my PCI card to five different slots, and finally found one where
>> > everything work properly on the second machine. I can at continue to make
>> > progress using the working slot, but I would really like to resolve the issue
> > > with the other slots. Does anyone have any idea what to try next?
> > > I am using these sources:
> > > linux-4.14.89.tar
> > > ipipe-core-4.14.89-x86-2.patch
> > > xenomai-3.0.8.tar
> > > I have attached the boot log, config, and cpu information.
> > > Thanks,
> > > -Jeff Webb

> > --

> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> > Corporate Competence Center Embedded Linux

> ________________________________

> CONFIDENTIALITY NOTICE: The contents of this e-mail message may be privileged,
> confidential, proprietary, Controlled Unclassified Information (CUI) or
> otherwise protected against disclosure or dissemination. This e-mail message
> and any files transmitted with it are intended solely for the use of the
> individual/individuals or entity/entities to whom they are addressed. If you
> are not the intended recipient of this message, any dissemination,
> distribution, printing or copying is strictly prohibited. If you have received
> this message in error, please e-mail or call the sender
> (jim.elliott@nta-inc.net, ) and destroy all copies.

RE: INTR-REMAP error with UDD driver

[Jeff Webb]

I would like to revive this thread from several months ago:

https://xenomai.org/pipermail/xenomai/2019-March/040498.html

The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver.  The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.

Please review the original thread for more details about the specific error.

Here are a few more tidbits I have gathered:

- The UDD driver / userspace code works fine on the other hardware

- The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.

- With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.

- An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.

In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar.  Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed.  This prompted me to try other linux / ipipe / xenomai combinations.  These are my findings:

Interrupts work:
xenomai-2.6.5   ipipe-core-3.18.20-x86-7.patch  (2016-07-05)
xenomai-3.0.9+  ipipe-core-3.18.20-x86-7.patch  (2016-07-05)
xenomai-3.0.9+  ipipe-core-4.1.18-x86-9.patch   (2017-05-25)

INTR-REMAP error:
xenomai-3.0.9+  ipipe-core-4.4.43-x86-6.patch   (2017-02-25)
xenomai-3.0.9+  ipipe-core-4.4.43-x86-7.patch   (2017-05-25)
xenomai-3.0.9+  ipipe-core-4.4.43-x86-8.patch   (2017-06-14)
xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
xenomai-3.0.9+  ipipe-core-4.9.38-x86-4.patch   (2017-10-03)
xenomai-3.0.9   ipipe-core-4.14.132-x86-6.patch (2019-07-03)

The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.

Hopefully this additional information can shed a little light on the matter.

Thanks in advance for any input you can provide,

-Jeff Webb

Re: INTR-REMAP error with UDD driver

[Per Oberg]

> I would like to revive this thread from several months ago:

> https://xenomai.org/pipermail/xenomai/2019-March/040498.html

> The issue is that on some hardware (a specific rack-mount PC with a PICMG
> daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP
> error when trying to receive legacy (not MSI) interrupts from a custom
> FPGA-based PCI card using a UDD driver. The card did work properly in one out
> of the five PCI slots on that machine, but UDD interrupts did not work in the
> other four slots.

This looks very close to my issue. This was under an experimenting phase of the project, so unfortunately I cannot re-test anything because I don't have the hardware. If I recall correctly the hardware involved where 

PC
---------------
One of these worked, the other did not (sorry, this is all I can remember)
* Advantech ARK-3500, embedded PC [1] (hard to find a  good description)
* Advantech AIMB-585, Motherboard [2] 

This definitely worked
* HP Z620 Workstation 


PCI/PCIe
* PCAN-PCI Express Four Channel, not manufactured any more (Closest is 2 Channel IPEH-003027)

[1] https://advdownload.advantech.com/productfile/PIS/ARK-3500/Product%20-%20Datasheet/ARK-3500_DS(11.16.16)20161116101733.pdf
[2] https://advdownload.advantech.com/productfile/PIS/AIMB-585/Product%20-%20Datasheet/AIMB-585_DS(12.28.17)20171228141058.pdf

> Please review the original thread for more details about the specific error.

> Here are a few more tidbits I have gathered:

> - The UDD driver / userspace code works fine on the other hardware

> - The UDD driver / userspace code works fine in one PCI slot out of five on this
> hardware.

> - With another backplane model, but same processor card, the problem occurs in
> all four of the PCI slots.

> - An almost identical pure-linux UIO version of the driver / userspace code
> works in all the cases I tested, even when the UDD version fails, and even with
> the same xenomai-patched kernel used for UDD testing.

This also fits my memory of the issue I had. 

> In one of the previous posts in this thread a few months ago, Per Öberg
> mentioned experiencing something similar. Based on the information that was
> shared, I tried my code with linux version 4.9.38, but it still failed. This
> prompted me to try other linux / ipipe / xenomai combinations. These are my
> findings:

> Interrupts work:
> xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)

> INTR-REMAP error:
> xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
> xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
> xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
> xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
> xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)

> The Xenomai 2.6.5 version of course does not use UDD, but uses the old
> pthread_intr_* userspace functions.

> Hopefully this additional information can shed a little light on the matter.

> Thanks in advance for any input you can provide,

> -Jeff Webb

Per Öberg 

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
> I would like to revive this thread from several months ago:
> 
> https://xenomai.org/pipermail/xenomai/2019-March/040498.html
> 
> The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver.  The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
> 
> Please review the original thread for more details about the specific error.
> 
> Here are a few more tidbits I have gathered:
> 
> - The UDD driver / userspace code works fine on the other hardware
> 
> - The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
> 
> - With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
> 
> - An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
> 
> In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar.  Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed.  This prompted me to try other linux / ipipe / xenomai combinations.  These are my findings:
> 
> Interrupts work:
> xenomai-2.6.5   ipipe-core-3.18.20-x86-7.patch  (2016-07-05)
> xenomai-3.0.9+  ipipe-core-3.18.20-x86-7.patch  (2016-07-05)
> xenomai-3.0.9+  ipipe-core-4.1.18-x86-9.patch   (2017-05-25)
> 
> INTR-REMAP error:
> xenomai-3.0.9+  ipipe-core-4.4.43-x86-6.patch   (2017-02-25)
> xenomai-3.0.9+  ipipe-core-4.4.43-x86-7.patch   (2017-05-25)
> xenomai-3.0.9+  ipipe-core-4.4.43-x86-8.patch   (2017-06-14)
> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
> xenomai-3.0.9+  ipipe-core-4.9.38-x86-4.patch   (2017-10-03)
> xenomai-3.0.9   ipipe-core-4.14.132-x86-6.patch (2019-07-03)
> 
> The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
> 
> Hopefully this additional information can shed a little light on the matter.
> 

This sounds like some RT interrupt enabling issue related to the IOAPIC 
in the x86 I-pipe patch. Please also test 4.19.

Are you using UDD_IRQ_CUSTOM or do you leave the interrupt registration 
to the UDD core?

And please share your kernel config.

BTW, interrupt remapping issues can be worked around by disabling the 
interrupt remapping feature (e.g. "intremap=off"). But that does not 
solve the unterlying issue, of course.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, November 14, 2019 1:50 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
> > I would like to revive this thread from several months ago:
> > https://xenomai.org/pipermail/xenomai/2019-March/040498.html
> > The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
> > Please review the original thread for more details about the specific error.
> > Here are a few more tidbits I have gathered:
> >
> > -   The UDD driver / userspace code works fine on the other hardware
> >
> > -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
> >
> > -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
> >
> > -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
> >
> >
> > In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
> > Interrupts work:
> > xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
> > INTR-REMAP error:
> > xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
> > xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
> > xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
> > xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
> > xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
> > xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
> > The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
> > Hopefully this additional information can shed a little light on the matter.
>
> This sounds like some RT interrupt enabling issue related to the IOAPIC
> in the x86 I-pipe patch. Please also test 4.19.

Ok, I will do this.

> Are you using UDD_IRQ_CUSTOM or do you leave the interrupt registration
> to the UDD core?

I just tell UDD the IRQ number and let it register the interrupt.

> And please share your kernel config.

I attached one to my original post earlier this year -- you should be able to download it from the link in the mailing list archive.  Let me know if you need something different.  I started with the standard Ubuntu desktop kernel config and tweaked options from there, so there is a lot of stuff enabled, obviously.

> BTW, interrupt remapping issues can be worked around by disabling the
> interrupt remapping feature (e.g. "intremap=off"). But that does not
> solve the unterlying issue, of course.

I can't remember if I tried this or not.  I will give it a go.  Obviously, it would be good to get this fixed in the patch, though.

Thank you (and Per Öberg) for your help.

-Jeff

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 14.11.19 14:16, Jeff Webb wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, November 14, 2019 1:50 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
>>> I would like to revive this thread from several months ago:
>>> https://xenomai.org/pipermail/xenomai/2019-March/040498.html
>>> The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
>>> Please review the original thread for more details about the specific error.
>>> Here are a few more tidbits I have gathered:
>>>
>>> -   The UDD driver / userspace code works fine on the other hardware
>>>
>>> -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
>>>
>>> -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
>>>
>>> -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
>>>
>>>
>>> In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
>>> Interrupts work:
>>> xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>> xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>> xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
>>> INTR-REMAP error:
>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
>>> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
>>> xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
>>> xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
>>> The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
>>> Hopefully this additional information can shed a little light on the matter.
>>
>> This sounds like some RT interrupt enabling issue related to the IOAPIC
>> in the x86 I-pipe patch. Please also test 4.19.
> 
> Ok, I will do this.
> 
>> Are you using UDD_IRQ_CUSTOM or do you leave the interrupt registration
>> to the UDD core?
> 
> I just tell UDD the IRQ number and let it register the interrupt.
> 
>> And please share your kernel config.
> 
> I attached one to my original post earlier this year -- you should be able to download it from the link in the mailing list archive.  Let me know if you need something different.  I started with the standard Ubuntu desktop kernel config and tweaked options from there, so there is a lot of stuff enabled, obviously.
> 
>> BTW, interrupt remapping issues can be worked around by disabling the
>> interrupt remapping feature (e.g. "intremap=off"). But that does not
>> solve the unterlying issue, of course.
> 
> I can't remember if I tried this or not.  I will give it a go.  Obviously, it would be good to get this fixed in the patch, though.
> 

I've reproduced some problem with INTx/IOAPIC and intremap=on on 4.4 in 
KVM. When we are lucky, it's the same as yours. Will debug that tomorrow.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, November 14, 2019 3:41 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 14.11.19 14:16, Jeff Webb wrote:
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Thursday, November 14, 2019 1:50 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
> >
> > > On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
> > >
> > > > I would like to revive this thread from several months ago:
> > > > https://xenomai.org/pipermail/xenomai/2019-March/040498.html
> > > > The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
> > > > Please review the original thread for more details about the specific error.
> > > > Here are a few more tidbits I have gathered:
> > > >
> > > > -   The UDD driver / userspace code works fine on the other hardware
> > > >
> > > > -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
> > > >
> > > > -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
> > > >
> > > > -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
> > > >
> > > >
> > > > In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
> > > > Interrupts work:
> > > > xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > > > xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > > > xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
> > > > INTR-REMAP error:
> > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
> > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
> > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
> > > > xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
> > > > xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
> > > > xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
> > > > The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
> > > > Hopefully this additional information can shed a little light on the matter.
> > >
> > > This sounds like some RT interrupt enabling issue related to the IOAPIC
> > > in the x86 I-pipe patch. Please also test 4.19.
> >
> > Ok, I will do this.

Today I had a chance to make a build based on the ipipe-core-4.19.75-x86-7.patch with xenomai-3.1-rc3.  I haven't had a chance to test it thoroughly, but I think it works.  I am sorry I didn't try this earlier, and thanks for the reminder to do so.  Since a 4.19 patch wasn't available back in early March when I first had the trouble, I got fixated on the fact that things worked with very old kernels and forgot to try 4.19 when I started looking into this again.  I will continue to test and let you know if I find any issues.

> > > Are you using UDD_IRQ_CUSTOM or do you leave the interrupt registration
> > > to the UDD core?
> >
> > I just tell UDD the IRQ number and let it register the interrupt.
> >
> > > And please share your kernel config.
> >
> > I attached one to my original post earlier this year -- you should be able to download it from the link in the mailing list archive. Let me know if you need something different. I started with the standard Ubuntu desktop kernel config and tweaked options from there, so there is a lot of stuff enabled, obviously.
> >
> > > BTW, interrupt remapping issues can be worked around by disabling the
> > > interrupt remapping feature (e.g. "intremap=off"). But that does not
> > > solve the unterlying issue, of course.
> >
> > I can't remember if I tried this or not. I will give it a go. Obviously, it would be good to get this fixed in the patch, though.
>
> I've reproduced some problem with INTx/IOAPIC and intremap=on on 4.4 in
> KVM. When we are lucky, it's the same as yours. Will debug that tomorrow.

Thank you so much for looking into this, Jan.

-Jeff

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 15.11.19 01:16, Jeff Webb wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, November 14, 2019 3:41 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 14.11.19 14:16, Jeff Webb wrote:
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Thursday, November 14, 2019 1:50 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
>>>
>>>> On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
>>>>
>>>>> I would like to revive this thread from several months ago:
>>>>> https://xenomai.org/pipermail/xenomai/2019-March/040498.html
>>>>> The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
>>>>> Please review the original thread for more details about the specific error.
>>>>> Here are a few more tidbits I have gathered:
>>>>>
>>>>> -   The UDD driver / userspace code works fine on the other hardware
>>>>>
>>>>> -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
>>>>>
>>>>> -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
>>>>>
>>>>> -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
>>>>>
>>>>>
>>>>> In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
>>>>> Interrupts work:
>>>>> xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>> xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>> xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
>>>>> INTR-REMAP error:
>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
>>>>> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
>>>>> xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
>>>>> xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
>>>>> The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
>>>>> Hopefully this additional information can shed a little light on the matter.
>>>>
>>>> This sounds like some RT interrupt enabling issue related to the IOAPIC
>>>> in the x86 I-pipe patch. Please also test 4.19.
>>>
>>> Ok, I will do this.
> 
> Today I had a chance to make a build based on the ipipe-core-4.19.75-x86-7.patch with xenomai-3.1-rc3.  I haven't had a chance to test it thoroughly, but I think it works.  I am sorry I didn't try this earlier, and thanks for the reminder to do so.  Since a 4.19 patch wasn't available back in early March when I first had the trouble, I got fixated on the fact that things worked with very old kernels and forgot to try 4.19 when I started looking into this again.  I will continue to test and let you know if I find any issues.
> 

I'm also not seeing the problems I have with 4.4 + intremap on 4.19, 
despite all debug features on. OK, let's check what we are missing down 
there...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

On Friday, November 15, 2019 7:07 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 15.11.19 01:16, Jeff Webb wrote:
>
> > On Thursday, November 14, 2019 3:41 PM, Jan Kiszka jan.kiszka@siemens.com wrote:
> >
> > > On 14.11.19 14:16, Jeff Webb wrote:
> > >
> > > > On Thursday, November 14, 2019 1:50 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
> > > >
> > > > > On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
> > > > >
> > > > > > I would like to revive this thread from several months ago:
> > > > > > https://xenomai.org/pipermail/xenomai/2019-March/040498.html
> > > > > > The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
> > > > > > Please review the original thread for more details about the specific error.
> > > > > > Here are a few more tidbits I have gathered:
> > > > > >
> > > > > > -   The UDD driver / userspace code works fine on the other hardware
> > > > > >
> > > > > > -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
> > > > > >
> > > > > > -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
> > > > > >
> > > > > > -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
> > > > > >
> > > > > >
> > > > > > In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
> > > > > > Interrupts work:
> > > > > > xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > > > > > xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
> > > > > > xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
> > > > > > INTR-REMAP error:
> > > > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
> > > > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
> > > > > > xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
> > > > > > xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
> > > > > > xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
> > > > > > xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
> > > > > > The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
> > > > > > Hopefully this additional information can shed a little light on the matter.
> > > > >
> > > > > This sounds like some RT interrupt enabling issue related to the IOAPIC
> > > > > in the x86 I-pipe patch. Please also test 4.19.
> > > >
> > > > Ok, I will do this.
> >
> > Today I had a chance to make a build based on the ipipe-core-4.19.75-x86-7.patch with xenomai-3.1-rc3. I haven't had a chance to test it thoroughly, but I think it works. I am sorry I didn't try this earlier, and thanks for the reminder to do so. Since a 4.19 patch wasn't available back in early March when I first had the trouble, I got fixated on the fact that things worked with very old kernels and forgot to try 4.19 when I started looking into this again. I will continue to test and let you know if I find any issues.
>
> I'm also not seeing the problems I have with 4.4 + intremap on 4.19,
> despite all debug features on. OK, let's check what we are missing down
> there...

Just FYI, I did go back and try intremap=off on 4.4.196-xenomai-3.1-rc3-cip38, and it works (as you expected).

Also, I looked at my kernel logs in more detail, and discovered that there are several instances of a kernel BUG message after boot, but then I don't see any more of them later on.  The intremap setting doesn't seem to affect the presence of these messages.  This is the message for 4.4.196-xenomai-3.1-rc3-cip38:

[   53.697717] kernel BUG at kernel/auditsc.c:1502!
[   53.698132] invalid opcode: 0000 [#1] SMP

There are similar messages with 4.19.75-xenomai-3.1-rc3:

[   65.546333] kernel BUG at kernel/auditsc.c:1527!
[   79.054396] invalid opcode: 0000 [#2] SMP PTI

I have attached some log file excerpts that cover several reboots with the two kernels and also with intermap=off for 4.4.  These messages may be unrelated to all this, but let me know if I should investigate further.

Thanks,

-Jeff

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: syslog.txt
URL: <http://xenomai.org/pipermail/xenomai/attachments/20191115/440d9883/attachment.txt>

Re: INTR-REMAP error with UDD driver

[Jan Kiszka]

On 16.11.19 00:06, Jeff Webb via Xenomai wrote:
> On Friday, November 15, 2019 7:07 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>> On 15.11.19 01:16, Jeff Webb wrote:
>>
>>> On Thursday, November 14, 2019 3:41 PM, Jan Kiszka jan.kiszka@siemens.com wrote:
>>>
>>>> On 14.11.19 14:16, Jeff Webb wrote:
>>>>
>>>>> On Thursday, November 14, 2019 1:50 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
>>>>>
>>>>>> On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
>>>>>>
>>>>>>> I would like to revive this thread from several months ago:
>>>>>>> https://xenomai.org/pipermail/xenomai/2019-March/040498.html
>>>>>>> The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
>>>>>>> Please review the original thread for more details about the specific error.
>>>>>>> Here are a few more tidbits I have gathered:
>>>>>>>
>>>>>>> -   The UDD driver / userspace code works fine on the other hardware
>>>>>>>
>>>>>>> -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
>>>>>>>
>>>>>>> -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
>>>>>>>
>>>>>>> -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
>>>>>>>
>>>>>>>
>>>>>>> In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
>>>>>>> Interrupts work:
>>>>>>> xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>>>> xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>>>> xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
>>>>>>> INTR-REMAP error:
>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
>>>>>>> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
>>>>>>> xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
>>>>>>> xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
>>>>>>> The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
>>>>>>> Hopefully this additional information can shed a little light on the matter.
>>>>>>
>>>>>> This sounds like some RT interrupt enabling issue related to the IOAPIC
>>>>>> in the x86 I-pipe patch. Please also test 4.19.
>>>>>
>>>>> Ok, I will do this.
>>>
>>> Today I had a chance to make a build based on the ipipe-core-4.19.75-x86-7.patch with xenomai-3.1-rc3. I haven't had a chance to test it thoroughly, but I think it works. I am sorry I didn't try this earlier, and thanks for the reminder to do so. Since a 4.19 patch wasn't available back in early March when I first had the trouble, I got fixated on the fact that things worked with very old kernels and forgot to try 4.19 when I started looking into this again. I will continue to test and let you know if I find any issues.
>>
>> I'm also not seeing the problems I have with 4.4 + intremap on 4.19,
>> despite all debug features on. OK, let's check what we are missing down
>> there...
>
> Just FYI, I did go back and try intremap=off on 4.4.196-xenomai-3.1-rc3-cip38, and it works (as you expected).
>
> Also, I looked at my kernel logs in more detail, and discovered that there are several instances of a kernel BUG message after boot, but then I don't see any more of them later on.  The intremap setting doesn't seem to affect the presence of these messages.  This is the message for 4.4.196-xenomai-3.1-rc3-cip38:
>
> [   53.697717] kernel BUG at kernel/auditsc.c:1502!
> [   53.698132] invalid opcode: 0000 [#1] SMP

I've never seen this before, though AUDITSYSCALL is enabled in my
kernel. Any special rules active according to auditctl -l? Can you
trigger this reliably? Config still the one from March?

>
> There are similar messages with 4.19.75-xenomai-3.1-rc3:
>
> [   65.546333] kernel BUG at kernel/auditsc.c:1527!
> [   79.054396] invalid opcode: 0000 [#2] SMP PTI
>
> I have attached some log file excerpts that cover several reboots with the two kernels and also with intermap=off for 4.4.  These messages may be unrelated to all this, but let me know if I should investigate further.

FWIW, I've released ipipe-core-4.4.199-cip39-x86-20 which at least
solved the intremap issue.

Jan

Re: INTR-REMAP error with UDD driver

[Jeff Webb]

On 11/16/19 12:51 PM, Jan Kiszka via Xenomai wrote:
> On 16.11.19 00:06, Jeff Webb via Xenomai wrote:
>> On Friday, November 15, 2019 7:07 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>> On 15.11.19 01:16, Jeff Webb wrote:
>>>
>>>> On Thursday, November 14, 2019 3:41 PM, Jan Kiszka jan.kiszka@siemens.com wrote:
>>>>
>>>>> On 14.11.19 14:16, Jeff Webb wrote:
>>>>>
>>>>>> On Thursday, November 14, 2019 1:50 AM, Jan Kiszka jan.kiszka@siemens.com wrote:
>>>>>>
>>>>>>> On 14.11.19 06:05, Jeff Webb via Xenomai wrote:
>>>>>>>
>>>>>>>> I would like to revive this thread from several months ago:
>>>>>>>>
>>>>>>>> https://xenomai.org/pipermail/xenomai/2019-March/040498.html
>>>>>>>>
>>>>>>>> The issue is that on some hardware (a specific rack-mount PC with a PICMG daughtercard on a backplane containing PCI and PCIe slots) I get an INTR-REMAP error when trying to receive legacy (not MSI) interrupts from a custom FPGA-based PCI card using a UDD driver. The card did work properly in one out of the five PCI slots on that machine, but UDD interrupts did not work in the other four slots.
>>>>>>>> Please review the original thread for more details about the specific error.
>>>>>>>> Here are a few more tidbits I have gathered:
>>>>>>>>
>>>>>>>> -   The UDD driver / userspace code works fine on the other hardware
>>>>>>>>
>>>>>>>> -   The UDD driver / userspace code works fine in one PCI slot out of five on this hardware.
>>>>>>>>
>>>>>>>> -   With another backplane model, but same processor card, the problem occurs in all four of the PCI slots.
>>>>>>>>
>>>>>>>> -   An almost identical pure-linux UIO version of the driver / userspace code works in all the cases I tested, even when the UDD version fails, and even with the same xenomai-patched kernel used for UDD testing.
>>>>>>>>
>>>>>>>>
>>>>>>>> In one of the previous posts in this thread a few months ago, Per Öberg mentioned experiencing something similar. Based on the information that was shared, I tried my code with linux version 4.9.38, but it still failed. This prompted me to try other linux / ipipe / xenomai combinations. These are my findings:
>>>>>>>> Interrupts work:
>>>>>>>> xenomai-2.6.5 ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>>>>> xenomai-3.0.9+ ipipe-core-3.18.20-x86-7.patch (2016-07-05)
>>>>>>>> xenomai-3.0.9+ ipipe-core-4.1.18-x86-9.patch (2017-05-25)
>>>>>>>> INTR-REMAP error:
>>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-6.patch (2017-02-25)
>>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-7.patch (2017-05-25)
>>>>>>>> xenomai-3.0.9+ ipipe-core-4.4.43-x86-8.patch (2017-06-14)
>>>>>>>> xenomai-3.1-rc3 ipipe-core-4.4.196-cip38-x86-19.patch (2019-11-04)
>>>>>>>> xenomai-3.0.9+ ipipe-core-4.9.38-x86-4.patch (2017-10-03)
>>>>>>>> xenomai-3.0.9 ipipe-core-4.14.132-x86-6.patch (2019-07-03)
>>>>>>>> The Xenomai 2.6.5 version of course does not use UDD, but uses the old pthread_intr_* userspace functions.
>>>>>>>> Hopefully this additional information can shed a little light on the matter.
>>>>>>>
>>>>>>> This sounds like some RT interrupt enabling issue related to the IOAPIC
>>>>>>> in the x86 I-pipe patch. Please also test 4.19.
>>>>>>
>>>>>> Ok, I will do this.
>>>>
>>>> Today I had a chance to make a build based on the ipipe-core-4.19.75-x86-7.patch with xenomai-3.1-rc3. I haven't had a chance to test it thoroughly, but I think it works. I am sorry I didn't try this earlier, and thanks for the reminder to do so. Since a 4.19 patch wasn't available back in early March when I first had the trouble, I got fixated on the fact that things worked with very old kernels and forgot to try 4.19 when I started looking into this again. I will continue to test and let you know if I find any issues.
>>>
>>> I'm also not seeing the problems I have with 4.4 + intremap on 4.19,
>>> despite all debug features on. OK, let's check what we are missing down
>>> there...
>>
>> Just FYI, I did go back and try intremap=off on 4.4.196-xenomai-3.1-rc3-cip38, and it works (as you expected).
>>
>> Also, I looked at my kernel logs in more detail, and discovered that there are several instances of a kernel BUG message after boot, but then I don't see any more of them later on.  The intremap setting doesn't seem to affect the presence of these messages.  This is the message for 4.4.196-xenomai-3.1-rc3-cip38:
>>
>> [   53.697717] kernel BUG at kernel/auditsc.c:1502!
>> [   53.698132] invalid opcode: 0000 [#1] SMP
>
> I've never seen this before, though AUDITSYSCALL is enabled in my
> kernel. Any special rules active according to auditctl -l? Can you
> trigger this reliably? Config still the one from March?

This BUG is triggered on several machines (with similar, but not identical hardware).  I have attached the output of "auditctl -l", as well as the corresponding dmesg output and config.  This could be unrelated to Xenomai, though, as I haven't tried an unpatched kernel.  Sorry to bother you and please disregard if you think it is unrelated to Xenomai.

>> There are similar messages with 4.19.75-xenomai-3.1-rc3:
>>
>> [   65.546333] kernel BUG at kernel/auditsc.c:1527!
>> [   79.054396] invalid opcode: 0000 [#2] SMP PTI
>>
>> I have attached some log file excerpts that cover several reboots with the two kernels and also with intermap=off for 4.4.  These messages may be unrelated to all this, but let me know if I should investigate further.
>
> FWIW, I've released ipipe-core-4.4.199-cip39-x86-20 which at least
> solved the intremap issue.

Thanks, Jan.  I confirmed that ipipe-core-4.4.199-cip39-x86-20 fixes the intremap issue for me as well.  I haven't done extensive testing with this kernel, as I have been using primarily ipipe-core-4.19.84-x86-8.patch, but I know the basic functionality is working.

-Jeff

-------------- next part --------------
A non-text attachment was scrubbed...
Name: auditctl.4.19.84-xenomai-3.1-rc3
Type: application/octet-stream
Size: 3163 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20191126/0ae44caa/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-4.19.84-xenomai-3.1-rc3-gcc7
Type: application/octet-stream
Size: 211149 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20191126/0ae44caa/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dmesg.log
Type: text/x-log
Size: 111526 bytes
Desc: not available
URL: <http://xenomai.org/pipermail/xenomai/attachments/20191126/0ae44caa/attachment.bin>