Legacy features in PCI Express devices

Legacy features in PCI Express devices

[Mason]

Hello,

There are two revisions of our PCI Express controller.

Rev 1 did not support the following features:

  1) legacy PCI interrupt delivery (INTx signals)
  2) I/O address space

Internally, someone stated that such missing support would prevent
some PCIe cards from working with our controller.

Are there really modern PCIe cards that require 1) and/or 2)
to function?

Can someone provide examples of such cards, so that I may test them
on both revisions?

I was told to check ath9k-based cards. Any other examples?

Looking around, I came across this thread:
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
"i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"

IIUC, although some PCIe boards do support MSI, the driver might not
put in the work to use that infrastructure, and instead reverts to
legacy interrupts. (So it is a SW issue, in a sense.)

Regards.

Legacy features in PCI Express devices

[Greg]

On Mon, 2017-03-13 at 17:10 +0100, Mason wrote:
> Hello,
> 
> There are two revisions of our PCI Express controller.
> 
> Rev 1 did not support the following features:
> 
>   1) legacy PCI interrupt delivery (INTx signals)

I'm not sure about this...

>   2) I/O address space

But yes, definitely some support this.

We're working on a new type of network controller that uses I/O for some
types of low latency feature support.

- Greg

> 
> Internally, someone stated that such missing support would prevent
> some PCIe cards from working with our controller.
> 
> Are there really modern PCIe cards that require 1) and/or 2)
> to function?
> 
> Can someone provide examples of such cards, so that I may test them
> on both revisions?
> 
> I was told to check ath9k-based cards. Any other examples?
> 
> Looking around, I came across this thread:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
> "i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"
> 
> IIUC, although some PCIe boards do support MSI, the driver might not
> put in the work to use that infrastructure, and instead reverts to
> legacy interrupts. (So it is a SW issue, in a sense.)
> 
> Regards.

Legacy features in PCI Express devices

[Robin Murphy]

On 13/03/17 16:10, Mason wrote:
> Hello,
> 
> There are two revisions of our PCI Express controller.
> 
> Rev 1 did not support the following features:
> 
>   1) legacy PCI interrupt delivery (INTx signals)
>   2) I/O address space
> 
> Internally, someone stated that such missing support would prevent
> some PCIe cards from working with our controller.
> 
> Are there really modern PCIe cards that require 1) and/or 2)
> to function?
> 
> Can someone provide examples of such cards, so that I may test them
> on both revisions?
> 
> I was told to check ath9k-based cards. Any other examples?
> 
> Looking around, I came across this thread:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
> "i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"
> 
> IIUC, although some PCIe boards do support MSI, the driver might not
> put in the work to use that infrastructure, and instead reverts to
> legacy interrupts. (So it is a SW issue, in a sense.)

Secondary to that category is endpoints which nominally support MSI, but
in a way which is unreliable or otherwise broken. My experience shows
that the Silicon Image SiI 3132 (as integrated on ARM Juno boards, but
seemingly also relatively common on 'generic' 2-port SATA cards) falls
into that category - using the command-line parameter to force MSIs
instead of legacy interrupts leads to the the machine barely reaching
userspace before something goes horribly wrong:

...
         Activating swap Swap Partition...
[   10.817806] Adding 524284k swap on /dev/sda2.  Priority:-1 extents:1
across:524284k SS
         Starting File System Check on /dev/disk/by-uuid/1000-9346...
[  OK  ] Activated swap Swap Partition.
[  OK  ] Activated swap
/dev/disk/by-partuui?6cda4-9f73-4e26-bf19-e71f9319b5ce.
[  OK  ] Reached target Swap.
         Mounting Temporary Directory...
[  OK  ] Mounted Temporary Directory.
[  OK  ] Started File System Check on /dev/disk/by-uuid/1000-9346.
[   46.036065] Unhandled fault: synchronous external abort (0x96000210)
at 0xffff0000092f5000
[   46.044273] Internal error: : 96000210 [#1] PREEMPT SMP
[   46.049445] Modules linked in:
[   46.052474] CPU: 0 PID: 122 Comm: scsi_eh_0 Not tainted 4.11.0-rc1+ #1753
[   46.059192] Hardware name: ARM Juno development board (r1) (DT)
[   46.065052] task: ffff8009763b8000 task.stack: ffff800975f70000
[   46.070918] PC is at ata_wait_register+0x2c/0xa0
[   46.075491] LR is at sil24_init_port+0x60/0x110
[   46.079974] pc : [<ffff0000085b342c>] lr : [<ffff0000085d2b78>]
pstate: 40000145
...

Robin.

Legacy features in PCI Express devices

[Mason]

On 13/03/2017 18:12, Robin Murphy wrote:

> On 13/03/17 16:10, Mason wrote:
>
>> There are two revisions of our PCI Express controller.
>>
>> Rev 1 did not support the following features:
>>
>>   1) legacy PCI interrupt delivery (INTx signals)
>>   2) I/O address space
>>
>> Internally, someone stated that such missing support would prevent
>> some PCIe cards from working with our controller.
>>
>> Are there really modern PCIe cards that require 1) and/or 2)
>> to function?
>>
>> Can someone provide examples of such cards, so that I may test them
>> on both revisions?
>>
>> I was told to check ath9k-based cards. Any other examples?
>>
>> Looking around, I came across this thread:
>> http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
>> "i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"
>>
>> IIUC, although some PCIe boards do support MSI, the driver might not
>> put in the work to use that infrastructure, and instead reverts to
>> legacy interrupts. (So it is a SW issue, in a sense.)
> 
> Secondary to that category is endpoints which nominally support MSI, but
> in a way which is unreliable or otherwise broken. My experience shows
> that the Silicon Image SiI 3132 (as integrated on ARM Juno boards, but
> seemingly also relatively common on 'generic' 2-port SATA cards) falls
> into that category - using the command-line parameter to force MSIs
> instead of legacy interrupts leads to the the machine barely reaching
> userspace before something goes horribly wrong:

Do drivers typically support *both* MSI and INTx?

Specifically, would the xhci driver support both?

If I remove MSI support from my kernel, I might be able to test
legacy interrupt support that way, right?

Regards.

Legacy features in PCI Express devices

[Robin Murphy]

On 13/03/17 17:39, Mason wrote:
> On 13/03/2017 18:12, Robin Murphy wrote:
> 
>> On 13/03/17 16:10, Mason wrote:
>>
>>> There are two revisions of our PCI Express controller.
>>>
>>> Rev 1 did not support the following features:
>>>
>>>   1) legacy PCI interrupt delivery (INTx signals)
>>>   2) I/O address space
>>>
>>> Internally, someone stated that such missing support would prevent
>>> some PCIe cards from working with our controller.
>>>
>>> Are there really modern PCIe cards that require 1) and/or 2)
>>> to function?
>>>
>>> Can someone provide examples of such cards, so that I may test them
>>> on both revisions?
>>>
>>> I was told to check ath9k-based cards. Any other examples?
>>>
>>> Looking around, I came across this thread:
>>> http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
>>> "i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"
>>>
>>> IIUC, although some PCIe boards do support MSI, the driver might not
>>> put in the work to use that infrastructure, and instead reverts to
>>> legacy interrupts. (So it is a SW issue, in a sense.)
>>
>> Secondary to that category is endpoints which nominally support MSI, but
>> in a way which is unreliable or otherwise broken. My experience shows
>> that the Silicon Image SiI 3132 (as integrated on ARM Juno boards, but
>> seemingly also relatively common on 'generic' 2-port SATA cards) falls
>> into that category - using the command-line parameter to force MSIs
>> instead of legacy interrupts leads to the the machine barely reaching
>> userspace before something goes horribly wrong:
> 
> Do drivers typically support *both* MSI and INTx?

I'm not sure about "typically", but it certainly happens. For example,
the Intel e1000e NIC driver is one I know of which can fall back from
MSI-X to MSI to legacy dynamically.

> Specifically, would the xhci driver support both?

Line 415 of xhci.c would appear to imply so.

> If I remove MSI support from my kernel, I might be able to test
> legacy interrupt support that way, right?

Indeed, disabling CONFIG_PCI_MSI should leave drivers with no other choice.

Robin.

> 
> Regards.
> 

Legacy features in PCI Express devices

[David Daney]

On 03/13/2017 09:10 AM, Mason wrote:
> Hello,
>
> There are two revisions of our PCI Express controller.
>
> Rev 1 did not support the following features:
>
>   1) legacy PCI interrupt delivery (INTx signals)
>   2) I/O address space
>
> Internally, someone stated that such missing support would prevent
> some PCIe cards from working with our controller.
>
> Are there really modern PCIe cards that require 1) and/or 2)
> to function?

It depends on your definition of "modern".  I have some JMicron AHCI 
SATA controllers that support legacy interrupts only.  These are cheap 
$20 PCIe devices I picked up at Fry's a couple of years ago.  Do they 
count as modern?

I/O address space is probably less important I would say.

David.

>
> Can someone provide examples of such cards, so that I may test them
> on both revisions?


[root at localhost ddaney]# lspci -s 0005:90:00.0 -vvv
0005:90:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE 
Controller (rev 03) (prog-if 01 [AHCI 1.0])
	Subsystem: JMicron Technology Corp. JMB363 SATA/IDE Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 26
	NUMA node: 0
	Region 5: Memory at 899010010000 (32-bit, non-prefetchable) [size=8K]
	Expansion ROM at 899010000000 [disabled] [size=64K]
	Capabilities: [68] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] Express (v1) Legacy Endpoint, MSI 01
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s 
unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- 
BWMgmt- ABWMgmt-
	Kernel driver in use: ahci

[root@localhost ddaney]# lspci -s 0005:90:00.0 -vvv -n
0005:90:00.0 0106: 197b:2363 (rev 03) (prog-if 01 [AHCI 1.0])
	Subsystem: 197b:2363
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 26
	NUMA node: 0
	Region 5: Memory at 899010010000 (32-bit, non-prefetchable) [size=8K]
	Expansion ROM@899010000000 [disabled] [size=64K]
	Capabilities: [68] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] Express (v1) Legacy Endpoint, MSI 01
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s 
unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- 
BWMgmt- ABWMgmt-
	Kernel driver in use: ahci



Look!  No MSI.

>
> I was told to check ath9k-based cards. Any other examples?
>
> Looking around, I came across this thread:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2016-March/418254.html
> "i.MX6 PCIe: Fix imx6_pcie_deassert_core_reset() polarity"
>
> IIUC, although some PCIe boards do support MSI, the driver might not
> put in the work to use that infrastructure, and instead reverts to
> legacy interrupts. (So it is a SW issue, in a sense.)
>
> Regards.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

Legacy features in PCI Express devices

[Bjorn Helgaas]

On Mon, Mar 13, 2017 at 05:10:57PM +0100, Mason wrote:
> Hello,
> 
> There are two revisions of our PCI Express controller.
> 
> Rev 1 did not support the following features:
> 
>   1) legacy PCI interrupt delivery (INTx signals)
>   2) I/O address space
> 
> Internally, someone stated that such missing support would prevent
> some PCIe cards from working with our controller.
> 
> Are there really modern PCIe cards that require 1) and/or 2)
> to function?

>From a spec point of view, all endpoints, including Legacy, PCI
Express, and Root Complex Integrated Endpoints, are "required to
support MSI or MSI-X or both if an interrupt resource is requested"
(PCIe r3.0, sec 1.3.2).

The same section says Legacy Endpoints are permitted to use I/O
Requests, but PCI Express and Root Complex Integrated Endpoints are
not.  There's a little wiggle room in the I/O BAR description; I would
interpret it to mean the latter two varieties are permitted to have
I/O BARs, but they must make the resources described by those BARs
available via a memory BAR as well, so they can operate with I/O
address space.

But that's only in theory; David has already given examples of devices
that don't support MSI, and Greg hinted at new devices that might
require I/O space.  I'm particularly curious about that last one,
because there are several host bridges that don't support I/O space at
all.

Bjorn