* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-26 18:05 Heiner Kallweit 2014-10-27 8:52 ` Lucas Stach 0 siblings, 1 reply; 13+ messages in thread From: Heiner Kallweit @ 2014-10-26 18:05 UTC (permalink / raw) To: linux-pci; +Cc: linux-arm-kernel, l.stach I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity. I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet. Rgds, Heiner ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-26 18:05 Setting cpu affinity for MSI-X on i.mx6 possible? Heiner Kallweit @ 2014-10-27 8:52 ` Lucas Stach 0 siblings, 0 replies; 13+ messages in thread From: Lucas Stach @ 2014-10-27 8:52 UTC (permalink / raw) To: Heiner Kallweit; +Cc: linux-pci, linux-arm-kernel Hello Heiner, Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: > I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. > igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. > Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback > for setting cpu affinity. > I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware > architecture or whether just a proper implementation is missing yet. > > Rgds, Heiner The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware. Regards, Lucas -- Pengutronix e.K. | Lucas Stach | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-27 8:52 ` Lucas Stach 0 siblings, 0 replies; 13+ messages in thread From: Lucas Stach @ 2014-10-27 8:52 UTC (permalink / raw) To: linux-arm-kernel Hello Heiner, Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: > I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. > igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. > Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback > for setting cpu affinity. > I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware > architecture or whether just a proper implementation is missing yet. > > Rgds, Heiner The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware. Regards, Lucas -- Pengutronix e.K. | Lucas Stach | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-27 8:52 ` Lucas Stach @ 2014-10-27 19:16 ` Heiner Kallweit -1 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-27 19:16 UTC (permalink / raw) To: Lucas Stach; +Cc: linux-pci, linux-arm-kernel Thanks, Lucas. It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? Rgds, Heiner [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) [ 4.235789] igb 0000:01:00.0: added PHC on eth0 [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) Am 27.10.2014 um 09:52 schrieb Lucas Stach: > Hello Heiner, > > Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: >> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. >> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. >> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback >> for setting cpu affinity. >> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware >> architecture or whether just a proper implementation is missing yet. >> >> Rgds, Heiner > > The MSI controller on imx6 doesn't support MSI-X so igb falls back to > regular MSI interrupts. Also all MSI interrupts are aggregated into a > single GIC interrupt line. This means the best we could do is switch all > PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a > per interrupt basis is impossible on this hardware. > > Regards, > Lucas > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-27 19:16 ` Heiner Kallweit 0 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-27 19:16 UTC (permalink / raw) To: linux-arm-kernel Thanks, Lucas. It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? Rgds, Heiner [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) [ 4.235789] igb 0000:01:00.0: added PHC on eth0 [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) Am 27.10.2014 um 09:52 schrieb Lucas Stach: > Hello Heiner, > > Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: >> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. >> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. >> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback >> for setting cpu affinity. >> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware >> architecture or whether just a proper implementation is missing yet. >> >> Rgds, Heiner > > The MSI controller on imx6 doesn't support MSI-X so igb falls back to > regular MSI interrupts. Also all MSI interrupts are aggregated into a > single GIC interrupt line. This means the best we could do is switch all > PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a > per interrupt basis is impossible on this hardware. > > Regards, > Lucas > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-27 19:16 ` Heiner Kallweit @ 2014-10-27 19:31 ` Bjorn Helgaas -1 siblings, 0 replies; 13+ messages in thread From: Bjorn Helgaas @ 2014-10-27 19:31 UTC (permalink / raw) To: Heiner Kallweit; +Cc: Lucas Stach, linux-pci@vger.kernel.org, linux-arm On Mon, Oct 27, 2014 at 1:16 PM, Heiner Kallweit <heiner.kallweit@web.de> wrote: > Thanks, Lucas. > > It's interesting however that the igb driver claims it's using MSI-X. > Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? > > Rgds, Heiner > > [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k > [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. > [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) > [ 4.235789] igb 0000:01:00.0: added PHC on eth0 > [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection > [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 > [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF > [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention to whether enabling MSI-X worked, but that path in the driver is not completely trivial, and I got tired of tracing the code to verify it. Bjorn ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-27 19:31 ` Bjorn Helgaas 0 siblings, 0 replies; 13+ messages in thread From: Bjorn Helgaas @ 2014-10-27 19:31 UTC (permalink / raw) To: linux-arm-kernel On Mon, Oct 27, 2014 at 1:16 PM, Heiner Kallweit <heiner.kallweit@web.de> wrote: > Thanks, Lucas. > > It's interesting however that the igb driver claims it's using MSI-X. > Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? > > Rgds, Heiner > > [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k > [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. > [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) > [ 4.235789] igb 0000:01:00.0: added PHC on eth0 > [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection > [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 > [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF > [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention to whether enabling MSI-X worked, but that path in the driver is not completely trivial, and I got tired of tracing the code to verify it. Bjorn ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-27 19:31 ` Bjorn Helgaas @ 2014-10-27 19:38 ` Heiner Kallweit -1 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-27 19:38 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Lucas Stach, linux-pci@vger.kernel.org, linux-arm Am 27.10.2014 um 20:31 schrieb Bjorn Helgaas: > On Mon, Oct 27, 2014 at 1:16 PM, Heiner Kallweit <heiner.kallweit@web.de> wrote: >> Thanks, Lucas. >> >> It's interesting however that the igb driver claims it's using MSI-X. >> Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? >> >> Rgds, Heiner >> >> [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k >> [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. >> [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) >> [ 4.235789] igb 0000:01:00.0: added PHC on eth0 >> [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection >> [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 >> [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF >> [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) > > What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention > to whether enabling MSI-X worked, but that path in the driver is not > completely trivial, and I got tired of tracing the code to verify it. > > Bjorn > Thanks, Bjorn. Here comes the full output of the lspci command. [root@utilite ~]# lspci -vvs01:00.0 01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation Device 0000 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 155 Region 0: Memory at 01100000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 1000 [disabled] [size=32] Region 3: Memory at 01120000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 00-a0-c9-ff-ff-00-00-00 Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Kernel driver in use: igb Kernel modules: igb ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-27 19:38 ` Heiner Kallweit 0 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-27 19:38 UTC (permalink / raw) To: linux-arm-kernel Am 27.10.2014 um 20:31 schrieb Bjorn Helgaas: > On Mon, Oct 27, 2014 at 1:16 PM, Heiner Kallweit <heiner.kallweit@web.de> wrote: >> Thanks, Lucas. >> >> It's interesting however that the igb driver claims it's using MSI-X. >> Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? >> >> Rgds, Heiner >> >> [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k >> [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. >> [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) >> [ 4.235789] igb 0000:01:00.0: added PHC on eth0 >> [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection >> [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 >> [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF >> [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) > > What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention > to whether enabling MSI-X worked, but that path in the driver is not > completely trivial, and I got tired of tracing the code to verify it. > > Bjorn > Thanks, Bjorn. Here comes the full output of the lspci command. [root at utilite ~]# lspci -vvs01:00.0 01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) Subsystem: Intel Corporation Device 0000 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 155 Region 0: Memory at 01100000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at 1000 [disabled] [size=32] Region 3: Memory@01120000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 00-a0-c9-ff-ff-00-00-00 Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Kernel driver in use: igb Kernel modules: igb ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-27 19:16 ` Heiner Kallweit @ 2014-10-28 12:10 ` Lucas Stach -1 siblings, 0 replies; 13+ messages in thread From: Lucas Stach @ 2014-10-28 12:10 UTC (permalink / raw) To: Heiner Kallweit; +Cc: linux-pci, linux-arm-kernel Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit: > Thanks, Lucas. > > It's interesting however that the igb driver claims it's using MSI-X. > Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? > Are you running a upstream or downstream vendor kernel? On my upstream kernel igb properly falls back to MSI. Regards, Lucas > Rgds, Heiner > > [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k > [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. > [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) > [ 4.235789] igb 0000:01:00.0: added PHC on eth0 > [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection > [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 > [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF > [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) > > > Am 27.10.2014 um 09:52 schrieb Lucas Stach: > > Hello Heiner, > > > > Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: > >> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. > >> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. > >> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback > >> for setting cpu affinity. > >> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware > >> architecture or whether just a proper implementation is missing yet. > >> > >> Rgds, Heiner > > > > The MSI controller on imx6 doesn't support MSI-X so igb falls back to > > regular MSI interrupts. Also all MSI interrupts are aggregated into a > > single GIC interrupt line. This means the best we could do is switch all > > PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a > > per interrupt basis is impossible on this hardware. > > > > Regards, > > Lucas > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Pengutronix e.K. | Lucas Stach | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-28 12:10 ` Lucas Stach 0 siblings, 0 replies; 13+ messages in thread From: Lucas Stach @ 2014-10-28 12:10 UTC (permalink / raw) To: linux-arm-kernel Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit: > Thanks, Lucas. > > It's interesting however that the igb driver claims it's using MSI-X. > Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? > Are you running a upstream or downstream vendor kernel? On my upstream kernel igb properly falls back to MSI. Regards, Lucas > Rgds, Heiner > > [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k > [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. > [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) > [ 4.235789] igb 0000:01:00.0: added PHC on eth0 > [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection > [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 > [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF > [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) > > > Am 27.10.2014 um 09:52 schrieb Lucas Stach: > > Hello Heiner, > > > > Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: > >> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. > >> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. > >> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback > >> for setting cpu affinity. > >> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware > >> architecture or whether just a proper implementation is missing yet. > >> > >> Rgds, Heiner > > > > The MSI controller on imx6 doesn't support MSI-X so igb falls back to > > regular MSI interrupts. Also all MSI interrupts are aggregated into a > > single GIC interrupt line. This means the best we could do is switch all > > PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a > > per interrupt basis is impossible on this hardware. > > > > Regards, > > Lucas > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Pengutronix e.K. | Lucas Stach | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Setting cpu affinity for MSI-X on i.mx6 possible? 2014-10-28 12:10 ` Lucas Stach @ 2014-10-28 18:20 ` Heiner Kallweit -1 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-28 18:20 UTC (permalink / raw) To: Lucas Stach; +Cc: linux-pci, linux-arm-kernel I'm using the latest vanilla linux-next kernel. Rgds, Heiner Am 28.10.2014 um 13:10 schrieb Lucas Stach: > Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit: >> Thanks, Lucas. >> >> It's interesting however that the igb driver claims it's using MSI-X. >> Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? >> > Are you running a upstream or downstream vendor kernel? > On my upstream kernel igb properly falls back to MSI. > > Regards, > Lucas > >> Rgds, Heiner >> >> [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k >> [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. >> [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) >> [ 4.235789] igb 0000:01:00.0: added PHC on eth0 >> [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection >> [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 >> [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF >> [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) >> >> >> Am 27.10.2014 um 09:52 schrieb Lucas Stach: >>> Hello Heiner, >>> >>> Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: >>>> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. >>>> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. >>>> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback >>>> for setting cpu affinity. >>>> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware >>>> architecture or whether just a proper implementation is missing yet. >>>> >>>> Rgds, Heiner >>> >>> The MSI controller on imx6 doesn't support MSI-X so igb falls back to >>> regular MSI interrupts. Also all MSI interrupts are aggregated into a >>> single GIC interrupt line. This means the best we could do is switch all >>> PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a >>> per interrupt basis is impossible on this hardware. >>> >>> Regards, >>> Lucas >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Setting cpu affinity for MSI-X on i.mx6 possible? @ 2014-10-28 18:20 ` Heiner Kallweit 0 siblings, 0 replies; 13+ messages in thread From: Heiner Kallweit @ 2014-10-28 18:20 UTC (permalink / raw) To: linux-arm-kernel I'm using the latest vanilla linux-next kernel. Rgds, Heiner Am 28.10.2014 um 13:10 schrieb Lucas Stach: > Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit: >> Thanks, Lucas. >> >> It's interesting however that the igb driver claims it's using MSI-X. >> Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI? >> > Are you running a upstream or downstream vendor kernel? > On my upstream kernel igb properly falls back to MSI. > > Regards, > Lucas > >> Rgds, Heiner >> >> [ 4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k >> [ 4.172930] igb: Copyright (c) 2007-2014 Intel Corporation. >> [ 4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142) >> [ 4.235789] igb 0000:01:00.0: added PHC on eth0 >> [ 4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection >> [ 4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58 >> [ 4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF >> [ 4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s) >> >> >> Am 27.10.2014 um 09:52 schrieb Lucas Stach: >>> Hello Heiner, >>> >>> Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit: >>>> I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE. >>>> igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. >>>> Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback >>>> for setting cpu affinity. >>>> I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware >>>> architecture or whether just a proper implementation is missing yet. >>>> >>>> Rgds, Heiner >>> >>> The MSI controller on imx6 doesn't support MSI-X so igb falls back to >>> regular MSI interrupts. Also all MSI interrupts are aggregated into a >>> single GIC interrupt line. This means the best we could do is switch all >>> PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a >>> per interrupt basis is impossible on this hardware. >>> >>> Regards, >>> Lucas >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >> the body of a message to majordomo at vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-10-28 18:20 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-10-26 18:05 Setting cpu affinity for MSI-X on i.mx6 possible? Heiner Kallweit 2014-10-27 8:52 ` Lucas Stach 2014-10-27 8:52 ` Lucas Stach 2014-10-27 19:16 ` Heiner Kallweit 2014-10-27 19:16 ` Heiner Kallweit 2014-10-27 19:31 ` Bjorn Helgaas 2014-10-27 19:31 ` Bjorn Helgaas 2014-10-27 19:38 ` Heiner Kallweit 2014-10-27 19:38 ` Heiner Kallweit 2014-10-28 12:10 ` Lucas Stach 2014-10-28 12:10 ` Lucas Stach 2014-10-28 18:20 ` Heiner Kallweit 2014-10-28 18:20 ` Heiner Kallweit
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.