* IOMMU groups: better with Intel root ports, or with PEX8606 switch?
@ 2013-12-29 2:31 Dana Goyette
2013-12-29 3:23 ` Alex Williamson
0 siblings, 1 reply; 15+ messages in thread
From: Dana Goyette @ 2013-12-29 2:31 UTC (permalink / raw)
To: kvm
I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
soon decide which one to keep.
The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
PEX8066 switch, which claims to support ACS. I'd expect the devices
downstream of the PLX switch to be in separate groups.
With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
applied and set for the Intel root ports, the devices behind the switch
remain stuck in the same group.
In terms of passing devices to different VMs, which is better: all
devices on different root ports, or all devices behind the one
ACS-supporting switch?
X10SAT IOMMU groups:
### Group 0 ###
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor
DRAM Controller (rev 06)
### Group 1 ###
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor PCI Express x16 Controller (rev 06)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
[AMD/ATI] Juniper PRO [Radeon HD 5750]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
Juniper HDMI Audio [Radeon HD 5700 Series]
### Group 2 ###
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor HD Audio Controller (rev 06)
### Group 4 ###
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB xHCI (rev 05)
### Group 5 ###
00:16.0 Communication controller: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series
Chipset Family KT Controller (rev 04)
### Group 6 ###
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
I217-LM (rev 05)
### Group 7 ###
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series
Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #1 (rev d5)
### Group 10 ###
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #2 (rev d5)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0
Host Controller (rev 03)
09:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI
Express to PCI Bridge [Cheetah Express] (rev 01)
0a:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221
IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)
### Group 11 ###
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #4 (rev d5)
### Group 12 ###
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #5 (rev d5)
### Group 13 ###
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #1 (rev 05)
### Group 14 ###
00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family
Server Advanced SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series
Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset
Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series
Chipset Family Thermal Management Controller (rev 05)
### Group 15 ###
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
Controller (rev 01)
### Group 16 ###
0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network
Connection (rev 03)
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch? 2013-12-29 2:31 IOMMU groups: better with Intel root ports, or with PEX8606 switch? Dana Goyette @ 2013-12-29 3:23 ` Alex Williamson 2013-12-29 7:32 ` Dana Goyette 0 siblings, 1 reply; 15+ messages in thread From: Alex Williamson @ 2013-12-29 3:23 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > soon decide which one to keep. > > The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > PEX8066 switch, which claims to support ACS. I'd expect the devices > downstream of the PLX switch to be in separate groups. > > With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > applied and set for the Intel root ports, the devices behind the switch > remain stuck in the same group. > > In terms of passing devices to different VMs, which is better: all > devices on different root ports, or all devices behind the one > ACS-supporting switch? Can you provide lspci -vvv info? If you're getting that for groups either the switch has ACS capabilities, but doesn't support the features we need or we're doing something wrong. Thanks, Alex ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch? 2013-12-29 3:23 ` Alex Williamson @ 2013-12-29 7:32 ` Dana Goyette 2013-12-30 4:16 ` Alex Williamson 0 siblings, 1 reply; 15+ messages in thread From: Dana Goyette @ 2013-12-29 7:32 UTC (permalink / raw) To: kvm On 12/28/2013 7:23 PM, Alex Williamson wrote: > On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to >> soon decide which one to keep. >> >> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >> PEX8066 switch, which claims to support ACS. I'd expect the devices >> downstream of the PLX switch to be in separate groups. >> >> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" >> applied and set for the Intel root ports, the devices behind the switch >> remain stuck in the same group. >> >> In terms of passing devices to different VMs, which is better: all >> devices on different root ports, or all devices behind the one >> ACS-supporting switch? > > Can you provide lspci -vvv info? If you're getting that for groups > either the switch has ACS capabilities, but doesn't support the features > we need or we're doing something wrong. Thanks, > > Alex > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > I initially tried attaching the output as a .txt file, but it's too large. Anyway, here's the output of lspci -nnvvv (you may notice that I moved the Radeon to a different slot). 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002b8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 8GT/s, Width x8, ASPM L0s L1, Latency L0 <1us, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Capabilities: [d94 v1] #19 Kernel driver in use: pcieport 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) (prog- if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: eee00000-eeefffff Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002d8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #3, Speed 8GT/s, Width x8, ASPM L0s L1, Latency L0 <256ns, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #2, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=03 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Capabilities: [d94 v1] #19 Kernel driver in use: pcieport 00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) Subsystem: Intel Corporation Device [8086:2010] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 69 Region 0: Memory at ee400000 (64-bit, non-prefetchable) [size=4M] Region 2: Memory at 80000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at f000 [size=64] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00018 Data: 0000 Capabilities: [d0] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a4] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: i915 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) Subsystem: Intel Corporation Device [8086:2010] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 70 Region 0: Memory at eef34000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee005f8 Data: 0000 Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- RBE- FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Kernel driver in use: snd_hda_intel 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) (prog-if 30 [XHCI]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 50 Region 0: Memory at eef20000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Address: 00000000fee003f8 Data: 0000 Kernel driver in use: xhci_hcd 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 11 Region 0: Memory at eef40000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller [8086:8c3d] (rev 04) (prog-if 02 [16550]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin B routed to IRQ 19 Region 0: I/O ports at f0c0 [size=8] Region 1: Memory at eef3e000 (32-bit, non-prefetchable) [size=4K] Capabilities: [c8] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Kernel driver in use: serial 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 05) Subsystem: Super Micro Computer Inc Device [15d9:153a] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 61 Region 0: Memory at eef00000 (32-bit, non-prefetchable) [size=128K] Region 1: Memory at eef3d000 (32-bit, non-prefetchable) [size=4K] Region 2: I/O ports at f060 [size=32] Capabilities: [c8] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00578 Data: 0000 Capabilities: [e0] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: e1000e 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 67 Region 0: Memory at eef30000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee005b8 Data: 0000 Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE- FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=2 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Kernel driver in use: snd_hda_intel 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: eed00000-eedfffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818] Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=04, subordinate=0b, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: ee800000-eebfffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #2, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818] Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: eec00000-eecfffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #3, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818] Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 128 bytes Bus: primary=00, secondary=0d, subordinate=45, sec-latency=0 I/O behind bridge: 00002000-00002fff Memory behind bridge: d8000000-ee0fffff Prefetchable memory behind bridge: 0000000090000000-00000000b1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #5, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Slot #4, PowerLimit 25.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818] Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) (prog-if 20 [EHCI]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 23 Region 0: Memory at eef3b000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) (prog-if 01 [AHCI 1.0]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin B routed to IRQ 59 Region 0: I/O ports at f0b0 [size=8] Region 1: I/O ports at f0a0 [size=4] Region 2: I/O ports at f090 [size=8] Region 3: I/O ports at f080 [size=4] Region 4: I/O ports at f040 [size=32] Region 5: Memory at eef3a000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00518 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Kernel driver in use: ahci 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 5 Region 0: Memory at eef39000 (64-bit, non-prefetchable) [size=256] Region 4: I/O ports at 0580 [size=32] 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin C routed to IRQ 5 Region 0: Memory at eef38000 (64-bit, non-prefetchable) [size=4K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] (prog-if 00 [VGA controller]) Subsystem: XFX Pine Group Inc. Device [1682:2980] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 71 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at eee20000 (64-bit, non-prefetchable) [size=128K] Region 4: I/O ports at e000 [size=256] Expansion ROM at eee00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00718 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: radeon 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] Subsystem: XFX Pine Group Inc. Device [1682:aa58] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 68 Region 0: Memory at eee40000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee005d8 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: snd_hda_intel 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 60 Region 0: I/O ports at d050 [size=8] Region 1: I/O ports at d040 [size=4] Region 2: I/O ports at d030 [size=8] Region 3: I/O ports at d020 [size=4] Region 4: I/O ports at d000 [size=32] Region 5: Memory at eed00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00538 Data: 0000 Capabilities: [78] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <2us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABC, TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Kernel driver in use: ahci 04:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 0: Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=04, secondary=05, subordinate=0b, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: ee800000-eeafffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00318 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-SlotPowerLimit 10.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <2us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=4 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=06 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32+ WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR32 TC/VC=01 Status: NegoPending- InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [448 v1] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 05:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=05, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00358 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #113, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 05:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=05, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00378 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #116, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 05:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=05, secondary=08, subordinate=08, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00398 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #117, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 05:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=05, secondary=09, subordinate=09, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: eea00000-eeafffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee003b8 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #119, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 05:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=05, secondary=0a, subordinate=0b, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: ee800000-ee9fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee003d8 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #9, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #121, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) (prog-if 30 [XHCI]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at eea00000 (64-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [90] MSI-X: Enable+ Count=8 Masked- Vector table: BAR=0 offset=00001000 PBA: BAR=0 offset=00001080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 71680ns Max no snoop latency: 71680ns Kernel driver in use: xhci_hcd 0a:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 1: Memory at ee900000 (32-bit, non-prefetchable) [size=4K] Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: ee800000-ee8fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Bridge: PM- B3+ Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [80] Subsystem: Device [3412:7856] Capabilities: [90] Express (v1) PCI/PCI-X Bridge, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn- 0b:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] [104c:823f] (rev 01) (prog-if 10 [OHCI]) Subsystem: Device [3412:7856] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 (500ns min, 1000ns max), Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 18 Region 0: Memory at ee804000 (32-bit, non-prefetchable) [size=2K] Region 1: Memory at ee800000 (32-bit, non-prefetchable) [size=16K] Capabilities: [44] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME+ Kernel driver in use: firewire_ohci 0c:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) Subsystem: Super Micro Computer Inc Device [15d9:1533] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: Memory at eec00000 (32-bit, non-prefetchable) [size=512K] Region 2: I/O ports at c000 [size=32] Region 3: Memory at eec80000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d8-6d-cf Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Kernel driver in use: igb ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch? 2013-12-29 7:32 ` Dana Goyette @ 2013-12-30 4:16 ` Alex Williamson 2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette 0 siblings, 1 reply; 15+ messages in thread From: Alex Williamson @ 2013-12-30 4:16 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > On 12/28/2013 7:23 PM, Alex Williamson wrote: > > On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > >> soon decide which one to keep. > >> > >> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >> PEX8066 switch, which claims to support ACS. I'd expect the devices > >> downstream of the PLX switch to be in separate groups. > >> > >> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > >> applied and set for the Intel root ports, the devices behind the switch > >> remain stuck in the same group. > >> > >> In terms of passing devices to different VMs, which is better: all > >> devices on different root ports, or all devices behind the one > >> ACS-supporting switch? > > > > Can you provide lspci -vvv info? If you're getting that for groups > > either the switch has ACS capabilities, but doesn't support the features > > we need or we're doing something wrong. Thanks, > > > I initially tried attaching the output as a .txt file, but it's too > large. Anyway, here's the output of lspci -nnvvv (you may notice that I > moved the Radeon to a different slot). Well, something seems amiss since the downstream switch ports all seem to support and enable the correct set of ACS capabilities. I'm tending to suspect something wrong with the ACS override patch or how it's being used since your IOMMU group is still based at the root port. Each root port is isolated from the other root ports though, so something is happening with the override patch. Can you provide the kernel command line you use to enable ACS overrides and the override patch you're using, as it applies to 3.13-rc5? Thanks, Alex ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2013-12-30 4:16 ` Alex Williamson @ 2013-12-31 0:13 ` Dana Goyette 2014-01-02 19:36 ` Alex Williamson 2014-01-04 0:03 ` Alex Williamson 0 siblings, 2 replies; 15+ messages in thread From: Dana Goyette @ 2013-12-31 0:13 UTC (permalink / raw) To: kvm On 12/29/2013 08:16 PM, Alex Williamson wrote: > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to >>>> soon decide which one to keep. >>>> >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>> downstream of the PLX switch to be in separate groups. >>>> >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" >>>> applied and set for the Intel root ports, the devices behind the switch >>>> remain stuck in the same group. >>>> >>>> In terms of passing devices to different VMs, which is better: all >>>> devices on different root ports, or all devices behind the one >>>> ACS-supporting switch? >>> >>> Can you provide lspci -vvv info? If you're getting that for groups >>> either the switch has ACS capabilities, but doesn't support the features >>> we need or we're doing something wrong. Thanks, >>> >> I initially tried attaching the output as a .txt file, but it's too >> large. Anyway, here's the output of lspci -nnvvv (you may notice that I >> moved the Radeon to a different slot). > > Well, something seems amiss since the downstream switch ports all seem > to support and enable the correct set of ACS capabilities. I'm tending > to suspect something wrong with the ACS override patch or how it's being > used since your IOMMU group is still based at the root port. Each root > port is isolated from the other root ports though, so something is > happening with the override patch. Can you provide the kernel command > line you use to enable ACS overrides and the override patch you're > using, as it applies to 3.13-rc5? Thanks, > > Alex > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > I'm using the original acs-override patch from this post: https://lkml.org/lkml/2013/5/30/513 Kernel parameter is: pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 When booting a kernel without the override patch, the following devices are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire controller; Intel I210 Ethernet controller. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette @ 2014-01-02 19:36 ` Alex Williamson 2014-01-02 21:01 ` Dana Goyette 2014-01-04 0:03 ` Alex Williamson 1 sibling, 1 reply; 15+ messages in thread From: Alex Williamson @ 2014-01-02 19:36 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm, linux-pci On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: > On 12/29/2013 08:16 PM, Alex Williamson wrote: > > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > >> On 12/28/2013 7:23 PM, Alex Williamson wrote: > >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > >>>> soon decide which one to keep. > >>>> > >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >>>> PEX8066 switch, which claims to support ACS. I'd expect the devices > >>>> downstream of the PLX switch to be in separate groups. > >>>> > >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > >>>> applied and set for the Intel root ports, the devices behind the switch > >>>> remain stuck in the same group. > >>>> > >>>> In terms of passing devices to different VMs, which is better: all > >>>> devices on different root ports, or all devices behind the one > >>>> ACS-supporting switch? > >>> > >>> Can you provide lspci -vvv info? If you're getting that for groups > >>> either the switch has ACS capabilities, but doesn't support the features > >>> we need or we're doing something wrong. Thanks, > >>> > >> I initially tried attaching the output as a .txt file, but it's too > >> large. Anyway, here's the output of lspci -nnvvv (you may notice that I > >> moved the Radeon to a different slot). > > > > Well, something seems amiss since the downstream switch ports all seem > > to support and enable the correct set of ACS capabilities. I'm tending > > to suspect something wrong with the ACS override patch or how it's being > > used since your IOMMU group is still based at the root port. Each root > > port is isolated from the other root ports though, so something is > > happening with the override patch. Can you provide the kernel command > > line you use to enable ACS overrides and the override patch you're > > using, as it applies to 3.13-rc5? Thanks, > > > > Alex > > > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > I'm using the original acs-override patch from this post: > https://lkml.org/lkml/2013/5/30/513 > > Kernel parameter is: > pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 > > When booting a kernel without the override patch, the following devices > are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA > controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire > controller; Intel I210 Ethernet controller. Ok, here's my shot in the dark; we must be detecting something about the upstream switch port to make it fail the ACS test and the only thing I can find that might do this is if the PCI config header on the upstream switch reported itself as a multifunction device. Multifunction upstream switch ports do need ACS capabilities to make sure that traffic isn't routed back through other functions. Single function devices do not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're looking to see whether the byte at 0xe has the MSB set. If it does, it lies that it's a multifunction device. If it doesn't I'll have to get the dart board back out. FWIW, you should be able to work around this by adding id:10b5:8606 to your list of overrides. Long term, if this is the problem, we'll want to add a quirk to sanitize the multifunction device flag. Thanks, Alex ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-02 19:36 ` Alex Williamson @ 2014-01-02 21:01 ` Dana Goyette 2014-01-02 21:14 ` Alex Williamson 2014-01-02 21:15 ` Dana Goyette 0 siblings, 2 replies; 15+ messages in thread From: Dana Goyette @ 2014-01-02 21:01 UTC (permalink / raw) To: kvm; +Cc: linux-pci On 01/02/2014 11:36 AM, Alex Williamson wrote: > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: >> On 12/29/2013 08:16 PM, Alex Williamson wrote: >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to >>>>>> soon decide which one to keep. >>>>>> >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>>>> downstream of the PLX switch to be in separate groups. >>>>>> >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" >>>>>> applied and set for the Intel root ports, the devices behind the switch >>>>>> remain stuck in the same group. >>>>>> >>>>>> In terms of passing devices to different VMs, which is better: all >>>>>> devices on different root ports, or all devices behind the one >>>>>> ACS-supporting switch? >>>>> >>>>> Can you provide lspci -vvv info? If you're getting that for groups >>>>> either the switch has ACS capabilities, but doesn't support the features >>>>> we need or we're doing something wrong. Thanks, >>>>> >>>> I initially tried attaching the output as a .txt file, but it's too >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I >>>> moved the Radeon to a different slot). >>> >>> Well, something seems amiss since the downstream switch ports all seem >>> to support and enable the correct set of ACS capabilities. I'm tending >>> to suspect something wrong with the ACS override patch or how it's being >>> used since your IOMMU group is still based at the root port. Each root >>> port is isolated from the other root ports though, so something is >>> happening with the override patch. Can you provide the kernel command >>> line you use to enable ACS overrides and the override patch you're >>> using, as it applies to 3.13-rc5? Thanks, >>> >>> Alex >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> I'm using the original acs-override patch from this post: >> https://lkml.org/lkml/2013/5/30/513 >> >> Kernel parameter is: >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 >> >> When booting a kernel without the override patch, the following devices >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire >> controller; Intel I210 Ethernet controller. > > Ok, here's my shot in the dark; we must be detecting something about the > upstream switch port to make it fail the ACS test and the only thing I > can find that might do this is if the PCI config header on the upstream > switch reported itself as a multifunction device. Multifunction > upstream switch ports do need ACS capabilities to make sure that traffic > isn't routed back through other functions. Single function devices do > not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're > looking to see whether the byte at 0xe has the MSB set. If it does, it > lies that it's a multifunction device. If it doesn't I'll have to get > the dart board back out. > > FWIW, you should be able to work around this by adding id:10b5:8606 to > your list of overrides. Long term, if this is the problem, we'll want > to add a quirk to sanitize the multifunction device flag. Thanks, > > Alex > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the upstream port of the PLX switch. 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 Memory behind bridge: ee800000-eeafffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-02 21:01 ` Dana Goyette @ 2014-01-02 21:14 ` Alex Williamson 2014-01-02 21:15 ` Dana Goyette 1 sibling, 0 replies; 15+ messages in thread From: Alex Williamson @ 2014-01-02 21:14 UTC (permalink / raw) To: Dana Goyette; +Cc: linux-pci, kvm On Thu, 2014-01-02 at 13:01 -0800, Dana Goyette wrote: > On 01/02/2014 11:36 AM, Alex Williamson wrote: > > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: > >> On 12/29/2013 08:16 PM, Alex Williamson wrote: > >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: > >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > >>>>>> soon decide which one to keep. > >>>>>> > >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices > >>>>>> downstream of the PLX switch to be in separate groups. > >>>>>> > >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > >>>>>> applied and set for the Intel root ports, the devices behind the switch > >>>>>> remain stuck in the same group. > >>>>>> > >>>>>> In terms of passing devices to different VMs, which is better: all > >>>>>> devices on different root ports, or all devices behind the one > >>>>>> ACS-supporting switch? > >>>>> > >>>>> Can you provide lspci -vvv info? If you're getting that for groups > >>>>> either the switch has ACS capabilities, but doesn't support the features > >>>>> we need or we're doing something wrong. Thanks, > >>>>> > >>>> I initially tried attaching the output as a .txt file, but it's too > >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I > >>>> moved the Radeon to a different slot). > >>> > >>> Well, something seems amiss since the downstream switch ports all seem > >>> to support and enable the correct set of ACS capabilities. I'm tending > >>> to suspect something wrong with the ACS override patch or how it's being > >>> used since your IOMMU group is still based at the root port. Each root > >>> port is isolated from the other root ports though, so something is > >>> happening with the override patch. Can you provide the kernel command > >>> line you use to enable ACS overrides and the override patch you're > >>> using, as it applies to 3.13-rc5? Thanks, > >>> > >>> Alex > >>> > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe kvm" in > >>> the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> > >> I'm using the original acs-override patch from this post: > >> https://lkml.org/lkml/2013/5/30/513 > >> > >> Kernel parameter is: > >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 > >> > >> When booting a kernel without the override patch, the following devices > >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA > >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire > >> controller; Intel I210 Ethernet controller. > > > > Ok, here's my shot in the dark; we must be detecting something about the > > upstream switch port to make it fail the ACS test and the only thing I > > can find that might do this is if the PCI config header on the upstream > > switch reported itself as a multifunction device. Multifunction > > upstream switch ports do need ACS capabilities to make sure that traffic > > isn't routed back through other functions. Single function devices do > > not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're > > looking to see whether the byte at 0xe has the MSB set. If it does, it > > lies that it's a multifunction device. If it doesn't I'll have to get > > the dart board back out. > > > > FWIW, you should be able to work around this by adding id:10b5:8606 to > > your list of overrides. Long term, if this is the problem, we'll want > > to add a quirk to sanitize the multifunction device flag. Thanks, > > > > Alex > > > > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the > upstream port of the PLX switch. Yes, it was 04:00.0 in the previous lspci listing. > 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] > Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 > Memory behind bridge: ee800000-eeafffff > Capabilities: [40] Power Management version 3 > Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ > Capabilities: [68] Express Upstream Port, MSI 00 > Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 > Port PCI Express Gen 2 (5.0 GT/s) Switch > Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 > Capabilities: [fb4] Advanced Error Reporting > Capabilities: [138] Power Budgeting <?> > Capabilities: [148] Virtual Channel > Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> > Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> > Kernel driver in use: pcieport > 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 Well, that wasn't it. I'll keep looking and probably send you a patch to figure out what's going wrong. Thanks, Alex ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-02 21:01 ` Dana Goyette 2014-01-02 21:14 ` Alex Williamson @ 2014-01-02 21:15 ` Dana Goyette 2014-01-02 21:22 ` Alex Williamson 1 sibling, 1 reply; 15+ messages in thread From: Dana Goyette @ 2014-01-02 21:15 UTC (permalink / raw) To: kvm; +Cc: linux-pci On 01/02/2014 01:01 PM, Dana Goyette wrote: > On 01/02/2014 11:36 AM, Alex Williamson wrote: >> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: >>> On 12/29/2013 08:16 PM, Alex Williamson wrote: >>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I >>>>>>> need to >>>>>>> soon decide which one to keep. >>>>>>> >>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>>>>> downstream of the PLX switch to be in separate groups. >>>>>>> >>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS >>>>>>> capabilities" >>>>>>> applied and set for the Intel root ports, the devices behind the >>>>>>> switch >>>>>>> remain stuck in the same group. >>>>>>> >>>>>>> In terms of passing devices to different VMs, which is better: all >>>>>>> devices on different root ports, or all devices behind the one >>>>>>> ACS-supporting switch? >>>>>> >>>>>> Can you provide lspci -vvv info? If you're getting that for groups >>>>>> either the switch has ACS capabilities, but doesn't support the >>>>>> features >>>>>> we need or we're doing something wrong. Thanks, >>>>>> >>>>> I initially tried attaching the output as a .txt file, but it's too >>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice >>>>> that I >>>>> moved the Radeon to a different slot). >>>> >>>> Well, something seems amiss since the downstream switch ports all seem >>>> to support and enable the correct set of ACS capabilities. I'm tending >>>> to suspect something wrong with the ACS override patch or how it's >>>> being >>>> used since your IOMMU group is still based at the root port. Each root >>>> port is isolated from the other root ports though, so something is >>>> happening with the override patch. Can you provide the kernel command >>>> line you use to enable ACS overrides and the override patch you're >>>> using, as it applies to 3.13-rc5? Thanks, >>>> >>>> Alex >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> I'm using the original acs-override patch from this post: >>> https://lkml.org/lkml/2013/5/30/513 >>> >>> Kernel parameter is: >>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 >>> >>> When booting a kernel without the override patch, the following devices >>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA >>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire >>> controller; Intel I210 Ethernet controller. >> >> Ok, here's my shot in the dark; we must be detecting something about the >> upstream switch port to make it fail the ACS test and the only thing I >> can find that might do this is if the PCI config header on the upstream >> switch reported itself as a multifunction device. Multifunction >> upstream switch ports do need ACS capabilities to make sure that traffic >> isn't routed back through other functions. Single function devices do >> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're >> looking to see whether the byte at 0xe has the MSB set. If it does, it >> lies that it's a multifunction device. If it doesn't I'll have to get >> the dart board back out. >> >> FWIW, you should be able to work around this by adding id:10b5:8606 to >> your list of overrides. Long term, if this is the problem, we'll want >> to add a quirk to sanitize the multifunction device flag. Thanks, >> >> Alex >> > > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the > upstream port of the PLX switch. > > 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] > Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 > Memory behind bridge: ee800000-eeafffff > Capabilities: [40] Power Management version 3 > Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ > Capabilities: [68] Express Upstream Port, MSI 00 > Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, > 6 Port PCI Express Gen 2 (5.0 GT/s) Switch > Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 > Capabilities: [fb4] Advanced Error Reporting > Capabilities: [138] Power Budgeting <?> > Capabilities: [148] Virtual Channel > Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 > Len=0cc <?> > Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 > Len=010 <?> > Kernel driver in use: pcieport > 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 > 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 > 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Just in case: The root port: 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0 Memory behind bridge: ee800000-eebfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818 Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00 The first downstream port: 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=06, secondary=07, subordinate=07, sec-latency=0 Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-02 21:15 ` Dana Goyette @ 2014-01-02 21:22 ` Alex Williamson 2014-01-02 21:25 ` Dana Goyette 0 siblings, 1 reply; 15+ messages in thread From: Alex Williamson @ 2014-01-02 21:22 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm, linux-pci On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote: > On 01/02/2014 01:01 PM, Dana Goyette wrote: > > On 01/02/2014 11:36 AM, Alex Williamson wrote: > >> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: > >>> On 12/29/2013 08:16 PM, Alex Williamson wrote: > >>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > >>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: > >>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I > >>>>>>> need to > >>>>>>> soon decide which one to keep. > >>>>>>> > >>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices > >>>>>>> downstream of the PLX switch to be in separate groups. > >>>>>>> > >>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS > >>>>>>> capabilities" > >>>>>>> applied and set for the Intel root ports, the devices behind the > >>>>>>> switch > >>>>>>> remain stuck in the same group. > >>>>>>> > >>>>>>> In terms of passing devices to different VMs, which is better: all > >>>>>>> devices on different root ports, or all devices behind the one > >>>>>>> ACS-supporting switch? > >>>>>> > >>>>>> Can you provide lspci -vvv info? If you're getting that for groups > >>>>>> either the switch has ACS capabilities, but doesn't support the > >>>>>> features > >>>>>> we need or we're doing something wrong. Thanks, > >>>>>> > >>>>> I initially tried attaching the output as a .txt file, but it's too > >>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice > >>>>> that I > >>>>> moved the Radeon to a different slot). > >>>> > >>>> Well, something seems amiss since the downstream switch ports all seem > >>>> to support and enable the correct set of ACS capabilities. I'm tending > >>>> to suspect something wrong with the ACS override patch or how it's > >>>> being > >>>> used since your IOMMU group is still based at the root port. Each root > >>>> port is isolated from the other root ports though, so something is > >>>> happening with the override patch. Can you provide the kernel command > >>>> line you use to enable ACS overrides and the override patch you're > >>>> using, as it applies to 3.13-rc5? Thanks, > >>>> > >>>> Alex > >>>> > >>>> -- > >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in > >>>> the body of a message to majordomo@vger.kernel.org > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>> > >>> I'm using the original acs-override patch from this post: > >>> https://lkml.org/lkml/2013/5/30/513 > >>> > >>> Kernel parameter is: > >>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 > >>> > >>> When booting a kernel without the override patch, the following devices > >>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA > >>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire > >>> controller; Intel I210 Ethernet controller. > >> > >> Ok, here's my shot in the dark; we must be detecting something about the > >> upstream switch port to make it fail the ACS test and the only thing I > >> can find that might do this is if the PCI config header on the upstream > >> switch reported itself as a multifunction device. Multifunction > >> upstream switch ports do need ACS capabilities to make sure that traffic > >> isn't routed back through other functions. Single function devices do > >> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're > >> looking to see whether the byte at 0xe has the MSB set. If it does, it > >> lies that it's a multifunction device. If it doesn't I'll have to get > >> the dart board back out. > >> > >> FWIW, you should be able to work around this by adding id:10b5:8606 to > >> your list of overrides. Long term, if this is the problem, we'll want > >> to add a quirk to sanitize the multifunction device flag. Thanks, > >> > >> Alex > >> > > > > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the > > upstream port of the PLX switch. > > > > 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) > > Flags: bus master, fast devsel, latency 0 > > Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] > > Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 > > Memory behind bridge: ee800000-eeafffff > > Capabilities: [40] Power Management version 3 > > Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ > > Capabilities: [68] Express Upstream Port, MSI 00 > > Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, > > 6 Port PCI Express Gen 2 (5.0 GT/s) Switch > > Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 > > Capabilities: [fb4] Advanced Error Reporting > > Capabilities: [138] Power Budgeting <?> > > Capabilities: [148] Virtual Channel > > Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 > > Len=0cc <?> > > Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 > > Len=010 <?> > > Kernel driver in use: pcieport > > 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 > > 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 > > 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 > > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > Just in case: > > The root port: > 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset > Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0 > Memory behind bridge: ee800000-eebfffff > Capabilities: [40] Express Root Port (Slot+), MSI 00 > Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- > Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818 > Capabilities: [a0] Power Management version 3 > Kernel driver in use: pcieport > 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00 > 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20 > 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00 > > The first downstream port: > > 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Bus: primary=06, secondary=07, subordinate=07, sec-latency=0 > Capabilities: [40] Power Management version 3 > Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ > Capabilities: [68] Express Downstream Port (Slot+), MSI 00 > Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 > Port PCI Express Gen 2 (5.0 GT/s) Switch > Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 > Capabilities: [fb4] Advanced Error Reporting > Capabilities: [148] Virtual Channel > Capabilities: [520] Access Control Services > Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> > Kernel driver in use: pcieport > 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00 > 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00 > 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00 > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 Thanks, can you confirm whether adding id:10b5:8606 to the overrides works? Alex ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-02 21:22 ` Alex Williamson @ 2014-01-02 21:25 ` Dana Goyette 0 siblings, 0 replies; 15+ messages in thread From: Dana Goyette @ 2014-01-02 21:25 UTC (permalink / raw) To: linux-pci; +Cc: kvm On 01/02/2014 01:22 PM, Alex Williamson wrote: > On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote: >> On 01/02/2014 01:01 PM, Dana Goyette wrote: >>> On 01/02/2014 11:36 AM, Alex Williamson wrote: >>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: >>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote: >>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I >>>>>>>>> need to >>>>>>>>> soon decide which one to keep. >>>>>>>>> >>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>>>>>>> downstream of the PLX switch to be in separate groups. >>>>>>>>> >>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS >>>>>>>>> capabilities" >>>>>>>>> applied and set for the Intel root ports, the devices behind the >>>>>>>>> switch >>>>>>>>> remain stuck in the same group. >>>>>>>>> >>>>>>>>> In terms of passing devices to different VMs, which is better: all >>>>>>>>> devices on different root ports, or all devices behind the one >>>>>>>>> ACS-supporting switch? >>>>>>>> >>>>>>>> Can you provide lspci -vvv info? If you're getting that for groups >>>>>>>> either the switch has ACS capabilities, but doesn't support the >>>>>>>> features >>>>>>>> we need or we're doing something wrong. Thanks, >>>>>>>> >>>>>>> I initially tried attaching the output as a .txt file, but it's too >>>>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice >>>>>>> that I >>>>>>> moved the Radeon to a different slot). >>>>>> >>>>>> Well, something seems amiss since the downstream switch ports all seem >>>>>> to support and enable the correct set of ACS capabilities. I'm tending >>>>>> to suspect something wrong with the ACS override patch or how it's >>>>>> being >>>>>> used since your IOMMU group is still based at the root port. Each root >>>>>> port is isolated from the other root ports though, so something is >>>>>> happening with the override patch. Can you provide the kernel command >>>>>> line you use to enable ACS overrides and the override patch you're >>>>>> using, as it applies to 3.13-rc5? Thanks, >>>>>> >>>>>> Alex >>>>>> >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>> I'm using the original acs-override patch from this post: >>>>> https://lkml.org/lkml/2013/5/30/513 >>>>> >>>>> Kernel parameter is: >>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 >>>>> >>>>> When booting a kernel without the override patch, the following devices >>>>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA >>>>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire >>>>> controller; Intel I210 Ethernet controller. >>>> >>>> Ok, here's my shot in the dark; we must be detecting something about the >>>> upstream switch port to make it fail the ACS test and the only thing I >>>> can find that might do this is if the PCI config header on the upstream >>>> switch reported itself as a multifunction device. Multifunction >>>> upstream switch ports do need ACS capabilities to make sure that traffic >>>> isn't routed back through other functions. Single function devices do >>>> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're >>>> looking to see whether the byte at 0xe has the MSB set. If it does, it >>>> lies that it's a multifunction device. If it doesn't I'll have to get >>>> the dart board back out. >>>> >>>> FWIW, you should be able to work around this by adding id:10b5:8606 to >>>> your list of overrides. Long term, if this is the problem, we'll want >>>> to add a quirk to sanitize the multifunction device flag. Thanks, >>>> >>>> Alex >>>> >>> >>> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the >>> upstream port of the PLX switch. >>> >>> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI >>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) >>> Flags: bus master, fast devsel, latency 0 >>> Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] >>> Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0 >>> Memory behind bridge: ee800000-eeafffff >>> Capabilities: [40] Power Management version 3 >>> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ >>> Capabilities: [68] Express Upstream Port, MSI 00 >>> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, >>> 6 Port PCI Express Gen 2 (5.0 GT/s) Switch >>> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 >>> Capabilities: [fb4] Advanced Error Reporting >>> Capabilities: [138] Power Budgeting <?> >>> Capabilities: [148] Virtual Channel >>> Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 >>> Len=0cc <?> >>> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 >>> Len=010 <?> >>> Kernel driver in use: pcieport >>> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00 >>> 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00 >>> 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 >>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> Just in case: >> >> The root port: >> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset >> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode]) >> Flags: bus master, fast devsel, latency 0 >> Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0 >> Memory behind bridge: ee800000-eebfffff >> Capabilities: [40] Express Root Port (Slot+), MSI 00 >> Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- >> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818 >> Capabilities: [a0] Power Management version 3 >> Kernel driver in use: pcieport >> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00 >> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20 >> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00 >> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00 >> >> The first downstream port: >> >> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI >> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) >> Flags: bus master, fast devsel, latency 0 >> Bus: primary=06, secondary=07, subordinate=07, sec-latency=0 >> Capabilities: [40] Power Management version 3 >> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ >> Capabilities: [68] Express Downstream Port (Slot+), MSI 00 >> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 >> Port PCI Express Gen 2 (5.0 GT/s) Switch >> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 >> Capabilities: [fb4] Advanced Error Reporting >> Capabilities: [148] Virtual Channel >> Capabilities: [520] Access Control Services >> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> >> Kernel driver in use: pcieport >> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00 >> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00 >> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00 >> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00 > > Thanks, can you confirm whether adding id:10b5:8606 to the overrides > works? > > Alex > Oddly, adding that ID doesn't seem to change anything. \boot\vmlinuz-3.13.0-rc5 ro root=UUID=8414610a-91ff-41b9-80c2-cc49ee34cce6 console=ttyS4,115200n8 console=tty0 intel_iommu=on,igfx_off pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 initrd=boot\initrd.img-3.13.0-rc5 ### Group 10 ### 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0a:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01) 0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette 2014-01-02 19:36 ` Alex Williamson @ 2014-01-04 0:03 ` Alex Williamson 2014-01-04 19:26 ` Dana Goyette 1 sibling, 1 reply; 15+ messages in thread From: Alex Williamson @ 2014-01-04 0:03 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: > On 12/29/2013 08:16 PM, Alex Williamson wrote: > > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > >> On 12/28/2013 7:23 PM, Alex Williamson wrote: > >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > >>>> soon decide which one to keep. > >>>> > >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >>>> PEX8066 switch, which claims to support ACS. I'd expect the devices > >>>> downstream of the PLX switch to be in separate groups. > >>>> > >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > >>>> applied and set for the Intel root ports, the devices behind the switch > >>>> remain stuck in the same group. > >>>> > >>>> In terms of passing devices to different VMs, which is better: all > >>>> devices on different root ports, or all devices behind the one > >>>> ACS-supporting switch? > >>> > >>> Can you provide lspci -vvv info? If you're getting that for groups > >>> either the switch has ACS capabilities, but doesn't support the features > >>> we need or we're doing something wrong. Thanks, > >>> > >> I initially tried attaching the output as a .txt file, but it's too > >> large. Anyway, here's the output of lspci -nnvvv (you may notice that I > >> moved the Radeon to a different slot). > > > > Well, something seems amiss since the downstream switch ports all seem > > to support and enable the correct set of ACS capabilities. I'm tending > > to suspect something wrong with the ACS override patch or how it's being > > used since your IOMMU group is still based at the root port. Each root > > port is isolated from the other root ports though, so something is > > happening with the override patch. Can you provide the kernel command > > line you use to enable ACS overrides and the override patch you're > > using, as it applies to 3.13-rc5? Thanks, > > > > Alex > > > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > I'm using the original acs-override patch from this post: > https://lkml.org/lkml/2013/5/30/513 > > Kernel parameter is: > pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 > > When booting a kernel without the override patch, the following devices > are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA > controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire > controller; Intel I210 Ethernet controller. Could you please try the patch below and send dmesg for the system once booted. This applies directly to upstream and includes the acs override patch. Thanks, Alex diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index b9e9bd8..2481c6c 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2473,6 +2473,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nomsi Do not use MSI for native PCIe PME signaling (this makes all PCIe root ports use INTx for all services). + pcie_acs_override = + [PCIE] Override missing PCIe ACS support for: + downstream + All downstream ports - full ACS capabilties + multifunction + All multifunction devices - multifunction ACS subset + id:nnnn:nnnn + Specfic device - full ACS capabilities + Specified as vid:did (vendor/device ID) in hex + pcmv= [HW,PCMCIA] BadgePAD 4 pd. [PARIDE] diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 43b9bfe..b31961d 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -4166,8 +4166,10 @@ static int intel_iommu_add_device(struct device *dev) pdev->bus->number, pdev->devfn)) return -ENODEV; + pr_info("%s(%s)\n", __func__, pci_name(pdev)); bridge = pci_find_upstream_pcie_bridge(pdev); if (bridge) { + pr_info("Upstream bridge %s\n", pci_name(bridge)); if (pci_is_pcie(bridge)) dma_pdev = pci_get_domain_bus_and_slot( pci_domain_nr(pdev->bus), @@ -4177,9 +4179,13 @@ static int intel_iommu_add_device(struct device *dev) } else dma_pdev = pci_dev_get(pdev); + pr_info("dma_pdev #1: %s\n", pci_name(dma_pdev)); + /* Account for quirked devices */ swap_pci_ref(&dma_pdev, pci_get_dma_source(dma_pdev)); + pr_info("dma_pdev #2: %s\n", pci_name(dma_pdev)); + /* * If it's a multifunction device that does not support our * required ACS flags, add to the same group as lowest numbered @@ -4204,6 +4210,8 @@ static int intel_iommu_add_device(struct device *dev) } } + pr_info("dma_pdev #3: %s\n", pci_name(dma_pdev)); + /* * Devices on the root bus go through the iommu. If that's not us, * find the next upstream device and test ACS up to the root bus. @@ -4225,6 +4233,8 @@ static int intel_iommu_add_device(struct device *dev) swap_pci_ref(&dma_pdev, pci_dev_get(bus->self)); } + pr_info("dma_pdev #4: %s\n", pci_name(dma_pdev)); + root_bus: group = iommu_group_get(&dma_pdev->dev); pci_dev_put(dma_pdev); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 07369f3..a6be83a 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2395,10 +2395,15 @@ static bool pci_acs_flags_enabled(struct pci_dev *pdev, u16 acs_flags) { int pos; u16 cap, ctrl; + bool ret; pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ACS); - if (!pos) - return false; + if (!pos) { + pr_info("%s no ACS capability on %s\n", + __func__, pci_name(pdev)); + ret = false; + goto out; + } /* * Except for egress control, capabilities are either required @@ -2409,7 +2414,11 @@ static bool pci_acs_flags_enabled(struct pci_dev *pdev, u16 acs_flags) acs_flags &= (cap | PCI_ACS_EC); pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl); - return (ctrl & acs_flags) == acs_flags; + ret = (ctrl & acs_flags) == acs_flags; +out: + pr_info("%s(%s, %04x) -> %s\n", __func__, pci_name(pdev), acs_flags, + ret ? "true" : "false"); + return ret; } /** @@ -2432,17 +2441,24 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags) { int ret; + pr_info("%s(%s, %04x)\n", __func__, pci_name(pdev), acs_flags); ret = pci_dev_specific_acs_enabled(pdev, acs_flags); - if (ret >= 0) + if (ret >= 0) { + pr_info("-> %s\n", ret > 0 ? "true" : "false"); return ret > 0; + } /* * Conventional PCI and PCI-X devices never support ACS, either * effectively or actually. The shared bus topology implies that * any device on the bus can receive or snoop DMA. */ - if (!pci_is_pcie(pdev)) + if (!pci_is_pcie(pdev)) { + pr_info("-> false\n"); return false; + } + + ret = true; switch (pci_pcie_type(pdev)) { /* @@ -2459,7 +2475,8 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags) */ case PCI_EXP_TYPE_PCI_BRIDGE: case PCI_EXP_TYPE_RC_EC: - return false; + ret = false; + break; /* * PCIe 3.0, 6.12.1.1 specifies that downstream and root ports should * implement ACS in order to indicate their peer-to-peer capabilities, @@ -2467,7 +2484,8 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags) */ case PCI_EXP_TYPE_DOWNSTREAM: case PCI_EXP_TYPE_ROOT_PORT: - return pci_acs_flags_enabled(pdev, acs_flags); + ret = pci_acs_flags_enabled(pdev, acs_flags); + break; /* * PCIe 3.0, 6.12.1.2 specifies ACS capabilities that should be * implemented by the remaining PCIe types to indicate peer-to-peer @@ -2482,14 +2500,15 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags) if (!pdev->multifunction) break; - return pci_acs_flags_enabled(pdev, acs_flags); + ret = pci_acs_flags_enabled(pdev, acs_flags); } /* * PCIe 3.0, 6.12.1.3 specifies no ACS capabilities are applicable * to single function devices with the exception of downstream ports. */ - return true; + pr_info("-> %s\n", ret ? "true" : "false"); + return ret; } /** @@ -2505,20 +2524,28 @@ bool pci_acs_path_enabled(struct pci_dev *start, struct pci_dev *end, u16 acs_flags) { struct pci_dev *pdev, *parent = start; + bool ret = true; + pr_info("%s(%s, %s, %04x)\n", __func__, pci_name(start), + end ? pci_name(end) : "NULL", acs_flags); do { pdev = parent; - if (!pci_acs_enabled(pdev, acs_flags)) - return false; + if (!pci_acs_enabled(pdev, acs_flags)) { + ret = false; + break; + } - if (pci_is_root_bus(pdev->bus)) - return (end == NULL); + if (pci_is_root_bus(pdev->bus)) { + ret = (end == NULL); + break; + } parent = pdev->bus->self; } while (pdev != end); - return true; + pr_info("-> %s\n", ret ? "true" : "false"); + return ret; } /** diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 3a02717..1e0ec58 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3423,11 +3423,130 @@ static int pci_quirk_amd_sb_acs(struct pci_dev *dev, u16 acs_flags) #endif } +static bool acs_on_downstream; +static bool acs_on_multifunction; + +#define NUM_ACS_IDS 16 +struct acs_on_id { + unsigned short vendor; + unsigned short device; +}; +static struct acs_on_id acs_on_ids[NUM_ACS_IDS]; +static u8 max_acs_id; + +static __init int pcie_acs_override_setup(char *p) +{ + if (!p) + return -EINVAL; + + while (*p) { + if (!strncmp(p, "downstream", 10)) { + acs_on_downstream = true; + pr_info("PCIe ACS bypass added for downstream\n"); + } + if (!strncmp(p, "multifunction", 13)) { + acs_on_multifunction = true; + pr_info("PCIe ACS bypass added for multifunction\n"); + } + if (!strncmp(p, "id:", 3)) { + char opt[5]; + int ret; + long val; + + if (max_acs_id >= NUM_ACS_IDS - 1) { + pr_warn("Out of PCIe ACS override slots (%d)\n", + NUM_ACS_IDS); + goto next; + } + + p += 3; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].vendor = val; + + p += strcspn(p, ":"); + if (*p != ':') { + pr_warn("PCIe ACS invalid ID\n"); + goto next; + } + + p++; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].device = val; + pr_info("PCIe ACS bypass added for %04x:%04x\n", + acs_on_ids[max_acs_id].vendor, + acs_on_ids[max_acs_id].device); + max_acs_id++; + } +next: + p += strcspn(p, ","); + if (*p == ',') + p++; + } + + if (acs_on_downstream || acs_on_multifunction || max_acs_id) + pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n"); + + return 0; +} +early_param("pcie_acs_override", pcie_acs_override_setup); + +static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags) +{ + int i; + + /* Never override ACS for legacy devices or devices with ACS caps */ + if (!pci_is_pcie(dev) /* || + pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) */) + return -ENOTTY; + + for (i = 0; i < max_acs_id; i++) { + if (acs_on_ids[i].vendor == dev->vendor && + acs_on_ids[i].device == dev->device) { + pr_info("PCIe ACS override match for %s\n", + pci_name(dev)); + return 1; + } + } + + switch (pci_pcie_type(dev)) { + case PCI_EXP_TYPE_DOWNSTREAM: + case PCI_EXP_TYPE_ROOT_PORT: + if (acs_on_downstream) { + pr_info("PCIe ACS override downstream %s\n", + pci_name(dev)); + return 1; + } + break; + case PCI_EXP_TYPE_ENDPOINT: + case PCI_EXP_TYPE_UPSTREAM: + case PCI_EXP_TYPE_LEG_END: + case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) { + pr_info("PCIe ACS override multifunction %s\n", + pci_name(dev)); + return 1; + } + } + + return -ENOTTY; +} + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; int (*acs_enabled)(struct pci_dev *dev, u16 acs_flags); } pci_dev_acs_enabled[] = { + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { PCI_VENDOR_ID_ATI, 0x4385, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x439c, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4383, pci_quirk_amd_sb_acs }, ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-04 0:03 ` Alex Williamson @ 2014-01-04 19:26 ` Dana Goyette 2014-01-04 20:22 ` Alex Williamson 0 siblings, 1 reply; 15+ messages in thread From: Dana Goyette @ 2014-01-04 19:26 UTC (permalink / raw) To: kvm On 01/03/2014 04:03 PM, Alex Williamson wrote: > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: >> On 12/29/2013 08:16 PM, Alex Williamson wrote: >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to >>>>>> soon decide which one to keep. >>>>>> >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>>>> downstream of the PLX switch to be in separate groups. >>>>>> >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" >>>>>> applied and set for the Intel root ports, the devices behind the switch >>>>>> remain stuck in the same group. >>>>>> >>>>>> In terms of passing devices to different VMs, which is better: all >>>>>> devices on different root ports, or all devices behind the one >>>>>> ACS-supporting switch? >>>>> Can you provide lspci -vvv info? If you're getting that for groups >>>>> either the switch has ACS capabilities, but doesn't support the features >>>>> we need or we're doing something wrong. Thanks, >>>>> >>>> I initially tried attaching the output as a .txt file, but it's too >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I >>>> moved the Radeon to a different slot). >>> Well, something seems amiss since the downstream switch ports all seem >>> to support and enable the correct set of ACS capabilities. I'm tending >>> to suspect something wrong with the ACS override patch or how it's being >>> used since your IOMMU group is still based at the root port. Each root >>> port is isolated from the other root ports though, so something is >>> happening with the override patch. Can you provide the kernel command >>> line you use to enable ACS overrides and the override patch you're >>> using, as it applies to 3.13-rc5? Thanks, >>> >>> Alex >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> I'm using the original acs-override patch from this post: >> https://lkml.org/lkml/2013/5/30/513 >> >> Kernel parameter is: >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 >> >> When booting a kernel without the override patch, the following devices >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire >> controller; Intel I210 Ethernet controller. > Could you please try the patch below and send dmesg for the system once > booted. This applies directly to upstream and includes the acs override > patch. Thanks, (removed patch from quote.) Here's the complete dmesg, with pcie_acs_override still set: http://pastebin.com/YHuKnrTb Most relevant section: [ 0.524362] DMAR: No ATSR found [ 0.524386] IOMMU 1 0xfed91000: using Queued invalidation [ 0.524389] IOMMU: Setting RMRR: [ 0.524398] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7bea1000 - 0x7beaffff] [ 0.524423] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7bea1000 - 0x7beaffff] [ 0.524441] IOMMU: Setting identity map for device 0000:00:14.0 [0x7bea1000 - 0x7beaffff] [ 0.524454] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.524461] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.524548] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.524551] intel_iommu_add_device(0000:00:00.0) [ 0.524552] dma_pdev #1: 0000:00:00.0 [ 0.524553] dma_pdev #2: 0000:00:00.0 [ 0.524554] dma_pdev #3: 0000:00:00.0 [ 0.524554] dma_pdev #4: 0000:00:00.0 [ 0.524565] intel_iommu_add_device(0000:00:01.0) [ 0.524566] dma_pdev #1: 0000:00:01.0 [ 0.524567] dma_pdev #2: 0000:00:01.0 [ 0.524569] pci_acs_enabled(0000:00:01.0, 001d) [ 0.524572] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 [ 0.524573] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false [ 0.524574] -> false [ 0.524575] pci_acs_enabled(0000:00:01.0, 001d) [ 0.524577] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 [ 0.524578] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false [ 0.524579] -> false [ 0.524580] dma_pdev #3: 0000:00:01.0 [ 0.524581] dma_pdev #4: 0000:00:01.0 [ 0.524584] intel_iommu_add_device(0000:00:01.1) [ 0.524586] dma_pdev #1: 0000:00:01.1 [ 0.524586] dma_pdev #2: 0000:00:01.1 [ 0.524587] pci_acs_enabled(0000:00:01.1, 001d) [ 0.524589] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 [ 0.524590] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false [ 0.524591] -> false [ 0.524592] pci_acs_enabled(0000:00:01.0, 001d) [ 0.524593] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 [ 0.524595] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false [ 0.524596] -> false [ 0.524596] dma_pdev #3: 0000:00:01.0 [ 0.524597] dma_pdev #4: 0000:00:01.0 [ 0.524599] intel_iommu_add_device(0000:00:02.0) [ 0.524601] dma_pdev #1: 0000:00:02.0 [ 0.524601] dma_pdev #2: 0000:00:02.0 [ 0.524602] dma_pdev #3: 0000:00:02.0 [ 0.524603] dma_pdev #4: 0000:00:02.0 [ 0.524606] intel_iommu_add_device(0000:00:03.0) [ 0.524607] dma_pdev #1: 0000:00:03.0 [ 0.524608] dma_pdev #2: 0000:00:03.0 [ 0.524609] dma_pdev #3: 0000:00:03.0 [ 0.524610] dma_pdev #4: 0000:00:03.0 [ 0.524613] intel_iommu_add_device(0000:00:14.0) [ 0.524614] dma_pdev #1: 0000:00:14.0 [ 0.524615] dma_pdev #2: 0000:00:14.0 [ 0.524616] dma_pdev #3: 0000:00:14.0 [ 0.524617] dma_pdev #4: 0000:00:14.0 [ 0.524620] intel_iommu_add_device(0000:00:16.0) [ 0.524621] dma_pdev #1: 0000:00:16.0 [ 0.524622] dma_pdev #2: 0000:00:16.0 [ 0.524623] pci_acs_enabled(0000:00:16.0, 001d) [ 0.524624] -> false [ 0.524625] pci_acs_enabled(0000:00:16.0, 001d) [ 0.524626] -> false [ 0.524626] dma_pdev #3: 0000:00:16.0 [ 0.524627] dma_pdev #4: 0000:00:16.0 [ 0.524630] intel_iommu_add_device(0000:00:16.3) [ 0.524631] dma_pdev #1: 0000:00:16.3 [ 0.524632] dma_pdev #2: 0000:00:16.3 [ 0.524633] pci_acs_enabled(0000:00:16.3, 001d) [ 0.524634] -> false [ 0.524635] pci_acs_enabled(0000:00:16.0, 001d) [ 0.524636] -> false [ 0.524637] dma_pdev #3: 0000:00:16.0 [ 0.524638] dma_pdev #4: 0000:00:16.0 [ 0.524640] intel_iommu_add_device(0000:00:19.0) [ 0.524641] dma_pdev #1: 0000:00:19.0 [ 0.524642] dma_pdev #2: 0000:00:19.0 [ 0.524643] dma_pdev #3: 0000:00:19.0 [ 0.524643] dma_pdev #4: 0000:00:19.0 [ 0.524646] intel_iommu_add_device(0000:00:1a.0) [ 0.524647] dma_pdev #1: 0000:00:1a.0 [ 0.524648] dma_pdev #2: 0000:00:1a.0 [ 0.524649] dma_pdev #3: 0000:00:1a.0 [ 0.524650] dma_pdev #4: 0000:00:1a.0 [ 0.524654] intel_iommu_add_device(0000:00:1b.0) [ 0.524655] dma_pdev #1: 0000:00:1b.0 [ 0.524656] dma_pdev #2: 0000:00:1b.0 [ 0.524657] dma_pdev #3: 0000:00:1b.0 [ 0.524658] dma_pdev #4: 0000:00:1b.0 [ 0.524661] intel_iommu_add_device(0000:00:1c.0) [ 0.524662] dma_pdev #1: 0000:00:1c.0 [ 0.524663] dma_pdev #2: 0000:00:1c.0 [ 0.524664] pci_acs_enabled(0000:00:1c.0, 001d) [ 0.524665] PCIe ACS override match for 0000:00:1c.0 [ 0.524666] -> true [ 0.524667] dma_pdev #3: 0000:00:1c.0 [ 0.524668] dma_pdev #4: 0000:00:1c.0 [ 0.524671] intel_iommu_add_device(0000:00:1c.1) [ 0.524672] dma_pdev #1: 0000:00:1c.1 [ 0.524673] dma_pdev #2: 0000:00:1c.1 [ 0.524674] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524681] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524682] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524683] -> false [ 0.524684] pci_acs_enabled(0000:00:1c.0, 001d) [ 0.524685] PCIe ACS override match for 0000:00:1c.0 [ 0.524686] -> true [ 0.524687] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524694] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524695] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524696] -> false [ 0.524697] dma_pdev #3: 0000:00:1c.1 [ 0.524698] dma_pdev #4: 0000:00:1c.1 [ 0.524700] intel_iommu_add_device(0000:00:1c.3) [ 0.524701] dma_pdev #1: 0000:00:1c.3 [ 0.524702] dma_pdev #2: 0000:00:1c.3 [ 0.524703] pci_acs_enabled(0000:00:1c.3, 001d) [ 0.524704] PCIe ACS override match for 0000:00:1c.3 [ 0.524705] -> true [ 0.524706] dma_pdev #3: 0000:00:1c.3 [ 0.524707] dma_pdev #4: 0000:00:1c.3 [ 0.524710] intel_iommu_add_device(0000:00:1c.4) [ 0.524711] dma_pdev #1: 0000:00:1c.4 [ 0.524712] dma_pdev #2: 0000:00:1c.4 [ 0.524713] pci_acs_enabled(0000:00:1c.4, 001d) [ 0.524714] PCIe ACS override match for 0000:00:1c.4 [ 0.524715] -> true [ 0.524715] dma_pdev #3: 0000:00:1c.4 [ 0.524716] dma_pdev #4: 0000:00:1c.4 [ 0.524719] intel_iommu_add_device(0000:00:1d.0) [ 0.524720] dma_pdev #1: 0000:00:1d.0 [ 0.524721] dma_pdev #2: 0000:00:1d.0 [ 0.524722] dma_pdev #3: 0000:00:1d.0 [ 0.524723] dma_pdev #4: 0000:00:1d.0 [ 0.524726] intel_iommu_add_device(0000:00:1f.0) [ 0.524727] dma_pdev #1: 0000:00:1f.0 [ 0.524728] dma_pdev #2: 0000:00:1f.0 [ 0.524728] pci_acs_enabled(0000:00:1f.0, 001d) [ 0.524729] -> false [ 0.524730] pci_acs_enabled(0000:00:1f.0, 001d) [ 0.524731] -> false [ 0.524732] dma_pdev #3: 0000:00:1f.0 [ 0.524733] dma_pdev #4: 0000:00:1f.0 [ 0.524736] intel_iommu_add_device(0000:00:1f.2) [ 0.524737] dma_pdev #1: 0000:00:1f.2 [ 0.524738] dma_pdev #2: 0000:00:1f.2 [ 0.524739] pci_acs_enabled(0000:00:1f.2, 001d) [ 0.524740] -> false [ 0.524741] pci_acs_enabled(0000:00:1f.0, 001d) [ 0.524741] -> false [ 0.524742] dma_pdev #3: 0000:00:1f.0 [ 0.524743] dma_pdev #4: 0000:00:1f.0 [ 0.524745] intel_iommu_add_device(0000:00:1f.3) [ 0.524746] dma_pdev #1: 0000:00:1f.3 [ 0.524747] dma_pdev #2: 0000:00:1f.3 [ 0.524748] pci_acs_enabled(0000:00:1f.3, 001d) [ 0.524749] -> false [ 0.524750] pci_acs_enabled(0000:00:1f.0, 001d) [ 0.524751] -> false [ 0.524752] dma_pdev #3: 0000:00:1f.0 [ 0.524752] dma_pdev #4: 0000:00:1f.0 [ 0.524754] intel_iommu_add_device(0000:00:1f.6) [ 0.524755] dma_pdev #1: 0000:00:1f.6 [ 0.524756] dma_pdev #2: 0000:00:1f.6 [ 0.524757] pci_acs_enabled(0000:00:1f.6, 001d) [ 0.524758] -> false [ 0.524759] pci_acs_enabled(0000:00:1f.0, 001d) [ 0.524760] -> false [ 0.524761] dma_pdev #3: 0000:00:1f.0 [ 0.524762] dma_pdev #4: 0000:00:1f.0 [ 0.524765] intel_iommu_add_device(0000:01:00.0) [ 0.524766] dma_pdev #1: 0000:01:00.0 [ 0.524767] dma_pdev #2: 0000:01:00.0 [ 0.524768] pci_acs_enabled(0000:01:00.0, 001d) [ 0.524771] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 [ 0.524772] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false [ 0.524773] -> false [ 0.524774] pci_acs_enabled(0000:01:00.0, 001d) [ 0.524776] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 [ 0.524777] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false [ 0.524778] -> false [ 0.524779] dma_pdev #3: 0000:01:00.0 [ 0.524780] pci_acs_path_enabled(0000:00:01.0, NULL, 001d) [ 0.524781] pci_acs_enabled(0000:00:01.0, 001d) [ 0.524783] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 [ 0.524784] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false [ 0.524785] -> false [ 0.524786] -> false [ 0.524787] dma_pdev #4: 0000:00:01.0 [ 0.524789] intel_iommu_add_device(0000:01:00.1) [ 0.524790] dma_pdev #1: 0000:01:00.1 [ 0.524791] dma_pdev #2: 0000:01:00.1 [ 0.524792] pci_acs_enabled(0000:01:00.1, 001d) [ 0.524794] pci_acs_flags_enabled no ACS capability on 0000:01:00.1 [ 0.524795] pci_acs_flags_enabled(0000:01:00.1, 001d) -> false [ 0.524796] -> false [ 0.524797] pci_acs_enabled(0000:01:00.0, 001d) [ 0.524799] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 [ 0.524800] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false [ 0.524801] -> false [ 0.524802] dma_pdev #3: 0000:01:00.0 [ 0.524803] pci_acs_path_enabled(0000:00:01.0, NULL, 001d) [ 0.524804] pci_acs_enabled(0000:00:01.0, 001d) [ 0.524806] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 [ 0.524807] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false [ 0.524808] -> false [ 0.524809] -> false [ 0.524809] dma_pdev #4: 0000:00:01.0 [ 0.524811] intel_iommu_add_device(0000:02:00.0) [ 0.524813] dma_pdev #1: 0000:02:00.0 [ 0.524813] dma_pdev #2: 0000:02:00.0 [ 0.524814] dma_pdev #3: 0000:02:00.0 [ 0.524815] pci_acs_path_enabled(0000:00:01.1, NULL, 001d) [ 0.524816] pci_acs_enabled(0000:00:01.1, 001d) [ 0.524818] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 [ 0.524819] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false [ 0.524820] -> false [ 0.524821] -> false [ 0.524822] dma_pdev #4: 0000:00:01.1 [ 0.524824] intel_iommu_add_device(0000:03:04.0) [ 0.524825] Upstream bridge 0000:02:00.0 [ 0.524830] dma_pdev #1: 0000:02:00.0 [ 0.524831] dma_pdev #2: 0000:02:00.0 [ 0.524832] dma_pdev #3: 0000:02:00.0 [ 0.524833] pci_acs_path_enabled(0000:00:01.1, NULL, 001d) [ 0.524834] pci_acs_enabled(0000:00:01.1, 001d) [ 0.524835] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 [ 0.524837] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false [ 0.524837] -> false [ 0.524838] -> false [ 0.524839] dma_pdev #4: 0000:00:01.1 [ 0.524841] intel_iommu_add_device(0000:04:00.0) [ 0.524842] dma_pdev #1: 0000:04:00.0 [ 0.524843] dma_pdev #2: 0000:04:00.0 [ 0.524844] dma_pdev #3: 0000:04:00.0 [ 0.524845] pci_acs_path_enabled(0000:00:1c.0, NULL, 001d) [ 0.524846] pci_acs_enabled(0000:00:1c.0, 001d) [ 0.524847] PCIe ACS override match for 0000:00:1c.0 [ 0.524848] -> true [ 0.524849] -> true [ 0.524849] dma_pdev #4: 0000:04:00.0 [ 0.524852] intel_iommu_add_device(0000:05:00.0) [ 0.524853] dma_pdev #1: 0000:05:00.0 [ 0.524854] dma_pdev #2: 0000:05:00.0 [ 0.524855] dma_pdev #3: 0000:05:00.0 [ 0.524856] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.524857] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524865] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524866] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524867] -> false [ 0.524868] -> false [ 0.524869] dma_pdev #4: 0000:00:1c.1 [ 0.524871] intel_iommu_add_device(0000:06:01.0) [ 0.524872] dma_pdev #1: 0000:06:01.0 [ 0.524872] dma_pdev #2: 0000:06:01.0 [ 0.524873] dma_pdev #3: 0000:06:01.0 [ 0.524874] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.524875] pci_acs_enabled(0000:05:00.0, 001d) [ 0.524876] PCIe ACS override match for 0000:05:00.0 [ 0.524877] -> true [ 0.524878] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524886] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524887] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524888] -> false [ 0.524888] -> false [ 0.524889] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.524891] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524898] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524899] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524900] -> false [ 0.524901] -> false [ 0.524901] dma_pdev #4: 0000:00:1c.1 [ 0.524903] intel_iommu_add_device(0000:06:04.0) [ 0.524904] dma_pdev #1: 0000:06:04.0 [ 0.524905] dma_pdev #2: 0000:06:04.0 [ 0.524906] dma_pdev #3: 0000:06:04.0 [ 0.524907] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.524908] pci_acs_enabled(0000:05:00.0, 001d) [ 0.524909] PCIe ACS override match for 0000:05:00.0 [ 0.524910] -> true [ 0.524911] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524913] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524914] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524915] -> false [ 0.524916] -> false [ 0.524917] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.524918] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524925] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524926] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524927] -> false [ 0.524928] -> false [ 0.524929] dma_pdev #4: 0000:00:1c.1 [ 0.524949] intel_iommu_add_device(0000:06:05.0) [ 0.524951] dma_pdev #1: 0000:06:05.0 [ 0.524951] dma_pdev #2: 0000:06:05.0 [ 0.524952] dma_pdev #3: 0000:06:05.0 [ 0.524953] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.524954] pci_acs_enabled(0000:05:00.0, 001d) [ 0.524955] PCIe ACS override match for 0000:05:00.0 [ 0.524957] -> true [ 0.524958] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524964] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524965] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524966] -> false [ 0.524967] -> false [ 0.524968] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.524969] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524976] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524977] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524978] -> false [ 0.524979] -> false [ 0.524979] dma_pdev #4: 0000:00:1c.1 [ 0.524983] intel_iommu_add_device(0000:06:07.0) [ 0.524984] dma_pdev #1: 0000:06:07.0 [ 0.524985] dma_pdev #2: 0000:06:07.0 [ 0.524986] dma_pdev #3: 0000:06:07.0 [ 0.524987] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.524988] pci_acs_enabled(0000:05:00.0, 001d) [ 0.524989] PCIe ACS override match for 0000:05:00.0 [ 0.524990] -> true [ 0.524991] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.524997] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.524998] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.524999] -> false [ 0.525000] -> false [ 0.525001] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.525002] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525004] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525006] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525007] -> false [ 0.525007] -> false [ 0.525008] dma_pdev #4: 0000:00:1c.1 [ 0.525010] intel_iommu_add_device(0000:06:09.0) [ 0.525011] dma_pdev #1: 0000:06:09.0 [ 0.525012] dma_pdev #2: 0000:06:09.0 [ 0.525013] dma_pdev #3: 0000:06:09.0 [ 0.525014] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.525015] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525016] PCIe ACS override match for 0000:05:00.0 [ 0.525017] -> true [ 0.525018] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525025] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525026] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525027] -> false [ 0.525028] -> false [ 0.525029] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.525030] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525037] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525038] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525039] -> false [ 0.525040] -> false [ 0.525041] dma_pdev #4: 0000:00:1c.1 [ 0.525043] intel_iommu_add_device(0000:0a:00.0) [ 0.525044] dma_pdev #1: 0000:0a:00.0 [ 0.525044] dma_pdev #2: 0000:0a:00.0 [ 0.525045] dma_pdev #3: 0000:0a:00.0 [ 0.525046] pci_acs_path_enabled(0000:06:07.0, NULL, 001d) [ 0.525048] pci_acs_enabled(0000:06:07.0, 001d) [ 0.525048] PCIe ACS override match for 0000:06:07.0 [ 0.525049] -> true [ 0.525050] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525051] PCIe ACS override match for 0000:05:00.0 [ 0.525052] -> true [ 0.525053] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525060] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525062] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525062] -> false [ 0.525063] -> false [ 0.525064] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.525065] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525066] PCIe ACS override match for 0000:05:00.0 [ 0.525067] -> true [ 0.525068] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525075] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525076] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525077] -> false [ 0.525078] -> false [ 0.525079] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.525080] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525087] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525088] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525089] -> false [ 0.525090] -> false [ 0.525091] dma_pdev #4: 0000:00:1c.1 [ 0.525093] intel_iommu_add_device(0000:0b:00.0) [ 0.525094] dma_pdev #1: 0000:0b:00.0 [ 0.525095] dma_pdev #2: 0000:0b:00.0 [ 0.525096] dma_pdev #3: 0000:0b:00.0 [ 0.525097] pci_acs_path_enabled(0000:06:09.0, NULL, 001d) [ 0.525098] pci_acs_enabled(0000:06:09.0, 001d) [ 0.525099] PCIe ACS override match for 0000:06:09.0 [ 0.525100] -> true [ 0.525101] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525102] PCIe ACS override match for 0000:05:00.0 [ 0.525103] -> true [ 0.525103] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525111] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525112] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525113] -> false [ 0.525114] -> false [ 0.525115] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.525116] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525117] PCIe ACS override match for 0000:05:00.0 [ 0.525118] -> true [ 0.525118] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525125] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525126] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525127] -> false [ 0.525128] -> false [ 0.525129] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.525130] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525137] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525138] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525139] -> false [ 0.525140] -> false [ 0.525141] dma_pdev #4: 0000:00:1c.1 [ 0.525143] intel_iommu_add_device(0000:0c:00.0) [ 0.525144] Upstream bridge 0000:0b:00.0 [ 0.525148] dma_pdev #1: 0000:0c:00.0 [ 0.525149] dma_pdev #2: 0000:0c:00.0 [ 0.525150] dma_pdev #3: 0000:0c:00.0 [ 0.525151] pci_acs_path_enabled(0000:0b:00.0, NULL, 001d) [ 0.525152] pci_acs_enabled(0000:0b:00.0, 001d) [ 0.525153] -> false [ 0.525154] -> false [ 0.525155] pci_acs_path_enabled(0000:06:09.0, NULL, 001d) [ 0.525156] pci_acs_enabled(0000:06:09.0, 001d) [ 0.525157] PCIe ACS override match for 0000:06:09.0 [ 0.525158] -> true [ 0.525158] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525159] PCIe ACS override match for 0000:05:00.0 [ 0.525160] -> true [ 0.525161] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525168] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525169] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525170] -> false [ 0.525171] -> false [ 0.525172] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) [ 0.525173] pci_acs_enabled(0000:05:00.0, 001d) [ 0.525174] PCIe ACS override match for 0000:05:00.0 [ 0.525175] -> true [ 0.525175] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525178] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525179] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525180] -> false [ 0.525181] -> false [ 0.525182] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) [ 0.525183] pci_acs_enabled(0000:00:1c.1, 001d) [ 0.525190] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 [ 0.525191] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false [ 0.525192] -> false [ 0.525193] -> false [ 0.525193] dma_pdev #4: 0000:00:1c.1 [ 0.525195] intel_iommu_add_device(0000:0d:00.0) [ 0.525196] dma_pdev #1: 0000:0d:00.0 [ 0.525197] dma_pdev #2: 0000:0d:00.0 [ 0.525198] dma_pdev #3: 0000:0d:00.0 [ 0.525199] pci_acs_path_enabled(0000:00:1c.3, NULL, 001d) [ 0.525200] pci_acs_enabled(0000:00:1c.3, 001d) [ 0.525201] PCIe ACS override match for 0000:00:1c.3 [ 0.525202] -> true [ 0.525203] -> true [ 0.525204] dma_pdev #4: 0000:0d:00.0 lspci as slots are now (since Radeon is back in first x16 slot): 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller [8086:8c3d] (rev 04) 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 05) 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] 02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 03:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 0b:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) 0c:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] [104c:823f] (rev 01) 0d:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-04 19:26 ` Dana Goyette @ 2014-01-04 20:22 ` Alex Williamson 2014-01-04 21:11 ` Dana Goyette 0 siblings, 1 reply; 15+ messages in thread From: Alex Williamson @ 2014-01-04 20:22 UTC (permalink / raw) To: Dana Goyette; +Cc: kvm On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote: > On 01/03/2014 04:03 PM, Alex Williamson wrote: > > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: > >> On 12/29/2013 08:16 PM, Alex Williamson wrote: > >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: > >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: > >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: > >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to > >>>>>> soon decide which one to keep. > >>>>>> > >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX > >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices > >>>>>> downstream of the PLX switch to be in separate groups. > >>>>>> > >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" > >>>>>> applied and set for the Intel root ports, the devices behind the switch > >>>>>> remain stuck in the same group. > >>>>>> > >>>>>> In terms of passing devices to different VMs, which is better: all > >>>>>> devices on different root ports, or all devices behind the one > >>>>>> ACS-supporting switch? > >>>>> Can you provide lspci -vvv info? If you're getting that for groups > >>>>> either the switch has ACS capabilities, but doesn't support the features > >>>>> we need or we're doing something wrong. Thanks, > >>>>> > >>>> I initially tried attaching the output as a .txt file, but it's too > >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I > >>>> moved the Radeon to a different slot). > >>> Well, something seems amiss since the downstream switch ports all seem > >>> to support and enable the correct set of ACS capabilities. I'm tending > >>> to suspect something wrong with the ACS override patch or how it's being > >>> used since your IOMMU group is still based at the root port. Each root > >>> port is isolated from the other root ports though, so something is > >>> happening with the override patch. Can you provide the kernel command > >>> line you use to enable ACS overrides and the override patch you're > >>> using, as it applies to 3.13-rc5? Thanks, > >>> > >>> Alex > >>> > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe kvm" in > >>> the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> > >> I'm using the original acs-override patch from this post: > >> https://lkml.org/lkml/2013/5/30/513 > >> > >> Kernel parameter is: > >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 Actually, you're not: pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 And we register all of them: [ 0.000000] PCIe ACS bypass added for 8086:8c10 [ 0.000000] PCIe ACS bypass added for 8086:8c16 [ 0.000000] PCIe ACS bypass added for 8086:8c18 [ 0.000000] PCIe ACS bypass added for 8086:ac1a [ 0.000000] PCIe ACS bypass added for 8086:8c1c [ 0.000000] PCIe ACS bypass added for 8086:8c1e [ 0.000000] PCIe ACS bypass added for 10b5:8606 However, note that the root port causing you trouble is 8086:8c12, which isn't provided as an override, therefore the code is doing the right thing and grouping all devices behind that root port together. > >> > >> When booting a kernel without the override patch, the following devices > >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA > >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire > >> controller; Intel I210 Ethernet controller. > > Could you please try the patch below and send dmesg for the system once > > booted. This applies directly to upstream and includes the acs override > > patch. Thanks, > > (removed patch from quote.) > > Here's the complete dmesg, with pcie_acs_override still set: > > http://pastebin.com/YHuKnrTb > > Most relevant section: > > [ 0.524362] DMAR: No ATSR found > [ 0.524386] IOMMU 1 0xfed91000: using Queued invalidation > [ 0.524389] IOMMU: Setting RMRR: > [ 0.524398] IOMMU: Setting identity map for device 0000:00:1d.0 > [0x7bea1000 - 0x7beaffff] > [ 0.524423] IOMMU: Setting identity map for device 0000:00:1a.0 > [0x7bea1000 - 0x7beaffff] > [ 0.524441] IOMMU: Setting identity map for device 0000:00:14.0 > [0x7bea1000 - 0x7beaffff] > [ 0.524454] IOMMU: Prepare 0-16MiB unity mapping for LPC > [ 0.524461] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 > - 0xffffff] > [ 0.524548] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O > [ 0.524551] intel_iommu_add_device(0000:00:00.0) > [ 0.524552] dma_pdev #1: 0000:00:00.0 > [ 0.524553] dma_pdev #2: 0000:00:00.0 > [ 0.524554] dma_pdev #3: 0000:00:00.0 > [ 0.524554] dma_pdev #4: 0000:00:00.0 > [ 0.524565] intel_iommu_add_device(0000:00:01.0) > [ 0.524566] dma_pdev #1: 0000:00:01.0 > [ 0.524567] dma_pdev #2: 0000:00:01.0 > [ 0.524569] pci_acs_enabled(0000:00:01.0, 001d) > [ 0.524572] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 > [ 0.524573] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false > [ 0.524574] -> false > [ 0.524575] pci_acs_enabled(0000:00:01.0, 001d) > [ 0.524577] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 > [ 0.524578] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false > [ 0.524579] -> false > [ 0.524580] dma_pdev #3: 0000:00:01.0 > [ 0.524581] dma_pdev #4: 0000:00:01.0 > [ 0.524584] intel_iommu_add_device(0000:00:01.1) > [ 0.524586] dma_pdev #1: 0000:00:01.1 > [ 0.524586] dma_pdev #2: 0000:00:01.1 > [ 0.524587] pci_acs_enabled(0000:00:01.1, 001d) > [ 0.524589] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 > [ 0.524590] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false > [ 0.524591] -> false > [ 0.524592] pci_acs_enabled(0000:00:01.0, 001d) > [ 0.524593] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 > [ 0.524595] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false > [ 0.524596] -> false > [ 0.524596] dma_pdev #3: 0000:00:01.0 > [ 0.524597] dma_pdev #4: 0000:00:01.0 > [ 0.524599] intel_iommu_add_device(0000:00:02.0) > [ 0.524601] dma_pdev #1: 0000:00:02.0 > [ 0.524601] dma_pdev #2: 0000:00:02.0 > [ 0.524602] dma_pdev #3: 0000:00:02.0 > [ 0.524603] dma_pdev #4: 0000:00:02.0 > [ 0.524606] intel_iommu_add_device(0000:00:03.0) > [ 0.524607] dma_pdev #1: 0000:00:03.0 > [ 0.524608] dma_pdev #2: 0000:00:03.0 > [ 0.524609] dma_pdev #3: 0000:00:03.0 > [ 0.524610] dma_pdev #4: 0000:00:03.0 > [ 0.524613] intel_iommu_add_device(0000:00:14.0) > [ 0.524614] dma_pdev #1: 0000:00:14.0 > [ 0.524615] dma_pdev #2: 0000:00:14.0 > [ 0.524616] dma_pdev #3: 0000:00:14.0 > [ 0.524617] dma_pdev #4: 0000:00:14.0 > [ 0.524620] intel_iommu_add_device(0000:00:16.0) > [ 0.524621] dma_pdev #1: 0000:00:16.0 > [ 0.524622] dma_pdev #2: 0000:00:16.0 > [ 0.524623] pci_acs_enabled(0000:00:16.0, 001d) > [ 0.524624] -> false > [ 0.524625] pci_acs_enabled(0000:00:16.0, 001d) > [ 0.524626] -> false > [ 0.524626] dma_pdev #3: 0000:00:16.0 > [ 0.524627] dma_pdev #4: 0000:00:16.0 > [ 0.524630] intel_iommu_add_device(0000:00:16.3) > [ 0.524631] dma_pdev #1: 0000:00:16.3 > [ 0.524632] dma_pdev #2: 0000:00:16.3 > [ 0.524633] pci_acs_enabled(0000:00:16.3, 001d) > [ 0.524634] -> false > [ 0.524635] pci_acs_enabled(0000:00:16.0, 001d) > [ 0.524636] -> false > [ 0.524637] dma_pdev #3: 0000:00:16.0 > [ 0.524638] dma_pdev #4: 0000:00:16.0 > [ 0.524640] intel_iommu_add_device(0000:00:19.0) > [ 0.524641] dma_pdev #1: 0000:00:19.0 > [ 0.524642] dma_pdev #2: 0000:00:19.0 > [ 0.524643] dma_pdev #3: 0000:00:19.0 > [ 0.524643] dma_pdev #4: 0000:00:19.0 > [ 0.524646] intel_iommu_add_device(0000:00:1a.0) > [ 0.524647] dma_pdev #1: 0000:00:1a.0 > [ 0.524648] dma_pdev #2: 0000:00:1a.0 > [ 0.524649] dma_pdev #3: 0000:00:1a.0 > [ 0.524650] dma_pdev #4: 0000:00:1a.0 > [ 0.524654] intel_iommu_add_device(0000:00:1b.0) > [ 0.524655] dma_pdev #1: 0000:00:1b.0 > [ 0.524656] dma_pdev #2: 0000:00:1b.0 > [ 0.524657] dma_pdev #3: 0000:00:1b.0 > [ 0.524658] dma_pdev #4: 0000:00:1b.0 > [ 0.524661] intel_iommu_add_device(0000:00:1c.0) > [ 0.524662] dma_pdev #1: 0000:00:1c.0 > [ 0.524663] dma_pdev #2: 0000:00:1c.0 > [ 0.524664] pci_acs_enabled(0000:00:1c.0, 001d) > [ 0.524665] PCIe ACS override match for 0000:00:1c.0 > [ 0.524666] -> true > [ 0.524667] dma_pdev #3: 0000:00:1c.0 > [ 0.524668] dma_pdev #4: 0000:00:1c.0 > [ 0.524671] intel_iommu_add_device(0000:00:1c.1) > [ 0.524672] dma_pdev #1: 0000:00:1c.1 > [ 0.524673] dma_pdev #2: 0000:00:1c.1 > [ 0.524674] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524681] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524682] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524683] -> false > [ 0.524684] pci_acs_enabled(0000:00:1c.0, 001d) > [ 0.524685] PCIe ACS override match for 0000:00:1c.0 > [ 0.524686] -> true > [ 0.524687] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524694] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524695] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524696] -> false > [ 0.524697] dma_pdev #3: 0000:00:1c.1 > [ 0.524698] dma_pdev #4: 0000:00:1c.1 > [ 0.524700] intel_iommu_add_device(0000:00:1c.3) > [ 0.524701] dma_pdev #1: 0000:00:1c.3 > [ 0.524702] dma_pdev #2: 0000:00:1c.3 > [ 0.524703] pci_acs_enabled(0000:00:1c.3, 001d) > [ 0.524704] PCIe ACS override match for 0000:00:1c.3 > [ 0.524705] -> true > [ 0.524706] dma_pdev #3: 0000:00:1c.3 > [ 0.524707] dma_pdev #4: 0000:00:1c.3 > [ 0.524710] intel_iommu_add_device(0000:00:1c.4) > [ 0.524711] dma_pdev #1: 0000:00:1c.4 > [ 0.524712] dma_pdev #2: 0000:00:1c.4 > [ 0.524713] pci_acs_enabled(0000:00:1c.4, 001d) > [ 0.524714] PCIe ACS override match for 0000:00:1c.4 > [ 0.524715] -> true > [ 0.524715] dma_pdev #3: 0000:00:1c.4 > [ 0.524716] dma_pdev #4: 0000:00:1c.4 > [ 0.524719] intel_iommu_add_device(0000:00:1d.0) > [ 0.524720] dma_pdev #1: 0000:00:1d.0 > [ 0.524721] dma_pdev #2: 0000:00:1d.0 > [ 0.524722] dma_pdev #3: 0000:00:1d.0 > [ 0.524723] dma_pdev #4: 0000:00:1d.0 > [ 0.524726] intel_iommu_add_device(0000:00:1f.0) > [ 0.524727] dma_pdev #1: 0000:00:1f.0 > [ 0.524728] dma_pdev #2: 0000:00:1f.0 > [ 0.524728] pci_acs_enabled(0000:00:1f.0, 001d) > [ 0.524729] -> false > [ 0.524730] pci_acs_enabled(0000:00:1f.0, 001d) > [ 0.524731] -> false > [ 0.524732] dma_pdev #3: 0000:00:1f.0 > [ 0.524733] dma_pdev #4: 0000:00:1f.0 > [ 0.524736] intel_iommu_add_device(0000:00:1f.2) > [ 0.524737] dma_pdev #1: 0000:00:1f.2 > [ 0.524738] dma_pdev #2: 0000:00:1f.2 > [ 0.524739] pci_acs_enabled(0000:00:1f.2, 001d) > [ 0.524740] -> false > [ 0.524741] pci_acs_enabled(0000:00:1f.0, 001d) > [ 0.524741] -> false > [ 0.524742] dma_pdev #3: 0000:00:1f.0 > [ 0.524743] dma_pdev #4: 0000:00:1f.0 > [ 0.524745] intel_iommu_add_device(0000:00:1f.3) > [ 0.524746] dma_pdev #1: 0000:00:1f.3 > [ 0.524747] dma_pdev #2: 0000:00:1f.3 > [ 0.524748] pci_acs_enabled(0000:00:1f.3, 001d) > [ 0.524749] -> false > [ 0.524750] pci_acs_enabled(0000:00:1f.0, 001d) > [ 0.524751] -> false > [ 0.524752] dma_pdev #3: 0000:00:1f.0 > [ 0.524752] dma_pdev #4: 0000:00:1f.0 > [ 0.524754] intel_iommu_add_device(0000:00:1f.6) > [ 0.524755] dma_pdev #1: 0000:00:1f.6 > [ 0.524756] dma_pdev #2: 0000:00:1f.6 > [ 0.524757] pci_acs_enabled(0000:00:1f.6, 001d) > [ 0.524758] -> false > [ 0.524759] pci_acs_enabled(0000:00:1f.0, 001d) > [ 0.524760] -> false > [ 0.524761] dma_pdev #3: 0000:00:1f.0 > [ 0.524762] dma_pdev #4: 0000:00:1f.0 > [ 0.524765] intel_iommu_add_device(0000:01:00.0) > [ 0.524766] dma_pdev #1: 0000:01:00.0 > [ 0.524767] dma_pdev #2: 0000:01:00.0 > [ 0.524768] pci_acs_enabled(0000:01:00.0, 001d) > [ 0.524771] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 > [ 0.524772] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false > [ 0.524773] -> false > [ 0.524774] pci_acs_enabled(0000:01:00.0, 001d) > [ 0.524776] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 > [ 0.524777] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false > [ 0.524778] -> false > [ 0.524779] dma_pdev #3: 0000:01:00.0 > [ 0.524780] pci_acs_path_enabled(0000:00:01.0, NULL, 001d) > [ 0.524781] pci_acs_enabled(0000:00:01.0, 001d) > [ 0.524783] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 > [ 0.524784] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false > [ 0.524785] -> false > [ 0.524786] -> false > [ 0.524787] dma_pdev #4: 0000:00:01.0 > [ 0.524789] intel_iommu_add_device(0000:01:00.1) > [ 0.524790] dma_pdev #1: 0000:01:00.1 > [ 0.524791] dma_pdev #2: 0000:01:00.1 > [ 0.524792] pci_acs_enabled(0000:01:00.1, 001d) > [ 0.524794] pci_acs_flags_enabled no ACS capability on 0000:01:00.1 > [ 0.524795] pci_acs_flags_enabled(0000:01:00.1, 001d) -> false > [ 0.524796] -> false > [ 0.524797] pci_acs_enabled(0000:01:00.0, 001d) > [ 0.524799] pci_acs_flags_enabled no ACS capability on 0000:01:00.0 > [ 0.524800] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false > [ 0.524801] -> false > [ 0.524802] dma_pdev #3: 0000:01:00.0 > [ 0.524803] pci_acs_path_enabled(0000:00:01.0, NULL, 001d) > [ 0.524804] pci_acs_enabled(0000:00:01.0, 001d) > [ 0.524806] pci_acs_flags_enabled no ACS capability on 0000:00:01.0 > [ 0.524807] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false > [ 0.524808] -> false > [ 0.524809] -> false > [ 0.524809] dma_pdev #4: 0000:00:01.0 > [ 0.524811] intel_iommu_add_device(0000:02:00.0) > [ 0.524813] dma_pdev #1: 0000:02:00.0 > [ 0.524813] dma_pdev #2: 0000:02:00.0 > [ 0.524814] dma_pdev #3: 0000:02:00.0 > [ 0.524815] pci_acs_path_enabled(0000:00:01.1, NULL, 001d) > [ 0.524816] pci_acs_enabled(0000:00:01.1, 001d) > [ 0.524818] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 > [ 0.524819] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false > [ 0.524820] -> false > [ 0.524821] -> false > [ 0.524822] dma_pdev #4: 0000:00:01.1 > [ 0.524824] intel_iommu_add_device(0000:03:04.0) > [ 0.524825] Upstream bridge 0000:02:00.0 > [ 0.524830] dma_pdev #1: 0000:02:00.0 > [ 0.524831] dma_pdev #2: 0000:02:00.0 > [ 0.524832] dma_pdev #3: 0000:02:00.0 > [ 0.524833] pci_acs_path_enabled(0000:00:01.1, NULL, 001d) > [ 0.524834] pci_acs_enabled(0000:00:01.1, 001d) > [ 0.524835] pci_acs_flags_enabled no ACS capability on 0000:00:01.1 > [ 0.524837] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false > [ 0.524837] -> false > [ 0.524838] -> false > [ 0.524839] dma_pdev #4: 0000:00:01.1 > [ 0.524841] intel_iommu_add_device(0000:04:00.0) > [ 0.524842] dma_pdev #1: 0000:04:00.0 > [ 0.524843] dma_pdev #2: 0000:04:00.0 > [ 0.524844] dma_pdev #3: 0000:04:00.0 > [ 0.524845] pci_acs_path_enabled(0000:00:1c.0, NULL, 001d) > [ 0.524846] pci_acs_enabled(0000:00:1c.0, 001d) > [ 0.524847] PCIe ACS override match for 0000:00:1c.0 > [ 0.524848] -> true > [ 0.524849] -> true > [ 0.524849] dma_pdev #4: 0000:04:00.0 > [ 0.524852] intel_iommu_add_device(0000:05:00.0) > [ 0.524853] dma_pdev #1: 0000:05:00.0 > [ 0.524854] dma_pdev #2: 0000:05:00.0 > [ 0.524855] dma_pdev #3: 0000:05:00.0 > [ 0.524856] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.524857] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524865] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524866] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524867] -> false > [ 0.524868] -> false > [ 0.524869] dma_pdev #4: 0000:00:1c.1 > [ 0.524871] intel_iommu_add_device(0000:06:01.0) > [ 0.524872] dma_pdev #1: 0000:06:01.0 > [ 0.524872] dma_pdev #2: 0000:06:01.0 > [ 0.524873] dma_pdev #3: 0000:06:01.0 > [ 0.524874] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.524875] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.524876] PCIe ACS override match for 0000:05:00.0 > [ 0.524877] -> true > [ 0.524878] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524886] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524887] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524888] -> false > [ 0.524888] -> false > [ 0.524889] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.524891] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524898] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524899] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524900] -> false > [ 0.524901] -> false > [ 0.524901] dma_pdev #4: 0000:00:1c.1 > [ 0.524903] intel_iommu_add_device(0000:06:04.0) > [ 0.524904] dma_pdev #1: 0000:06:04.0 > [ 0.524905] dma_pdev #2: 0000:06:04.0 > [ 0.524906] dma_pdev #3: 0000:06:04.0 > [ 0.524907] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.524908] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.524909] PCIe ACS override match for 0000:05:00.0 > [ 0.524910] -> true > [ 0.524911] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524913] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524914] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524915] -> false > [ 0.524916] -> false > [ 0.524917] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.524918] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524925] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524926] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524927] -> false > [ 0.524928] -> false > [ 0.524929] dma_pdev #4: 0000:00:1c.1 > [ 0.524949] intel_iommu_add_device(0000:06:05.0) > [ 0.524951] dma_pdev #1: 0000:06:05.0 > [ 0.524951] dma_pdev #2: 0000:06:05.0 > [ 0.524952] dma_pdev #3: 0000:06:05.0 > [ 0.524953] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.524954] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.524955] PCIe ACS override match for 0000:05:00.0 > [ 0.524957] -> true > [ 0.524958] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524964] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524965] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524966] -> false > [ 0.524967] -> false > [ 0.524968] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.524969] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524976] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524977] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524978] -> false > [ 0.524979] -> false > [ 0.524979] dma_pdev #4: 0000:00:1c.1 > [ 0.524983] intel_iommu_add_device(0000:06:07.0) > [ 0.524984] dma_pdev #1: 0000:06:07.0 > [ 0.524985] dma_pdev #2: 0000:06:07.0 > [ 0.524986] dma_pdev #3: 0000:06:07.0 > [ 0.524987] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.524988] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.524989] PCIe ACS override match for 0000:05:00.0 > [ 0.524990] -> true > [ 0.524991] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.524997] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.524998] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.524999] -> false > [ 0.525000] -> false > [ 0.525001] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.525002] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525004] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525006] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525007] -> false > [ 0.525007] -> false > [ 0.525008] dma_pdev #4: 0000:00:1c.1 > [ 0.525010] intel_iommu_add_device(0000:06:09.0) > [ 0.525011] dma_pdev #1: 0000:06:09.0 > [ 0.525012] dma_pdev #2: 0000:06:09.0 > [ 0.525013] dma_pdev #3: 0000:06:09.0 > [ 0.525014] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.525015] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525016] PCIe ACS override match for 0000:05:00.0 > [ 0.525017] -> true > [ 0.525018] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525025] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525026] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525027] -> false > [ 0.525028] -> false > [ 0.525029] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.525030] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525037] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525038] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525039] -> false > [ 0.525040] -> false > [ 0.525041] dma_pdev #4: 0000:00:1c.1 > [ 0.525043] intel_iommu_add_device(0000:0a:00.0) > [ 0.525044] dma_pdev #1: 0000:0a:00.0 > [ 0.525044] dma_pdev #2: 0000:0a:00.0 > [ 0.525045] dma_pdev #3: 0000:0a:00.0 > [ 0.525046] pci_acs_path_enabled(0000:06:07.0, NULL, 001d) > [ 0.525048] pci_acs_enabled(0000:06:07.0, 001d) > [ 0.525048] PCIe ACS override match for 0000:06:07.0 > [ 0.525049] -> true > [ 0.525050] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525051] PCIe ACS override match for 0000:05:00.0 > [ 0.525052] -> true > [ 0.525053] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525060] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525062] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525062] -> false > [ 0.525063] -> false > [ 0.525064] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.525065] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525066] PCIe ACS override match for 0000:05:00.0 > [ 0.525067] -> true > [ 0.525068] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525075] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525076] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525077] -> false > [ 0.525078] -> false > [ 0.525079] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.525080] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525087] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525088] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525089] -> false > [ 0.525090] -> false > [ 0.525091] dma_pdev #4: 0000:00:1c.1 > [ 0.525093] intel_iommu_add_device(0000:0b:00.0) > [ 0.525094] dma_pdev #1: 0000:0b:00.0 > [ 0.525095] dma_pdev #2: 0000:0b:00.0 > [ 0.525096] dma_pdev #3: 0000:0b:00.0 > [ 0.525097] pci_acs_path_enabled(0000:06:09.0, NULL, 001d) > [ 0.525098] pci_acs_enabled(0000:06:09.0, 001d) > [ 0.525099] PCIe ACS override match for 0000:06:09.0 > [ 0.525100] -> true > [ 0.525101] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525102] PCIe ACS override match for 0000:05:00.0 > [ 0.525103] -> true > [ 0.525103] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525111] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525112] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525113] -> false > [ 0.525114] -> false > [ 0.525115] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.525116] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525117] PCIe ACS override match for 0000:05:00.0 > [ 0.525118] -> true > [ 0.525118] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525125] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525126] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525127] -> false > [ 0.525128] -> false > [ 0.525129] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.525130] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525137] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525138] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525139] -> false > [ 0.525140] -> false > [ 0.525141] dma_pdev #4: 0000:00:1c.1 > [ 0.525143] intel_iommu_add_device(0000:0c:00.0) > [ 0.525144] Upstream bridge 0000:0b:00.0 > [ 0.525148] dma_pdev #1: 0000:0c:00.0 > [ 0.525149] dma_pdev #2: 0000:0c:00.0 > [ 0.525150] dma_pdev #3: 0000:0c:00.0 > [ 0.525151] pci_acs_path_enabled(0000:0b:00.0, NULL, 001d) > [ 0.525152] pci_acs_enabled(0000:0b:00.0, 001d) > [ 0.525153] -> false > [ 0.525154] -> false > [ 0.525155] pci_acs_path_enabled(0000:06:09.0, NULL, 001d) > [ 0.525156] pci_acs_enabled(0000:06:09.0, 001d) > [ 0.525157] PCIe ACS override match for 0000:06:09.0 > [ 0.525158] -> true > [ 0.525158] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525159] PCIe ACS override match for 0000:05:00.0 > [ 0.525160] -> true > [ 0.525161] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525168] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525169] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525170] -> false > [ 0.525171] -> false > [ 0.525172] pci_acs_path_enabled(0000:05:00.0, NULL, 001d) > [ 0.525173] pci_acs_enabled(0000:05:00.0, 001d) > [ 0.525174] PCIe ACS override match for 0000:05:00.0 > [ 0.525175] -> true > [ 0.525175] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525178] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525179] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525180] -> false > [ 0.525181] -> false > [ 0.525182] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d) > [ 0.525183] pci_acs_enabled(0000:00:1c.1, 001d) > [ 0.525190] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1 > [ 0.525191] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false > [ 0.525192] -> false > [ 0.525193] -> false > [ 0.525193] dma_pdev #4: 0000:00:1c.1 > [ 0.525195] intel_iommu_add_device(0000:0d:00.0) > [ 0.525196] dma_pdev #1: 0000:0d:00.0 > [ 0.525197] dma_pdev #2: 0000:0d:00.0 > [ 0.525198] dma_pdev #3: 0000:0d:00.0 > [ 0.525199] pci_acs_path_enabled(0000:00:1c.3, NULL, 001d) > [ 0.525200] pci_acs_enabled(0000:00:1c.3, 001d) > [ 0.525201] PCIe ACS override match for 0000:00:1c.3 > [ 0.525202] -> true > [ 0.525203] -> true > [ 0.525204] dma_pdev #4: 0000:0d:00.0 > > lspci as slots are now (since Radeon is back in first x16 slot): > > 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor > DRAM Controller [8086:0c08] (rev 06) > 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) > 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) > 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 > v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) > 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen > Core Processor HD Audio Controller [8086:0c0c] (rev 06) > 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series > Chipset Family USB xHCI [8086:8c31] (rev 05) > 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 > Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) > 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series > Chipset Family KT Controller [8086:8c3d] (rev 04) > 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet > Connection I217-LM [8086:153a] (rev 05) > 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series > Chipset High Definition Audio Controller [8086:8c20] (rev 05) > 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series > Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) > 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series > Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) > 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series > Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) > 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series > Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) > 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series > Chipset Family USB EHCI #1 [8086:8c26] (rev 05) > 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family > Server Advanced SKU LPC Controller [8086:8c56] (rev 05) > 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series > Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) > 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset > Family SMBus Controller [8086:8c22] (rev 05) > 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series > Chipset Family Thermal Management Controller [8086:8c24] (rev 05) > 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. > [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] > 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] > Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] > 02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI > Express-to-PCI Bridge [10b5:8112] (rev aa) > 03:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc > CMI8788 [Oxygen HD Audio] [13f6:8788] > 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial > ATA Controller [1b21:0612] (rev 01) > 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port > PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) > 0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB > 3.0 Host Controller [1912:0014] (rev 03) > 0b:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI > Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) > 0c:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments > XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] > [104c:823f] (rev 01) > 0d:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit > Network Connection [8086:1533] (rev 03) > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch? 2014-01-04 20:22 ` Alex Williamson @ 2014-01-04 21:11 ` Dana Goyette 0 siblings, 0 replies; 15+ messages in thread From: Dana Goyette @ 2014-01-04 21:11 UTC (permalink / raw) To: kvm On 01/04/2014 12:22 PM, Alex Williamson wrote: > On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote: >> On 01/03/2014 04:03 PM, Alex Williamson wrote: >>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote: >>>> On 12/29/2013 08:16 PM, Alex Williamson wrote: >>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote: >>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote: >>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote: >>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to >>>>>>>> soon decide which one to keep. >>>>>>>> >>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX >>>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices >>>>>>>> downstream of the PLX switch to be in separate groups. >>>>>>>> >>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities" >>>>>>>> applied and set for the Intel root ports, the devices behind the switch >>>>>>>> remain stuck in the same group. >>>>>>>> >>>>>>>> In terms of passing devices to different VMs, which is better: all >>>>>>>> devices on different root ports, or all devices behind the one >>>>>>>> ACS-supporting switch? >>>>>>> Can you provide lspci -vvv info? If you're getting that for groups >>>>>>> either the switch has ACS capabilities, but doesn't support the features >>>>>>> we need or we're doing something wrong. Thanks, >>>>>>> >>>>>> I initially tried attaching the output as a .txt file, but it's too >>>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I >>>>>> moved the Radeon to a different slot). >>>>> Well, something seems amiss since the downstream switch ports all seem >>>>> to support and enable the correct set of ACS capabilities. I'm tending >>>>> to suspect something wrong with the ACS override patch or how it's being >>>>> used since your IOMMU group is still based at the root port. Each root >>>>> port is isolated from the other root ports though, so something is >>>>> happening with the override patch. Can you provide the kernel command >>>>> line you use to enable ACS overrides and the override patch you're >>>>> using, as it applies to 3.13-rc5? Thanks, >>>>> >>>>> Alex >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>> I'm using the original acs-override patch from this post: >>>> https://lkml.org/lkml/2013/5/30/513 >>>> >>>> Kernel parameter is: >>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18 > > Actually, you're not: > > pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 > > And we register all of them: > > [ 0.000000] PCIe ACS bypass added for 8086:8c10 > [ 0.000000] PCIe ACS bypass added for 8086:8c16 > [ 0.000000] PCIe ACS bypass added for 8086:8c18 > [ 0.000000] PCIe ACS bypass added for 8086:ac1a > [ 0.000000] PCIe ACS bypass added for 8086:8c1c > [ 0.000000] PCIe ACS bypass added for 8086:8c1e > [ 0.000000] PCIe ACS bypass added for 10b5:8606 > > However, note that the root port causing you trouble is 8086:8c12, which > isn't provided as an override, therefore the code is doing the right > thing and grouping all devices behind that root port together. > Thanks for catching that -- I certainly missed it! I've added the override for that root port and removed the override for the PLX switch; now all the ports are indeed in separate groups. Do we yet know if it'll be possible to properly isolate the Intel root ports, without this ACS override? ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2014-01-04 21:11 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-12-29 2:31 IOMMU groups: better with Intel root ports, or with PEX8606 switch? Dana Goyette 2013-12-29 3:23 ` Alex Williamson 2013-12-29 7:32 ` Dana Goyette 2013-12-30 4:16 ` Alex Williamson 2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette 2014-01-02 19:36 ` Alex Williamson 2014-01-02 21:01 ` Dana Goyette 2014-01-02 21:14 ` Alex Williamson 2014-01-02 21:15 ` Dana Goyette 2014-01-02 21:22 ` Alex Williamson 2014-01-02 21:25 ` Dana Goyette 2014-01-04 0:03 ` Alex Williamson 2014-01-04 19:26 ` Dana Goyette 2014-01-04 20:22 ` Alex Williamson 2014-01-04 21:11 ` Dana Goyette
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox