* IOMMU groups: better with Intel root ports, or with PEX8606 switch?
@ 2013-12-29 2:31 Dana Goyette
2013-12-29 3:23 ` Alex Williamson
0 siblings, 1 reply; 15+ messages in thread
From: Dana Goyette @ 2013-12-29 2:31 UTC (permalink / raw)
To: kvm
I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
soon decide which one to keep.
The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
PEX8066 switch, which claims to support ACS. I'd expect the devices
downstream of the PLX switch to be in separate groups.
With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
applied and set for the Intel root ports, the devices behind the switch
remain stuck in the same group.
In terms of passing devices to different VMs, which is better: all
devices on different root ports, or all devices behind the one
ACS-supporting switch?
X10SAT IOMMU groups:
### Group 0 ###
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor
DRAM Controller (rev 06)
### Group 1 ###
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor PCI Express x16 Controller (rev 06)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
[AMD/ATI] Juniper PRO [Radeon HD 5750]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
Juniper HDMI Audio [Radeon HD 5700 Series]
### Group 2 ###
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor HD Audio Controller (rev 06)
### Group 4 ###
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB xHCI (rev 05)
### Group 5 ###
00:16.0 Communication controller: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series
Chipset Family KT Controller (rev 04)
### Group 6 ###
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
I217-LM (rev 05)
### Group 7 ###
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series
Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #1 (rev d5)
### Group 10 ###
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #2 (rev d5)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0
Host Controller (rev 03)
09:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI
Express to PCI Bridge [Cheetah Express] (rev 01)
0a:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221
IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)
### Group 11 ###
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #4 (rev d5)
### Group 12 ###
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #5 (rev d5)
### Group 13 ###
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #1 (rev 05)
### Group 14 ###
00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family
Server Advanced SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series
Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset
Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series
Chipset Family Thermal Management Controller (rev 05)
### Group 15 ###
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
Controller (rev 01)
### Group 16 ###
0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network
Connection (rev 03)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch?
2013-12-29 2:31 IOMMU groups: better with Intel root ports, or with PEX8606 switch? Dana Goyette
@ 2013-12-29 3:23 ` Alex Williamson
2013-12-29 7:32 ` Dana Goyette
0 siblings, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2013-12-29 3:23 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm
On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> soon decide which one to keep.
>
> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> PEX8066 switch, which claims to support ACS. I'd expect the devices
> downstream of the PLX switch to be in separate groups.
>
> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> applied and set for the Intel root ports, the devices behind the switch
> remain stuck in the same group.
>
> In terms of passing devices to different VMs, which is better: all
> devices on different root ports, or all devices behind the one
> ACS-supporting switch?
Can you provide lspci -vvv info? If you're getting that for groups
either the switch has ACS capabilities, but doesn't support the features
we need or we're doing something wrong. Thanks,
Alex
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch?
2013-12-29 3:23 ` Alex Williamson
@ 2013-12-29 7:32 ` Dana Goyette
2013-12-30 4:16 ` Alex Williamson
0 siblings, 1 reply; 15+ messages in thread
From: Dana Goyette @ 2013-12-29 7:32 UTC (permalink / raw)
To: kvm
On 12/28/2013 7:23 PM, Alex Williamson wrote:
> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>> soon decide which one to keep.
>>
>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>> downstream of the PLX switch to be in separate groups.
>>
>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>> applied and set for the Intel root ports, the devices behind the switch
>> remain stuck in the same group.
>>
>> In terms of passing devices to different VMs, which is better: all
>> devices on different root ports, or all devices behind the one
>> ACS-supporting switch?
>
> Can you provide lspci -vvv info? If you're getting that for groups
> either the switch has ACS capabilities, but doesn't support the features
> we need or we're doing something wrong. Thanks,
>
> Alex
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
I initially tried attaching the output as a .txt file, but it's too
large. Anyway, here's the output of lspci -nnvvv (you may notice that I
moved the Radeon to a different slot).
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor
DRAM Controller [8086:0c08] (rev 06)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002b8 Data: 0000
Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #2, Speed 8GT/s, Width x8, ASPM L0s L1,
Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -3.5dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [140 v1] Root Complex Link
Desc: PortNumber=02 ComponentID=01 EltType=Config
Link0: Desc: TargetPort=00 TargetComponent=01
AssocRCRB- LinkType=MemMapped LinkValid+
Addr: 00000000fed19000
Capabilities: [d94 v1] #19
Kernel driver in use: pcieport
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) (prog-
if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: eee00000-eeefffff
Prefetchable memory behind bridge:
00000000c0000000-00000000cfffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002d8 Data: 0000
Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #3, Speed 8GT/s, Width x8, ASPM L0s L1,
Latency L0 <256ns, L1 <8us
ClockPM- Surprise- LLActRep- BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+
DLActive- BWMgmt+ ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #2, PowerLimit 75.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -3.5dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [140 v1] Root Complex Link
Desc: PortNumber=03 ComponentID=01 EltType=Config
Link0: Desc: TargetPort=00 TargetComponent=01
AssocRCRB- LinkType=MemMapped LinkValid+
Addr: 00000000fed19000
Capabilities: [d94 v1] #19
Kernel driver in use: pcieport
00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v3
Processor Integrated Graphics Controller [8086:041a] (rev 06)
Subsystem: Intel Corporation Device [8086:2010]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 69
Region 0: Memory at ee400000 (64-bit, non-prefetchable) [size=4M]
Region 2: Memory at 80000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=64]
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee00018 Data: 0000
Capabilities: [d0] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a4] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: i915
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor HD Audio Controller [8086:0c0c] (rev 06)
Subsystem: Intel Corporation Device [8086:2010]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 70
Region 0: Memory at eef34000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee005f8 Data: 0000
Capabilities: [70] Express (v1) Root Complex Integrated
Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
ExtTag- RBE- FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown,
Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: snd_hda_intel
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
Chipset Family USB xHCI [8086:8c31] (rev 05) (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 50
Region 0: Memory at eef20000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Address: 00000000fee003f8 Data: 0000
Kernel driver in use: xhci_hcd
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at eef40000 (64-bit, non-prefetchable) [size=16]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series
Chipset Family KT Controller [8086:8c3d] (rev 04) (prog-if 02
[16550])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at f0c0 [size=8]
Region 1: Memory at eef3e000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [c8] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: serial
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet
Connection I217-LM [8086:153a] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:153a]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 61
Region 0: Memory at eef00000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at eef3d000 (32-bit, non-prefetchable) [size=4K]
Region 2: I/O ports at f060 [size=32]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00578 Data: 0000
Capabilities: [e0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: e1000e
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series
Chipset High Definition Audio Controller [8086:8c20] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 67
Region 0: Memory at eef30000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee005b8 Data: 0000
Capabilities: [70] Express (v1) Root Complex Integrated
Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE- FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr+ TransPend-
LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown,
Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=2 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Kernel driver in use: snd_hda_intel
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: eed00000-eedfffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr+ TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <512ns, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #0, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Super Micro Computer Inc Device
[15d9:0818]
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=0b, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee800000-eebfffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <1us, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive+ BWMgmt+ ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #1, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Super Micro Computer Inc Device
[15d9:0818]
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: eec00000-eecfffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr+ TransPend-
LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <512ns, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #3, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Super Micro Computer Inc Device
[15d9:0818]
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 128 bytes
Bus: primary=00, secondary=0d, subordinate=45, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d8000000-ee0fffff
Prefetchable memory behind bridge:
0000000090000000-00000000b1ffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq-
AuxPwr+ TransPend-
LnkCap: Port #5, Speed 5GT/s, Width x4, ASPM L0s L1,
Latency L0 <1us, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled-
Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug+ Surprise+
Slot #4, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown,
Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+
PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+ ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Super Micro Computer Inc Device
[15d9:0818]
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #1 [8086:8c26] (rev 05) (prog-if 20 [EHCI])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 23
Region 0: Memory at eef3b000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: ehci-pci
00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family
Server Advanced SKU LPC Controller [8086:8c56] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series
Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02]
(rev 05) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 59
Region 0: I/O ports at f0b0 [size=8]
Region 1: I/O ports at f0a0 [size=4]
Region 2: I/O ports at f090 [size=8]
Region 3: I/O ports at f080 [size=4]
Region 4: I/O ports at f040 [size=32]
Region 5: Memory at eef3a000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee00518 Data: 0000
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
Kernel driver in use: ahci
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset
Family SMBus Controller [8086:8c22] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin C routed to IRQ 5
Region 0: Memory at eef39000 (64-bit, non-prefetchable) [size=256]
Region 4: I/O ports at 0580 [size=32]
00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series
Chipset Family Thermal Management Controller [8086:8c24] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin C routed to IRQ 5
Region 0: Memory at eef38000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] (prog-if 00 [VGA
controller])
Subsystem: XFX Pine Group Inc. Device [1682:2980]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 71
Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at eee20000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at e000 [size=256]
Expansion ROM at eee00000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x16, ASPM L0s L1,
Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00718 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001
Rev=1 Len=010 <?>
Capabilities: [150 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn-
ChkCap+ ChkEn-
Kernel driver in use: radeon
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]
Subsystem: XFX Pine Group Inc. Device [1682:aa58]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 68
Region 0: Memory at eee40000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x16, ASPM L0s L1,
Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee005d8 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001
Rev=1 Len=010 <?>
Capabilities: [150 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn-
ChkCap+ ChkEn-
Kernel driver in use: snd_hda_intel
03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial
ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 60
Region 0: I/O ports at d050 [size=8]
Region 1: I/O ports at d040 [size=4]
Region 2: I/O ports at d030 [size=8]
Region 3: I/O ports at d020 [size=4]
Region 4: I/O ports at d000 [size=32]
Region 5: Memory at eed00000 (32-bit, non-prefetchable) [size=512]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee00538 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM unknown,
Latency L0 <512ns, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Kernel driver in use: ahci
04:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR+ <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
Bus: primary=04, secondary=05, subordinate=0b, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee800000-eeafffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee00318 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
FLReset-SlotPowerLimit 10.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <1us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 1f, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=4
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=06 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32+ WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=WRR32 TC/VC=01
Status: NegoPending- InProgress-
Port Arbitration Table <?>
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [448 v1] Vendor Specific Information: ID=0000
Rev=0 Len=0cc <?>
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
05:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee00358 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <2us, L1 <4us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #113, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 1f, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [520 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
05:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=05, secondary=07, subordinate=07, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee00378 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <2us, L1 <4us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #116, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 1f, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [520 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
05:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=05, secondary=08, subordinate=08, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee00398 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <2us, L1 <4us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #117, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 1f, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [520 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
05:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR+ <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=05, secondary=09, subordinate=09, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: eea00000-eeafffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee003b8 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <1us, L1 <2us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk-
DLActive+ BWMgmt+ ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #119, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 14, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [520 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
05:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=05, secondary=0a, subordinate=0b, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee800000-ee9fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Address: 00000000fee003d8 Data: 0000
Masking: 00000001 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #9, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <2us, L1 <4us
ClockPM- Surprise+ LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk-
DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd-
HotPlug- Surprise-
Slot #121, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt-
PresDet+ Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6
Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 1f, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=1 RefClk=100ns PATEntryBits=1
Arb: Fixed+ WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [520 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+
UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [950 v1] Vendor Specific Information: ID=0001
Rev=0 Len=010 <?>
Kernel driver in use: pcieport
09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB
3.0 Host Controller [1912:0014] (rev 03) (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc Device [15d9:0818]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at eea00000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
Vector table: BAR=0 offset=00001000
PBA: BAR=0 offset=00001080
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1,
Latency L0 <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn-
ChkCap- ChkEn-
Capabilities: [150 v1] Latency Tolerance Reporting
Max snoop latency: 71680ns
Max no snoop latency: 71680ns
Kernel driver in use: xhci_hcd
0a:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI
Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 1: Memory at ee900000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: ee800000-ee8fffff
Prefetchable memory behind bridge:
00000000fff00000-00000000000fffff
Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Bridge: PM- B3+
Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [80] Subsystem: Device [3412:7856]
Capabilities: [90] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
Latency L0 <1us, L1 <16us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 14, GenCap+ CGenEn-
ChkCap+ ChkEn-
0b:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments
XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
[104c:823f] (rev
01) (prog-if 10 [OHCI])
Subsystem: Device [3412:7856]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (500ns min, 1000ns max), Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 18
Region 0: Memory at ee804000 (32-bit, non-prefetchable) [size=2K]
Region 1: Memory at ee800000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [44] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME+
Kernel driver in use: firewire_ohci
0c:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit
Network Connection [8086:1533] (rev 03)
Subsystem: Super Micro Computer Inc Device [15d9:1533]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 19
Region 0: Memory at eec00000 (32-bit, non-prefetchable) [size=512K]
Region 2: I/O ports at c000 [size=32]
Region 3: Memory at eec80000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
Latency L0 <2us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-d8-6d-cf
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Kernel driver in use: igb
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups: better with Intel root ports, or with PEX8606 switch?
2013-12-29 7:32 ` Dana Goyette
@ 2013-12-30 4:16 ` Alex Williamson
2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette
0 siblings, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2013-12-30 4:16 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm
On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> > On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >> soon decide which one to keep.
> >>
> >> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >> downstream of the PLX switch to be in separate groups.
> >>
> >> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >> applied and set for the Intel root ports, the devices behind the switch
> >> remain stuck in the same group.
> >>
> >> In terms of passing devices to different VMs, which is better: all
> >> devices on different root ports, or all devices behind the one
> >> ACS-supporting switch?
> >
> > Can you provide lspci -vvv info? If you're getting that for groups
> > either the switch has ACS capabilities, but doesn't support the features
> > we need or we're doing something wrong. Thanks,
> >
> I initially tried attaching the output as a .txt file, but it's too
> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
> moved the Radeon to a different slot).
Well, something seems amiss since the downstream switch ports all seem
to support and enable the correct set of ACS capabilities. I'm tending
to suspect something wrong with the ACS override patch or how it's being
used since your IOMMU group is still based at the root port. Each root
port is isolated from the other root ports though, so something is
happening with the override patch. Can you provide the kernel command
line you use to enable ACS overrides and the override patch you're
using, as it applies to 3.13-rc5? Thanks,
Alex
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2013-12-30 4:16 ` Alex Williamson
@ 2013-12-31 0:13 ` Dana Goyette
2014-01-02 19:36 ` Alex Williamson
2014-01-04 0:03 ` Alex Williamson
0 siblings, 2 replies; 15+ messages in thread
From: Dana Goyette @ 2013-12-31 0:13 UTC (permalink / raw)
To: kvm
On 12/29/2013 08:16 PM, Alex Williamson wrote:
> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>> soon decide which one to keep.
>>>>
>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>> downstream of the PLX switch to be in separate groups.
>>>>
>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>> applied and set for the Intel root ports, the devices behind the switch
>>>> remain stuck in the same group.
>>>>
>>>> In terms of passing devices to different VMs, which is better: all
>>>> devices on different root ports, or all devices behind the one
>>>> ACS-supporting switch?
>>>
>>> Can you provide lspci -vvv info? If you're getting that for groups
>>> either the switch has ACS capabilities, but doesn't support the features
>>> we need or we're doing something wrong. Thanks,
>>>
>> I initially tried attaching the output as a .txt file, but it's too
>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
>> moved the Radeon to a different slot).
>
> Well, something seems amiss since the downstream switch ports all seem
> to support and enable the correct set of ACS capabilities. I'm tending
> to suspect something wrong with the ACS override patch or how it's being
> used since your IOMMU group is still based at the root port. Each root
> port is isolated from the other root ports though, so something is
> happening with the override patch. Can you provide the kernel command
> line you use to enable ACS overrides and the override patch you're
> using, as it applies to 3.13-rc5? Thanks,
>
> Alex
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
I'm using the original acs-override patch from this post:
https://lkml.org/lkml/2013/5/30/513
Kernel parameter is:
pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
When booting a kernel without the override patch, the following devices
are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
controller; Intel I210 Ethernet controller.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette
@ 2014-01-02 19:36 ` Alex Williamson
2014-01-02 21:01 ` Dana Goyette
2014-01-04 0:03 ` Alex Williamson
1 sibling, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2014-01-02 19:36 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm, linux-pci
On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>> soon decide which one to keep.
> >>>>
> >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >>>> downstream of the PLX switch to be in separate groups.
> >>>>
> >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>> applied and set for the Intel root ports, the devices behind the switch
> >>>> remain stuck in the same group.
> >>>>
> >>>> In terms of passing devices to different VMs, which is better: all
> >>>> devices on different root ports, or all devices behind the one
> >>>> ACS-supporting switch?
> >>>
> >>> Can you provide lspci -vvv info? If you're getting that for groups
> >>> either the switch has ACS capabilities, but doesn't support the features
> >>> we need or we're doing something wrong. Thanks,
> >>>
> >> I initially tried attaching the output as a .txt file, but it's too
> >> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
> >> moved the Radeon to a different slot).
> >
> > Well, something seems amiss since the downstream switch ports all seem
> > to support and enable the correct set of ACS capabilities. I'm tending
> > to suspect something wrong with the ACS override patch or how it's being
> > used since your IOMMU group is still based at the root port. Each root
> > port is isolated from the other root ports though, so something is
> > happening with the override patch. Can you provide the kernel command
> > line you use to enable ACS overrides and the override patch you're
> > using, as it applies to 3.13-rc5? Thanks,
> >
> > Alex
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> I'm using the original acs-override patch from this post:
> https://lkml.org/lkml/2013/5/30/513
>
> Kernel parameter is:
> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>
> When booting a kernel without the override patch, the following devices
> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> controller; Intel I210 Ethernet controller.
Ok, here's my shot in the dark; we must be detecting something about the
upstream switch port to make it fail the ACS test and the only thing I
can find that might do this is if the PCI config header on the upstream
switch reported itself as a multifunction device. Multifunction
upstream switch ports do need ACS capabilities to make sure that traffic
isn't routed back through other functions. Single function devices do
not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
looking to see whether the byte at 0xe has the MSB set. If it does, it
lies that it's a multifunction device. If it doesn't I'll have to get
the dart board back out.
FWIW, you should be able to work around this by adding id:10b5:8606 to
your list of overrides. Long term, if this is the problem, we'll want
to add a quirk to sanitize the multifunction device flag. Thanks,
Alex
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-02 19:36 ` Alex Williamson
@ 2014-01-02 21:01 ` Dana Goyette
2014-01-02 21:14 ` Alex Williamson
2014-01-02 21:15 ` Dana Goyette
0 siblings, 2 replies; 15+ messages in thread
From: Dana Goyette @ 2014-01-02 21:01 UTC (permalink / raw)
To: kvm; +Cc: linux-pci
On 01/02/2014 11:36 AM, Alex Williamson wrote:
> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>> soon decide which one to keep.
>>>>>>
>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>
>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>> remain stuck in the same group.
>>>>>>
>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>> devices on different root ports, or all devices behind the one
>>>>>> ACS-supporting switch?
>>>>>
>>>>> Can you provide lspci -vvv info? If you're getting that for groups
>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>> we need or we're doing something wrong. Thanks,
>>>>>
>>>> I initially tried attaching the output as a .txt file, but it's too
>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>> moved the Radeon to a different slot).
>>>
>>> Well, something seems amiss since the downstream switch ports all seem
>>> to support and enable the correct set of ACS capabilities. I'm tending
>>> to suspect something wrong with the ACS override patch or how it's being
>>> used since your IOMMU group is still based at the root port. Each root
>>> port is isolated from the other root ports though, so something is
>>> happening with the override patch. Can you provide the kernel command
>>> line you use to enable ACS overrides and the override patch you're
>>> using, as it applies to 3.13-rc5? Thanks,
>>>
>>> Alex
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> I'm using the original acs-override patch from this post:
>> https://lkml.org/lkml/2013/5/30/513
>>
>> Kernel parameter is:
>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>
>> When booting a kernel without the override patch, the following devices
>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>> controller; Intel I210 Ethernet controller.
>
> Ok, here's my shot in the dark; we must be detecting something about the
> upstream switch port to make it fail the ACS test and the only thing I
> can find that might do this is if the PCI config header on the upstream
> switch reported itself as a multifunction device. Multifunction
> upstream switch ports do need ACS capabilities to make sure that traffic
> isn't routed back through other functions. Single function devices do
> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
> looking to see whether the byte at 0xe has the MSB set. If it does, it
> lies that it's a multifunction device. If it doesn't I'll have to get
> the dart board back out.
>
> FWIW, you should be able to work around this by adding id:10b5:8606 to
> your list of overrides. Long term, if this is the problem, we'll want
> to add a quirk to sanitize the multifunction device flag. Thanks,
>
> Alex
>
04:00.0 is the ASMedia SATA controller; I'll assume you meant the
upstream port of the PLX switch.
05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
Memory behind bridge: ee800000-eeafffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
Port PCI Express Gen 2 (5.0 GT/s) Switch
Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [148] Virtual Channel
Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?>
Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport
00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-02 21:01 ` Dana Goyette
@ 2014-01-02 21:14 ` Alex Williamson
2014-01-02 21:15 ` Dana Goyette
1 sibling, 0 replies; 15+ messages in thread
From: Alex Williamson @ 2014-01-02 21:14 UTC (permalink / raw)
To: Dana Goyette; +Cc: linux-pci, kvm
On Thu, 2014-01-02 at 13:01 -0800, Dana Goyette wrote:
> On 01/02/2014 11:36 AM, Alex Williamson wrote:
> > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> >> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>>>> soon decide which one to keep.
> >>>>>>
> >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >>>>>> downstream of the PLX switch to be in separate groups.
> >>>>>>
> >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>>>> applied and set for the Intel root ports, the devices behind the switch
> >>>>>> remain stuck in the same group.
> >>>>>>
> >>>>>> In terms of passing devices to different VMs, which is better: all
> >>>>>> devices on different root ports, or all devices behind the one
> >>>>>> ACS-supporting switch?
> >>>>>
> >>>>> Can you provide lspci -vvv info? If you're getting that for groups
> >>>>> either the switch has ACS capabilities, but doesn't support the features
> >>>>> we need or we're doing something wrong. Thanks,
> >>>>>
> >>>> I initially tried attaching the output as a .txt file, but it's too
> >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
> >>>> moved the Radeon to a different slot).
> >>>
> >>> Well, something seems amiss since the downstream switch ports all seem
> >>> to support and enable the correct set of ACS capabilities. I'm tending
> >>> to suspect something wrong with the ACS override patch or how it's being
> >>> used since your IOMMU group is still based at the root port. Each root
> >>> port is isolated from the other root ports though, so something is
> >>> happening with the override patch. Can you provide the kernel command
> >>> line you use to enable ACS overrides and the override patch you're
> >>> using, as it applies to 3.13-rc5? Thanks,
> >>>
> >>> Alex
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>
> >> I'm using the original acs-override patch from this post:
> >> https://lkml.org/lkml/2013/5/30/513
> >>
> >> Kernel parameter is:
> >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
> >>
> >> When booting a kernel without the override patch, the following devices
> >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> >> controller; Intel I210 Ethernet controller.
> >
> > Ok, here's my shot in the dark; we must be detecting something about the
> > upstream switch port to make it fail the ACS test and the only thing I
> > can find that might do this is if the PCI config header on the upstream
> > switch reported itself as a multifunction device. Multifunction
> > upstream switch ports do need ACS capabilities to make sure that traffic
> > isn't routed back through other functions. Single function devices do
> > not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
> > looking to see whether the byte at 0xe has the MSB set. If it does, it
> > lies that it's a multifunction device. If it doesn't I'll have to get
> > the dart board back out.
> >
> > FWIW, you should be able to work around this by adding id:10b5:8606 to
> > your list of overrides. Long term, if this is the problem, we'll want
> > to add a quirk to sanitize the multifunction device flag. Thanks,
> >
> > Alex
> >
>
> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
> upstream port of the PLX switch.
Yes, it was 04:00.0 in the previous lspci listing.
> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
> Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
> Memory behind bridge: ee800000-eeafffff
> Capabilities: [40] Power Management version 3
> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> Capabilities: [68] Express Upstream Port, MSI 00
> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
> Port PCI Express Gen 2 (5.0 GT/s) Switch
> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> Capabilities: [fb4] Advanced Error Reporting
> Capabilities: [138] Power Budgeting <?>
> Capabilities: [148] Virtual Channel
> Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?>
> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
> Kernel driver in use: pcieport
> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
Well, that wasn't it. I'll keep looking and probably send you a patch
to figure out what's going wrong. Thanks,
Alex
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-02 21:01 ` Dana Goyette
2014-01-02 21:14 ` Alex Williamson
@ 2014-01-02 21:15 ` Dana Goyette
2014-01-02 21:22 ` Alex Williamson
1 sibling, 1 reply; 15+ messages in thread
From: Dana Goyette @ 2014-01-02 21:15 UTC (permalink / raw)
To: kvm; +Cc: linux-pci
On 01/02/2014 01:01 PM, Dana Goyette wrote:
> On 01/02/2014 11:36 AM, Alex Williamson wrote:
>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
>>>>>>> need to
>>>>>>> soon decide which one to keep.
>>>>>>>
>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>
>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
>>>>>>> capabilities"
>>>>>>> applied and set for the Intel root ports, the devices behind the
>>>>>>> switch
>>>>>>> remain stuck in the same group.
>>>>>>>
>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>> ACS-supporting switch?
>>>>>>
>>>>>> Can you provide lspci -vvv info? If you're getting that for groups
>>>>>> either the switch has ACS capabilities, but doesn't support the
>>>>>> features
>>>>>> we need or we're doing something wrong. Thanks,
>>>>>>
>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice
>>>>> that I
>>>>> moved the Radeon to a different slot).
>>>>
>>>> Well, something seems amiss since the downstream switch ports all seem
>>>> to support and enable the correct set of ACS capabilities. I'm tending
>>>> to suspect something wrong with the ACS override patch or how it's
>>>> being
>>>> used since your IOMMU group is still based at the root port. Each root
>>>> port is isolated from the other root ports though, so something is
>>>> happening with the override patch. Can you provide the kernel command
>>>> line you use to enable ACS overrides and the override patch you're
>>>> using, as it applies to 3.13-rc5? Thanks,
>>>>
>>>> Alex
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>> I'm using the original acs-override patch from this post:
>>> https://lkml.org/lkml/2013/5/30/513
>>>
>>> Kernel parameter is:
>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>>
>>> When booting a kernel without the override patch, the following devices
>>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>>> controller; Intel I210 Ethernet controller.
>>
>> Ok, here's my shot in the dark; we must be detecting something about the
>> upstream switch port to make it fail the ACS test and the only thing I
>> can find that might do this is if the PCI config header on the upstream
>> switch reported itself as a multifunction device. Multifunction
>> upstream switch ports do need ACS capabilities to make sure that traffic
>> isn't routed back through other functions. Single function devices do
>> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
>> looking to see whether the byte at 0xe has the MSB set. If it does, it
>> lies that it's a multifunction device. If it doesn't I'll have to get
>> the dart board back out.
>>
>> FWIW, you should be able to work around this by adding id:10b5:8606 to
>> your list of overrides. Long term, if this is the problem, we'll want
>> to add a quirk to sanitize the multifunction device flag. Thanks,
>>
>> Alex
>>
>
> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
> upstream port of the PLX switch.
>
> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
> Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
> Memory behind bridge: ee800000-eeafffff
> Capabilities: [40] Power Management version 3
> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> Capabilities: [68] Express Upstream Port, MSI 00
> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
> 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> Capabilities: [fb4] Advanced Error Reporting
> Capabilities: [138] Power Budgeting <?>
> Capabilities: [148] Virtual Channel
> Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
> Len=0cc <?>
> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
> Len=010 <?>
> Kernel driver in use: pcieport
> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
> 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
> 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Just in case:
The root port:
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
Memory behind bridge: ee800000-eebfffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport
00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
The first downstream port:
06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
Port PCI Express Gen 2 (5.0 GT/s) Switch
Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [520] Access Control Services
Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport
00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-02 21:15 ` Dana Goyette
@ 2014-01-02 21:22 ` Alex Williamson
2014-01-02 21:25 ` Dana Goyette
0 siblings, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2014-01-02 21:22 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm, linux-pci
On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote:
> On 01/02/2014 01:01 PM, Dana Goyette wrote:
> > On 01/02/2014 11:36 AM, Alex Williamson wrote:
> >> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> >>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> >>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
> >>>>>>> need to
> >>>>>>> soon decide which one to keep.
> >>>>>>>
> >>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >>>>>>> downstream of the PLX switch to be in separate groups.
> >>>>>>>
> >>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
> >>>>>>> capabilities"
> >>>>>>> applied and set for the Intel root ports, the devices behind the
> >>>>>>> switch
> >>>>>>> remain stuck in the same group.
> >>>>>>>
> >>>>>>> In terms of passing devices to different VMs, which is better: all
> >>>>>>> devices on different root ports, or all devices behind the one
> >>>>>>> ACS-supporting switch?
> >>>>>>
> >>>>>> Can you provide lspci -vvv info? If you're getting that for groups
> >>>>>> either the switch has ACS capabilities, but doesn't support the
> >>>>>> features
> >>>>>> we need or we're doing something wrong. Thanks,
> >>>>>>
> >>>>> I initially tried attaching the output as a .txt file, but it's too
> >>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice
> >>>>> that I
> >>>>> moved the Radeon to a different slot).
> >>>>
> >>>> Well, something seems amiss since the downstream switch ports all seem
> >>>> to support and enable the correct set of ACS capabilities. I'm tending
> >>>> to suspect something wrong with the ACS override patch or how it's
> >>>> being
> >>>> used since your IOMMU group is still based at the root port. Each root
> >>>> port is isolated from the other root ports though, so something is
> >>>> happening with the override patch. Can you provide the kernel command
> >>>> line you use to enable ACS overrides and the override patch you're
> >>>> using, as it applies to 3.13-rc5? Thanks,
> >>>>
> >>>> Alex
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>
> >>> I'm using the original acs-override patch from this post:
> >>> https://lkml.org/lkml/2013/5/30/513
> >>>
> >>> Kernel parameter is:
> >>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
> >>>
> >>> When booting a kernel without the override patch, the following devices
> >>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> >>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> >>> controller; Intel I210 Ethernet controller.
> >>
> >> Ok, here's my shot in the dark; we must be detecting something about the
> >> upstream switch port to make it fail the ACS test and the only thing I
> >> can find that might do this is if the PCI config header on the upstream
> >> switch reported itself as a multifunction device. Multifunction
> >> upstream switch ports do need ACS capabilities to make sure that traffic
> >> isn't routed back through other functions. Single function devices do
> >> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
> >> looking to see whether the byte at 0xe has the MSB set. If it does, it
> >> lies that it's a multifunction device. If it doesn't I'll have to get
> >> the dart board back out.
> >>
> >> FWIW, you should be able to work around this by adding id:10b5:8606 to
> >> your list of overrides. Long term, if this is the problem, we'll want
> >> to add a quirk to sanitize the multifunction device flag. Thanks,
> >>
> >> Alex
> >>
> >
> > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
> > upstream port of the PLX switch.
> >
> > 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
> > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> > Flags: bus master, fast devsel, latency 0
> > Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
> > Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
> > Memory behind bridge: ee800000-eeafffff
> > Capabilities: [40] Power Management version 3
> > Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> > Capabilities: [68] Express Upstream Port, MSI 00
> > Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
> > 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
> > Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> > Capabilities: [fb4] Advanced Error Reporting
> > Capabilities: [138] Power Budgeting <?>
> > Capabilities: [148] Virtual Channel
> > Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
> > Len=0cc <?>
> > Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
> > Len=010 <?>
> > Kernel driver in use: pcieport
> > 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
> > 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
> > 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
> > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> Just in case:
>
> The root port:
> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
> Memory behind bridge: ee800000-eebfffff
> Capabilities: [40] Express Root Port (Slot+), MSI 00
> Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
> Capabilities: [a0] Power Management version 3
> Kernel driver in use: pcieport
> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
>
> The first downstream port:
>
> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> Flags: bus master, fast devsel, latency 0
> Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
> Capabilities: [40] Power Management version 3
> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> Capabilities: [68] Express Downstream Port (Slot+), MSI 00
> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
> Port PCI Express Gen 2 (5.0 GT/s) Switch
> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> Capabilities: [fb4] Advanced Error Reporting
> Capabilities: [148] Virtual Channel
> Capabilities: [520] Access Control Services
> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
> Kernel driver in use: pcieport
> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
Thanks, can you confirm whether adding id:10b5:8606 to the overrides
works?
Alex
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-02 21:22 ` Alex Williamson
@ 2014-01-02 21:25 ` Dana Goyette
0 siblings, 0 replies; 15+ messages in thread
From: Dana Goyette @ 2014-01-02 21:25 UTC (permalink / raw)
To: linux-pci; +Cc: kvm
On 01/02/2014 01:22 PM, Alex Williamson wrote:
> On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote:
>> On 01/02/2014 01:01 PM, Dana Goyette wrote:
>>> On 01/02/2014 11:36 AM, Alex Williamson wrote:
>>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
>>>>>>>>> need to
>>>>>>>>> soon decide which one to keep.
>>>>>>>>>
>>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>>
>>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
>>>>>>>>> capabilities"
>>>>>>>>> applied and set for the Intel root ports, the devices behind the
>>>>>>>>> switch
>>>>>>>>> remain stuck in the same group.
>>>>>>>>>
>>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>>> ACS-supporting switch?
>>>>>>>>
>>>>>>>> Can you provide lspci -vvv info? If you're getting that for groups
>>>>>>>> either the switch has ACS capabilities, but doesn't support the
>>>>>>>> features
>>>>>>>> we need or we're doing something wrong. Thanks,
>>>>>>>>
>>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice
>>>>>>> that I
>>>>>>> moved the Radeon to a different slot).
>>>>>>
>>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>>> to support and enable the correct set of ACS capabilities. I'm tending
>>>>>> to suspect something wrong with the ACS override patch or how it's
>>>>>> being
>>>>>> used since your IOMMU group is still based at the root port. Each root
>>>>>> port is isolated from the other root ports though, so something is
>>>>>> happening with the override patch. Can you provide the kernel command
>>>>>> line you use to enable ACS overrides and the override patch you're
>>>>>> using, as it applies to 3.13-rc5? Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>> I'm using the original acs-override patch from this post:
>>>>> https://lkml.org/lkml/2013/5/30/513
>>>>>
>>>>> Kernel parameter is:
>>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>>>>
>>>>> When booting a kernel without the override patch, the following devices
>>>>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>>>>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>>>>> controller; Intel I210 Ethernet controller.
>>>>
>>>> Ok, here's my shot in the dark; we must be detecting something about the
>>>> upstream switch port to make it fail the ACS test and the only thing I
>>>> can find that might do this is if the PCI config header on the upstream
>>>> switch reported itself as a multifunction device. Multifunction
>>>> upstream switch ports do need ACS capabilities to make sure that traffic
>>>> isn't routed back through other functions. Single function devices do
>>>> not. To test that theory, please provide 'lspci -vxs 4:00.0'. We're
>>>> looking to see whether the byte at 0xe has the MSB set. If it does, it
>>>> lies that it's a multifunction device. If it doesn't I'll have to get
>>>> the dart board back out.
>>>>
>>>> FWIW, you should be able to work around this by adding id:10b5:8606 to
>>>> your list of overrides. Long term, if this is the problem, we'll want
>>>> to add a quirk to sanitize the multifunction device flag. Thanks,
>>>>
>>>> Alex
>>>>
>>>
>>> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
>>> upstream port of the PLX switch.
>>>
>>> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>>> Flags: bus master, fast devsel, latency 0
>>> Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
>>> Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
>>> Memory behind bridge: ee800000-eeafffff
>>> Capabilities: [40] Power Management version 3
>>> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>>> Capabilities: [68] Express Upstream Port, MSI 00
>>> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
>>> 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
>>> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>>> Capabilities: [fb4] Advanced Error Reporting
>>> Capabilities: [138] Power Budgeting <?>
>>> Capabilities: [148] Virtual Channel
>>> Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
>>> Len=0cc <?>
>>> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
>>> Len=010 <?>
>>> Kernel driver in use: pcieport
>>> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
>>> 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
>>> 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>> Just in case:
>>
>> The root port:
>> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
>> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
>> Flags: bus master, fast devsel, latency 0
>> Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
>> Memory behind bridge: ee800000-eebfffff
>> Capabilities: [40] Express Root Port (Slot+), MSI 00
>> Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
>> Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
>> Capabilities: [a0] Power Management version 3
>> Kernel driver in use: pcieport
>> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
>> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
>> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
>>
>> The first downstream port:
>>
>> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>> Flags: bus master, fast devsel, latency 0
>> Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
>> Capabilities: [40] Power Management version 3
>> Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>> Capabilities: [68] Express Downstream Port (Slot+), MSI 00
>> Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
>> Port PCI Express Gen 2 (5.0 GT/s) Switch
>> Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>> Capabilities: [fb4] Advanced Error Reporting
>> Capabilities: [148] Virtual Channel
>> Capabilities: [520] Access Control Services
>> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
>> Kernel driver in use: pcieport
>> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
>> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
>> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>
> Thanks, can you confirm whether adding id:10b5:8606 to the overrides
> works?
>
> Alex
>
Oddly, adding that ID doesn't seem to change anything.
\boot\vmlinuz-3.13.0-rc5 ro
root=UUID=8414610a-91ff-41b9-80c2-cc49ee34cce6 console=ttyS4,115200n8
console=tty0 intel_iommu=on,igfx_off
pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606
initrd=boot\initrd.img-3.13.0-rc5
### Group 10 ###
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #2 (rev d5)
05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
06:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0
Host Controller (rev 03)
0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI
Express to PCI Bridge [Cheetah Express] (rev 01)
0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221
IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette
2014-01-02 19:36 ` Alex Williamson
@ 2014-01-04 0:03 ` Alex Williamson
2014-01-04 19:26 ` Dana Goyette
1 sibling, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2014-01-04 0:03 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm
On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>> soon decide which one to keep.
> >>>>
> >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >>>> downstream of the PLX switch to be in separate groups.
> >>>>
> >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>> applied and set for the Intel root ports, the devices behind the switch
> >>>> remain stuck in the same group.
> >>>>
> >>>> In terms of passing devices to different VMs, which is better: all
> >>>> devices on different root ports, or all devices behind the one
> >>>> ACS-supporting switch?
> >>>
> >>> Can you provide lspci -vvv info? If you're getting that for groups
> >>> either the switch has ACS capabilities, but doesn't support the features
> >>> we need or we're doing something wrong. Thanks,
> >>>
> >> I initially tried attaching the output as a .txt file, but it's too
> >> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
> >> moved the Radeon to a different slot).
> >
> > Well, something seems amiss since the downstream switch ports all seem
> > to support and enable the correct set of ACS capabilities. I'm tending
> > to suspect something wrong with the ACS override patch or how it's being
> > used since your IOMMU group is still based at the root port. Each root
> > port is isolated from the other root ports though, so something is
> > happening with the override patch. Can you provide the kernel command
> > line you use to enable ACS overrides and the override patch you're
> > using, as it applies to 3.13-rc5? Thanks,
> >
> > Alex
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> I'm using the original acs-override patch from this post:
> https://lkml.org/lkml/2013/5/30/513
>
> Kernel parameter is:
> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>
> When booting a kernel without the override patch, the following devices
> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> controller; Intel I210 Ethernet controller.
Could you please try the patch below and send dmesg for the system once
booted. This applies directly to upstream and includes the acs override
patch. Thanks,
Alex
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index b9e9bd8..2481c6c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2473,6 +2473,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
nomsi Do not use MSI for native PCIe PME signaling (this makes
all PCIe root ports use INTx for all services).
+ pcie_acs_override =
+ [PCIE] Override missing PCIe ACS support for:
+ downstream
+ All downstream ports - full ACS capabilties
+ multifunction
+ All multifunction devices - multifunction ACS subset
+ id:nnnn:nnnn
+ Specfic device - full ACS capabilities
+ Specified as vid:did (vendor/device ID) in hex
+
pcmv= [HW,PCMCIA] BadgePAD 4
pd. [PARIDE]
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 43b9bfe..b31961d 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4166,8 +4166,10 @@ static int intel_iommu_add_device(struct device *dev)
pdev->bus->number, pdev->devfn))
return -ENODEV;
+ pr_info("%s(%s)\n", __func__, pci_name(pdev));
bridge = pci_find_upstream_pcie_bridge(pdev);
if (bridge) {
+ pr_info("Upstream bridge %s\n", pci_name(bridge));
if (pci_is_pcie(bridge))
dma_pdev = pci_get_domain_bus_and_slot(
pci_domain_nr(pdev->bus),
@@ -4177,9 +4179,13 @@ static int intel_iommu_add_device(struct device *dev)
} else
dma_pdev = pci_dev_get(pdev);
+ pr_info("dma_pdev #1: %s\n", pci_name(dma_pdev));
+
/* Account for quirked devices */
swap_pci_ref(&dma_pdev, pci_get_dma_source(dma_pdev));
+ pr_info("dma_pdev #2: %s\n", pci_name(dma_pdev));
+
/*
* If it's a multifunction device that does not support our
* required ACS flags, add to the same group as lowest numbered
@@ -4204,6 +4210,8 @@ static int intel_iommu_add_device(struct device *dev)
}
}
+ pr_info("dma_pdev #3: %s\n", pci_name(dma_pdev));
+
/*
* Devices on the root bus go through the iommu. If that's not us,
* find the next upstream device and test ACS up to the root bus.
@@ -4225,6 +4233,8 @@ static int intel_iommu_add_device(struct device *dev)
swap_pci_ref(&dma_pdev, pci_dev_get(bus->self));
}
+ pr_info("dma_pdev #4: %s\n", pci_name(dma_pdev));
+
root_bus:
group = iommu_group_get(&dma_pdev->dev);
pci_dev_put(dma_pdev);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 07369f3..a6be83a 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2395,10 +2395,15 @@ static bool pci_acs_flags_enabled(struct pci_dev *pdev, u16 acs_flags)
{
int pos;
u16 cap, ctrl;
+ bool ret;
pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ACS);
- if (!pos)
- return false;
+ if (!pos) {
+ pr_info("%s no ACS capability on %s\n",
+ __func__, pci_name(pdev));
+ ret = false;
+ goto out;
+ }
/*
* Except for egress control, capabilities are either required
@@ -2409,7 +2414,11 @@ static bool pci_acs_flags_enabled(struct pci_dev *pdev, u16 acs_flags)
acs_flags &= (cap | PCI_ACS_EC);
pci_read_config_word(pdev, pos + PCI_ACS_CTRL, &ctrl);
- return (ctrl & acs_flags) == acs_flags;
+ ret = (ctrl & acs_flags) == acs_flags;
+out:
+ pr_info("%s(%s, %04x) -> %s\n", __func__, pci_name(pdev), acs_flags,
+ ret ? "true" : "false");
+ return ret;
}
/**
@@ -2432,17 +2441,24 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags)
{
int ret;
+ pr_info("%s(%s, %04x)\n", __func__, pci_name(pdev), acs_flags);
ret = pci_dev_specific_acs_enabled(pdev, acs_flags);
- if (ret >= 0)
+ if (ret >= 0) {
+ pr_info("-> %s\n", ret > 0 ? "true" : "false");
return ret > 0;
+ }
/*
* Conventional PCI and PCI-X devices never support ACS, either
* effectively or actually. The shared bus topology implies that
* any device on the bus can receive or snoop DMA.
*/
- if (!pci_is_pcie(pdev))
+ if (!pci_is_pcie(pdev)) {
+ pr_info("-> false\n");
return false;
+ }
+
+ ret = true;
switch (pci_pcie_type(pdev)) {
/*
@@ -2459,7 +2475,8 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags)
*/
case PCI_EXP_TYPE_PCI_BRIDGE:
case PCI_EXP_TYPE_RC_EC:
- return false;
+ ret = false;
+ break;
/*
* PCIe 3.0, 6.12.1.1 specifies that downstream and root ports should
* implement ACS in order to indicate their peer-to-peer capabilities,
@@ -2467,7 +2484,8 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags)
*/
case PCI_EXP_TYPE_DOWNSTREAM:
case PCI_EXP_TYPE_ROOT_PORT:
- return pci_acs_flags_enabled(pdev, acs_flags);
+ ret = pci_acs_flags_enabled(pdev, acs_flags);
+ break;
/*
* PCIe 3.0, 6.12.1.2 specifies ACS capabilities that should be
* implemented by the remaining PCIe types to indicate peer-to-peer
@@ -2482,14 +2500,15 @@ bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags)
if (!pdev->multifunction)
break;
- return pci_acs_flags_enabled(pdev, acs_flags);
+ ret = pci_acs_flags_enabled(pdev, acs_flags);
}
/*
* PCIe 3.0, 6.12.1.3 specifies no ACS capabilities are applicable
* to single function devices with the exception of downstream ports.
*/
- return true;
+ pr_info("-> %s\n", ret ? "true" : "false");
+ return ret;
}
/**
@@ -2505,20 +2524,28 @@ bool pci_acs_path_enabled(struct pci_dev *start,
struct pci_dev *end, u16 acs_flags)
{
struct pci_dev *pdev, *parent = start;
+ bool ret = true;
+ pr_info("%s(%s, %s, %04x)\n", __func__, pci_name(start),
+ end ? pci_name(end) : "NULL", acs_flags);
do {
pdev = parent;
- if (!pci_acs_enabled(pdev, acs_flags))
- return false;
+ if (!pci_acs_enabled(pdev, acs_flags)) {
+ ret = false;
+ break;
+ }
- if (pci_is_root_bus(pdev->bus))
- return (end == NULL);
+ if (pci_is_root_bus(pdev->bus)) {
+ ret = (end == NULL);
+ break;
+ }
parent = pdev->bus->self;
} while (pdev != end);
- return true;
+ pr_info("-> %s\n", ret ? "true" : "false");
+ return ret;
}
/**
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 3a02717..1e0ec58 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3423,11 +3423,130 @@ static int pci_quirk_amd_sb_acs(struct pci_dev *dev, u16 acs_flags)
#endif
}
+static bool acs_on_downstream;
+static bool acs_on_multifunction;
+
+#define NUM_ACS_IDS 16
+struct acs_on_id {
+ unsigned short vendor;
+ unsigned short device;
+};
+static struct acs_on_id acs_on_ids[NUM_ACS_IDS];
+static u8 max_acs_id;
+
+static __init int pcie_acs_override_setup(char *p)
+{
+ if (!p)
+ return -EINVAL;
+
+ while (*p) {
+ if (!strncmp(p, "downstream", 10)) {
+ acs_on_downstream = true;
+ pr_info("PCIe ACS bypass added for downstream\n");
+ }
+ if (!strncmp(p, "multifunction", 13)) {
+ acs_on_multifunction = true;
+ pr_info("PCIe ACS bypass added for multifunction\n");
+ }
+ if (!strncmp(p, "id:", 3)) {
+ char opt[5];
+ int ret;
+ long val;
+
+ if (max_acs_id >= NUM_ACS_IDS - 1) {
+ pr_warn("Out of PCIe ACS override slots (%d)\n",
+ NUM_ACS_IDS);
+ goto next;
+ }
+
+ p += 3;
+ snprintf(opt, 5, "%s", p);
+ ret = kstrtol(opt, 16, &val);
+ if (ret) {
+ pr_warn("PCIe ACS ID parse error %d\n", ret);
+ goto next;
+ }
+ acs_on_ids[max_acs_id].vendor = val;
+
+ p += strcspn(p, ":");
+ if (*p != ':') {
+ pr_warn("PCIe ACS invalid ID\n");
+ goto next;
+ }
+
+ p++;
+ snprintf(opt, 5, "%s", p);
+ ret = kstrtol(opt, 16, &val);
+ if (ret) {
+ pr_warn("PCIe ACS ID parse error %d\n", ret);
+ goto next;
+ }
+ acs_on_ids[max_acs_id].device = val;
+ pr_info("PCIe ACS bypass added for %04x:%04x\n",
+ acs_on_ids[max_acs_id].vendor,
+ acs_on_ids[max_acs_id].device);
+ max_acs_id++;
+ }
+next:
+ p += strcspn(p, ",");
+ if (*p == ',')
+ p++;
+ }
+
+ if (acs_on_downstream || acs_on_multifunction || max_acs_id)
+ pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n");
+
+ return 0;
+}
+early_param("pcie_acs_override", pcie_acs_override_setup);
+
+static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags)
+{
+ int i;
+
+ /* Never override ACS for legacy devices or devices with ACS caps */
+ if (!pci_is_pcie(dev) /* ||
+ pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS) */)
+ return -ENOTTY;
+
+ for (i = 0; i < max_acs_id; i++) {
+ if (acs_on_ids[i].vendor == dev->vendor &&
+ acs_on_ids[i].device == dev->device) {
+ pr_info("PCIe ACS override match for %s\n",
+ pci_name(dev));
+ return 1;
+ }
+ }
+
+ switch (pci_pcie_type(dev)) {
+ case PCI_EXP_TYPE_DOWNSTREAM:
+ case PCI_EXP_TYPE_ROOT_PORT:
+ if (acs_on_downstream) {
+ pr_info("PCIe ACS override downstream %s\n",
+ pci_name(dev));
+ return 1;
+ }
+ break;
+ case PCI_EXP_TYPE_ENDPOINT:
+ case PCI_EXP_TYPE_UPSTREAM:
+ case PCI_EXP_TYPE_LEG_END:
+ case PCI_EXP_TYPE_RC_END:
+ if (acs_on_multifunction && dev->multifunction) {
+ pr_info("PCIe ACS override multifunction %s\n",
+ pci_name(dev));
+ return 1;
+ }
+ }
+
+ return -ENOTTY;
+}
+
static const struct pci_dev_acs_enabled {
u16 vendor;
u16 device;
int (*acs_enabled)(struct pci_dev *dev, u16 acs_flags);
} pci_dev_acs_enabled[] = {
+ { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides },
{ PCI_VENDOR_ID_ATI, 0x4385, pci_quirk_amd_sb_acs },
{ PCI_VENDOR_ID_ATI, 0x439c, pci_quirk_amd_sb_acs },
{ PCI_VENDOR_ID_ATI, 0x4383, pci_quirk_amd_sb_acs },
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-04 0:03 ` Alex Williamson
@ 2014-01-04 19:26 ` Dana Goyette
2014-01-04 20:22 ` Alex Williamson
0 siblings, 1 reply; 15+ messages in thread
From: Dana Goyette @ 2014-01-04 19:26 UTC (permalink / raw)
To: kvm
On 01/03/2014 04:03 PM, Alex Williamson wrote:
> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>> soon decide which one to keep.
>>>>>>
>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>
>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>> remain stuck in the same group.
>>>>>>
>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>> devices on different root ports, or all devices behind the one
>>>>>> ACS-supporting switch?
>>>>> Can you provide lspci -vvv info? If you're getting that for groups
>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>> we need or we're doing something wrong. Thanks,
>>>>>
>>>> I initially tried attaching the output as a .txt file, but it's too
>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>> moved the Radeon to a different slot).
>>> Well, something seems amiss since the downstream switch ports all seem
>>> to support and enable the correct set of ACS capabilities. I'm tending
>>> to suspect something wrong with the ACS override patch or how it's being
>>> used since your IOMMU group is still based at the root port. Each root
>>> port is isolated from the other root ports though, so something is
>>> happening with the override patch. Can you provide the kernel command
>>> line you use to enable ACS overrides and the override patch you're
>>> using, as it applies to 3.13-rc5? Thanks,
>>>
>>> Alex
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> I'm using the original acs-override patch from this post:
>> https://lkml.org/lkml/2013/5/30/513
>>
>> Kernel parameter is:
>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>
>> When booting a kernel without the override patch, the following devices
>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>> controller; Intel I210 Ethernet controller.
> Could you please try the patch below and send dmesg for the system once
> booted. This applies directly to upstream and includes the acs override
> patch. Thanks,
(removed patch from quote.)
Here's the complete dmesg, with pcie_acs_override still set:
http://pastebin.com/YHuKnrTb
Most relevant section:
[ 0.524362] DMAR: No ATSR found
[ 0.524386] IOMMU 1 0xfed91000: using Queued invalidation
[ 0.524389] IOMMU: Setting RMRR:
[ 0.524398] IOMMU: Setting identity map for device 0000:00:1d.0
[0x7bea1000 - 0x7beaffff]
[ 0.524423] IOMMU: Setting identity map for device 0000:00:1a.0
[0x7bea1000 - 0x7beaffff]
[ 0.524441] IOMMU: Setting identity map for device 0000:00:14.0
[0x7bea1000 - 0x7beaffff]
[ 0.524454] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.524461] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0
- 0xffffff]
[ 0.524548] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[ 0.524551] intel_iommu_add_device(0000:00:00.0)
[ 0.524552] dma_pdev #1: 0000:00:00.0
[ 0.524553] dma_pdev #2: 0000:00:00.0
[ 0.524554] dma_pdev #3: 0000:00:00.0
[ 0.524554] dma_pdev #4: 0000:00:00.0
[ 0.524565] intel_iommu_add_device(0000:00:01.0)
[ 0.524566] dma_pdev #1: 0000:00:01.0
[ 0.524567] dma_pdev #2: 0000:00:01.0
[ 0.524569] pci_acs_enabled(0000:00:01.0, 001d)
[ 0.524572] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
[ 0.524573] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
[ 0.524574] -> false
[ 0.524575] pci_acs_enabled(0000:00:01.0, 001d)
[ 0.524577] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
[ 0.524578] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
[ 0.524579] -> false
[ 0.524580] dma_pdev #3: 0000:00:01.0
[ 0.524581] dma_pdev #4: 0000:00:01.0
[ 0.524584] intel_iommu_add_device(0000:00:01.1)
[ 0.524586] dma_pdev #1: 0000:00:01.1
[ 0.524586] dma_pdev #2: 0000:00:01.1
[ 0.524587] pci_acs_enabled(0000:00:01.1, 001d)
[ 0.524589] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
[ 0.524590] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
[ 0.524591] -> false
[ 0.524592] pci_acs_enabled(0000:00:01.0, 001d)
[ 0.524593] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
[ 0.524595] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
[ 0.524596] -> false
[ 0.524596] dma_pdev #3: 0000:00:01.0
[ 0.524597] dma_pdev #4: 0000:00:01.0
[ 0.524599] intel_iommu_add_device(0000:00:02.0)
[ 0.524601] dma_pdev #1: 0000:00:02.0
[ 0.524601] dma_pdev #2: 0000:00:02.0
[ 0.524602] dma_pdev #3: 0000:00:02.0
[ 0.524603] dma_pdev #4: 0000:00:02.0
[ 0.524606] intel_iommu_add_device(0000:00:03.0)
[ 0.524607] dma_pdev #1: 0000:00:03.0
[ 0.524608] dma_pdev #2: 0000:00:03.0
[ 0.524609] dma_pdev #3: 0000:00:03.0
[ 0.524610] dma_pdev #4: 0000:00:03.0
[ 0.524613] intel_iommu_add_device(0000:00:14.0)
[ 0.524614] dma_pdev #1: 0000:00:14.0
[ 0.524615] dma_pdev #2: 0000:00:14.0
[ 0.524616] dma_pdev #3: 0000:00:14.0
[ 0.524617] dma_pdev #4: 0000:00:14.0
[ 0.524620] intel_iommu_add_device(0000:00:16.0)
[ 0.524621] dma_pdev #1: 0000:00:16.0
[ 0.524622] dma_pdev #2: 0000:00:16.0
[ 0.524623] pci_acs_enabled(0000:00:16.0, 001d)
[ 0.524624] -> false
[ 0.524625] pci_acs_enabled(0000:00:16.0, 001d)
[ 0.524626] -> false
[ 0.524626] dma_pdev #3: 0000:00:16.0
[ 0.524627] dma_pdev #4: 0000:00:16.0
[ 0.524630] intel_iommu_add_device(0000:00:16.3)
[ 0.524631] dma_pdev #1: 0000:00:16.3
[ 0.524632] dma_pdev #2: 0000:00:16.3
[ 0.524633] pci_acs_enabled(0000:00:16.3, 001d)
[ 0.524634] -> false
[ 0.524635] pci_acs_enabled(0000:00:16.0, 001d)
[ 0.524636] -> false
[ 0.524637] dma_pdev #3: 0000:00:16.0
[ 0.524638] dma_pdev #4: 0000:00:16.0
[ 0.524640] intel_iommu_add_device(0000:00:19.0)
[ 0.524641] dma_pdev #1: 0000:00:19.0
[ 0.524642] dma_pdev #2: 0000:00:19.0
[ 0.524643] dma_pdev #3: 0000:00:19.0
[ 0.524643] dma_pdev #4: 0000:00:19.0
[ 0.524646] intel_iommu_add_device(0000:00:1a.0)
[ 0.524647] dma_pdev #1: 0000:00:1a.0
[ 0.524648] dma_pdev #2: 0000:00:1a.0
[ 0.524649] dma_pdev #3: 0000:00:1a.0
[ 0.524650] dma_pdev #4: 0000:00:1a.0
[ 0.524654] intel_iommu_add_device(0000:00:1b.0)
[ 0.524655] dma_pdev #1: 0000:00:1b.0
[ 0.524656] dma_pdev #2: 0000:00:1b.0
[ 0.524657] dma_pdev #3: 0000:00:1b.0
[ 0.524658] dma_pdev #4: 0000:00:1b.0
[ 0.524661] intel_iommu_add_device(0000:00:1c.0)
[ 0.524662] dma_pdev #1: 0000:00:1c.0
[ 0.524663] dma_pdev #2: 0000:00:1c.0
[ 0.524664] pci_acs_enabled(0000:00:1c.0, 001d)
[ 0.524665] PCIe ACS override match for 0000:00:1c.0
[ 0.524666] -> true
[ 0.524667] dma_pdev #3: 0000:00:1c.0
[ 0.524668] dma_pdev #4: 0000:00:1c.0
[ 0.524671] intel_iommu_add_device(0000:00:1c.1)
[ 0.524672] dma_pdev #1: 0000:00:1c.1
[ 0.524673] dma_pdev #2: 0000:00:1c.1
[ 0.524674] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524681] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524682] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524683] -> false
[ 0.524684] pci_acs_enabled(0000:00:1c.0, 001d)
[ 0.524685] PCIe ACS override match for 0000:00:1c.0
[ 0.524686] -> true
[ 0.524687] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524694] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524695] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524696] -> false
[ 0.524697] dma_pdev #3: 0000:00:1c.1
[ 0.524698] dma_pdev #4: 0000:00:1c.1
[ 0.524700] intel_iommu_add_device(0000:00:1c.3)
[ 0.524701] dma_pdev #1: 0000:00:1c.3
[ 0.524702] dma_pdev #2: 0000:00:1c.3
[ 0.524703] pci_acs_enabled(0000:00:1c.3, 001d)
[ 0.524704] PCIe ACS override match for 0000:00:1c.3
[ 0.524705] -> true
[ 0.524706] dma_pdev #3: 0000:00:1c.3
[ 0.524707] dma_pdev #4: 0000:00:1c.3
[ 0.524710] intel_iommu_add_device(0000:00:1c.4)
[ 0.524711] dma_pdev #1: 0000:00:1c.4
[ 0.524712] dma_pdev #2: 0000:00:1c.4
[ 0.524713] pci_acs_enabled(0000:00:1c.4, 001d)
[ 0.524714] PCIe ACS override match for 0000:00:1c.4
[ 0.524715] -> true
[ 0.524715] dma_pdev #3: 0000:00:1c.4
[ 0.524716] dma_pdev #4: 0000:00:1c.4
[ 0.524719] intel_iommu_add_device(0000:00:1d.0)
[ 0.524720] dma_pdev #1: 0000:00:1d.0
[ 0.524721] dma_pdev #2: 0000:00:1d.0
[ 0.524722] dma_pdev #3: 0000:00:1d.0
[ 0.524723] dma_pdev #4: 0000:00:1d.0
[ 0.524726] intel_iommu_add_device(0000:00:1f.0)
[ 0.524727] dma_pdev #1: 0000:00:1f.0
[ 0.524728] dma_pdev #2: 0000:00:1f.0
[ 0.524728] pci_acs_enabled(0000:00:1f.0, 001d)
[ 0.524729] -> false
[ 0.524730] pci_acs_enabled(0000:00:1f.0, 001d)
[ 0.524731] -> false
[ 0.524732] dma_pdev #3: 0000:00:1f.0
[ 0.524733] dma_pdev #4: 0000:00:1f.0
[ 0.524736] intel_iommu_add_device(0000:00:1f.2)
[ 0.524737] dma_pdev #1: 0000:00:1f.2
[ 0.524738] dma_pdev #2: 0000:00:1f.2
[ 0.524739] pci_acs_enabled(0000:00:1f.2, 001d)
[ 0.524740] -> false
[ 0.524741] pci_acs_enabled(0000:00:1f.0, 001d)
[ 0.524741] -> false
[ 0.524742] dma_pdev #3: 0000:00:1f.0
[ 0.524743] dma_pdev #4: 0000:00:1f.0
[ 0.524745] intel_iommu_add_device(0000:00:1f.3)
[ 0.524746] dma_pdev #1: 0000:00:1f.3
[ 0.524747] dma_pdev #2: 0000:00:1f.3
[ 0.524748] pci_acs_enabled(0000:00:1f.3, 001d)
[ 0.524749] -> false
[ 0.524750] pci_acs_enabled(0000:00:1f.0, 001d)
[ 0.524751] -> false
[ 0.524752] dma_pdev #3: 0000:00:1f.0
[ 0.524752] dma_pdev #4: 0000:00:1f.0
[ 0.524754] intel_iommu_add_device(0000:00:1f.6)
[ 0.524755] dma_pdev #1: 0000:00:1f.6
[ 0.524756] dma_pdev #2: 0000:00:1f.6
[ 0.524757] pci_acs_enabled(0000:00:1f.6, 001d)
[ 0.524758] -> false
[ 0.524759] pci_acs_enabled(0000:00:1f.0, 001d)
[ 0.524760] -> false
[ 0.524761] dma_pdev #3: 0000:00:1f.0
[ 0.524762] dma_pdev #4: 0000:00:1f.0
[ 0.524765] intel_iommu_add_device(0000:01:00.0)
[ 0.524766] dma_pdev #1: 0000:01:00.0
[ 0.524767] dma_pdev #2: 0000:01:00.0
[ 0.524768] pci_acs_enabled(0000:01:00.0, 001d)
[ 0.524771] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
[ 0.524772] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
[ 0.524773] -> false
[ 0.524774] pci_acs_enabled(0000:01:00.0, 001d)
[ 0.524776] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
[ 0.524777] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
[ 0.524778] -> false
[ 0.524779] dma_pdev #3: 0000:01:00.0
[ 0.524780] pci_acs_path_enabled(0000:00:01.0, NULL, 001d)
[ 0.524781] pci_acs_enabled(0000:00:01.0, 001d)
[ 0.524783] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
[ 0.524784] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
[ 0.524785] -> false
[ 0.524786] -> false
[ 0.524787] dma_pdev #4: 0000:00:01.0
[ 0.524789] intel_iommu_add_device(0000:01:00.1)
[ 0.524790] dma_pdev #1: 0000:01:00.1
[ 0.524791] dma_pdev #2: 0000:01:00.1
[ 0.524792] pci_acs_enabled(0000:01:00.1, 001d)
[ 0.524794] pci_acs_flags_enabled no ACS capability on 0000:01:00.1
[ 0.524795] pci_acs_flags_enabled(0000:01:00.1, 001d) -> false
[ 0.524796] -> false
[ 0.524797] pci_acs_enabled(0000:01:00.0, 001d)
[ 0.524799] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
[ 0.524800] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
[ 0.524801] -> false
[ 0.524802] dma_pdev #3: 0000:01:00.0
[ 0.524803] pci_acs_path_enabled(0000:00:01.0, NULL, 001d)
[ 0.524804] pci_acs_enabled(0000:00:01.0, 001d)
[ 0.524806] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
[ 0.524807] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
[ 0.524808] -> false
[ 0.524809] -> false
[ 0.524809] dma_pdev #4: 0000:00:01.0
[ 0.524811] intel_iommu_add_device(0000:02:00.0)
[ 0.524813] dma_pdev #1: 0000:02:00.0
[ 0.524813] dma_pdev #2: 0000:02:00.0
[ 0.524814] dma_pdev #3: 0000:02:00.0
[ 0.524815] pci_acs_path_enabled(0000:00:01.1, NULL, 001d)
[ 0.524816] pci_acs_enabled(0000:00:01.1, 001d)
[ 0.524818] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
[ 0.524819] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
[ 0.524820] -> false
[ 0.524821] -> false
[ 0.524822] dma_pdev #4: 0000:00:01.1
[ 0.524824] intel_iommu_add_device(0000:03:04.0)
[ 0.524825] Upstream bridge 0000:02:00.0
[ 0.524830] dma_pdev #1: 0000:02:00.0
[ 0.524831] dma_pdev #2: 0000:02:00.0
[ 0.524832] dma_pdev #3: 0000:02:00.0
[ 0.524833] pci_acs_path_enabled(0000:00:01.1, NULL, 001d)
[ 0.524834] pci_acs_enabled(0000:00:01.1, 001d)
[ 0.524835] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
[ 0.524837] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
[ 0.524837] -> false
[ 0.524838] -> false
[ 0.524839] dma_pdev #4: 0000:00:01.1
[ 0.524841] intel_iommu_add_device(0000:04:00.0)
[ 0.524842] dma_pdev #1: 0000:04:00.0
[ 0.524843] dma_pdev #2: 0000:04:00.0
[ 0.524844] dma_pdev #3: 0000:04:00.0
[ 0.524845] pci_acs_path_enabled(0000:00:1c.0, NULL, 001d)
[ 0.524846] pci_acs_enabled(0000:00:1c.0, 001d)
[ 0.524847] PCIe ACS override match for 0000:00:1c.0
[ 0.524848] -> true
[ 0.524849] -> true
[ 0.524849] dma_pdev #4: 0000:04:00.0
[ 0.524852] intel_iommu_add_device(0000:05:00.0)
[ 0.524853] dma_pdev #1: 0000:05:00.0
[ 0.524854] dma_pdev #2: 0000:05:00.0
[ 0.524855] dma_pdev #3: 0000:05:00.0
[ 0.524856] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.524857] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524865] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524866] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524867] -> false
[ 0.524868] -> false
[ 0.524869] dma_pdev #4: 0000:00:1c.1
[ 0.524871] intel_iommu_add_device(0000:06:01.0)
[ 0.524872] dma_pdev #1: 0000:06:01.0
[ 0.524872] dma_pdev #2: 0000:06:01.0
[ 0.524873] dma_pdev #3: 0000:06:01.0
[ 0.524874] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.524875] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.524876] PCIe ACS override match for 0000:05:00.0
[ 0.524877] -> true
[ 0.524878] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524886] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524887] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524888] -> false
[ 0.524888] -> false
[ 0.524889] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.524891] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524898] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524899] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524900] -> false
[ 0.524901] -> false
[ 0.524901] dma_pdev #4: 0000:00:1c.1
[ 0.524903] intel_iommu_add_device(0000:06:04.0)
[ 0.524904] dma_pdev #1: 0000:06:04.0
[ 0.524905] dma_pdev #2: 0000:06:04.0
[ 0.524906] dma_pdev #3: 0000:06:04.0
[ 0.524907] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.524908] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.524909] PCIe ACS override match for 0000:05:00.0
[ 0.524910] -> true
[ 0.524911] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524913] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524914] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524915] -> false
[ 0.524916] -> false
[ 0.524917] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.524918] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524925] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524926] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524927] -> false
[ 0.524928] -> false
[ 0.524929] dma_pdev #4: 0000:00:1c.1
[ 0.524949] intel_iommu_add_device(0000:06:05.0)
[ 0.524951] dma_pdev #1: 0000:06:05.0
[ 0.524951] dma_pdev #2: 0000:06:05.0
[ 0.524952] dma_pdev #3: 0000:06:05.0
[ 0.524953] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.524954] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.524955] PCIe ACS override match for 0000:05:00.0
[ 0.524957] -> true
[ 0.524958] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524964] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524965] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524966] -> false
[ 0.524967] -> false
[ 0.524968] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.524969] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524976] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524977] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524978] -> false
[ 0.524979] -> false
[ 0.524979] dma_pdev #4: 0000:00:1c.1
[ 0.524983] intel_iommu_add_device(0000:06:07.0)
[ 0.524984] dma_pdev #1: 0000:06:07.0
[ 0.524985] dma_pdev #2: 0000:06:07.0
[ 0.524986] dma_pdev #3: 0000:06:07.0
[ 0.524987] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.524988] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.524989] PCIe ACS override match for 0000:05:00.0
[ 0.524990] -> true
[ 0.524991] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.524997] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.524998] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.524999] -> false
[ 0.525000] -> false
[ 0.525001] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.525002] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525004] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525006] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525007] -> false
[ 0.525007] -> false
[ 0.525008] dma_pdev #4: 0000:00:1c.1
[ 0.525010] intel_iommu_add_device(0000:06:09.0)
[ 0.525011] dma_pdev #1: 0000:06:09.0
[ 0.525012] dma_pdev #2: 0000:06:09.0
[ 0.525013] dma_pdev #3: 0000:06:09.0
[ 0.525014] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.525015] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525016] PCIe ACS override match for 0000:05:00.0
[ 0.525017] -> true
[ 0.525018] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525025] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525026] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525027] -> false
[ 0.525028] -> false
[ 0.525029] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.525030] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525037] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525038] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525039] -> false
[ 0.525040] -> false
[ 0.525041] dma_pdev #4: 0000:00:1c.1
[ 0.525043] intel_iommu_add_device(0000:0a:00.0)
[ 0.525044] dma_pdev #1: 0000:0a:00.0
[ 0.525044] dma_pdev #2: 0000:0a:00.0
[ 0.525045] dma_pdev #3: 0000:0a:00.0
[ 0.525046] pci_acs_path_enabled(0000:06:07.0, NULL, 001d)
[ 0.525048] pci_acs_enabled(0000:06:07.0, 001d)
[ 0.525048] PCIe ACS override match for 0000:06:07.0
[ 0.525049] -> true
[ 0.525050] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525051] PCIe ACS override match for 0000:05:00.0
[ 0.525052] -> true
[ 0.525053] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525060] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525062] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525062] -> false
[ 0.525063] -> false
[ 0.525064] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.525065] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525066] PCIe ACS override match for 0000:05:00.0
[ 0.525067] -> true
[ 0.525068] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525075] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525076] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525077] -> false
[ 0.525078] -> false
[ 0.525079] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.525080] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525087] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525088] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525089] -> false
[ 0.525090] -> false
[ 0.525091] dma_pdev #4: 0000:00:1c.1
[ 0.525093] intel_iommu_add_device(0000:0b:00.0)
[ 0.525094] dma_pdev #1: 0000:0b:00.0
[ 0.525095] dma_pdev #2: 0000:0b:00.0
[ 0.525096] dma_pdev #3: 0000:0b:00.0
[ 0.525097] pci_acs_path_enabled(0000:06:09.0, NULL, 001d)
[ 0.525098] pci_acs_enabled(0000:06:09.0, 001d)
[ 0.525099] PCIe ACS override match for 0000:06:09.0
[ 0.525100] -> true
[ 0.525101] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525102] PCIe ACS override match for 0000:05:00.0
[ 0.525103] -> true
[ 0.525103] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525111] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525112] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525113] -> false
[ 0.525114] -> false
[ 0.525115] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.525116] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525117] PCIe ACS override match for 0000:05:00.0
[ 0.525118] -> true
[ 0.525118] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525125] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525126] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525127] -> false
[ 0.525128] -> false
[ 0.525129] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.525130] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525137] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525138] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525139] -> false
[ 0.525140] -> false
[ 0.525141] dma_pdev #4: 0000:00:1c.1
[ 0.525143] intel_iommu_add_device(0000:0c:00.0)
[ 0.525144] Upstream bridge 0000:0b:00.0
[ 0.525148] dma_pdev #1: 0000:0c:00.0
[ 0.525149] dma_pdev #2: 0000:0c:00.0
[ 0.525150] dma_pdev #3: 0000:0c:00.0
[ 0.525151] pci_acs_path_enabled(0000:0b:00.0, NULL, 001d)
[ 0.525152] pci_acs_enabled(0000:0b:00.0, 001d)
[ 0.525153] -> false
[ 0.525154] -> false
[ 0.525155] pci_acs_path_enabled(0000:06:09.0, NULL, 001d)
[ 0.525156] pci_acs_enabled(0000:06:09.0, 001d)
[ 0.525157] PCIe ACS override match for 0000:06:09.0
[ 0.525158] -> true
[ 0.525158] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525159] PCIe ACS override match for 0000:05:00.0
[ 0.525160] -> true
[ 0.525161] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525168] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525169] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525170] -> false
[ 0.525171] -> false
[ 0.525172] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
[ 0.525173] pci_acs_enabled(0000:05:00.0, 001d)
[ 0.525174] PCIe ACS override match for 0000:05:00.0
[ 0.525175] -> true
[ 0.525175] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525178] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525179] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525180] -> false
[ 0.525181] -> false
[ 0.525182] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
[ 0.525183] pci_acs_enabled(0000:00:1c.1, 001d)
[ 0.525190] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
[ 0.525191] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
[ 0.525192] -> false
[ 0.525193] -> false
[ 0.525193] dma_pdev #4: 0000:00:1c.1
[ 0.525195] intel_iommu_add_device(0000:0d:00.0)
[ 0.525196] dma_pdev #1: 0000:0d:00.0
[ 0.525197] dma_pdev #2: 0000:0d:00.0
[ 0.525198] dma_pdev #3: 0000:0d:00.0
[ 0.525199] pci_acs_path_enabled(0000:00:1c.3, NULL, 001d)
[ 0.525200] pci_acs_enabled(0000:00:1c.3, 001d)
[ 0.525201] PCIe ACS override match for 0000:00:1c.3
[ 0.525202] -> true
[ 0.525203] -> true
[ 0.525204] dma_pdev #4: 0000:0d:00.0
lspci as slots are now (since Radeon is back in first x16 slot):
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor
DRAM Controller [8086:0c08] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200
v3 Processor Integrated Graphics Controller [8086:041a] (rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series
Chipset Family KT Controller [8086:8c3d] (rev 04)
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet
Connection I217-LM [8086:153a] (rev 05)
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series
Chipset High Definition Audio Controller [8086:8c20] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5)
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family
Server Advanced SKU LPC Controller [8086:8c56] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series
Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset
Family SMBus Controller [8086:8c22] (rev 05)
00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series
Chipset Family Thermal Management Controller [8086:8c24] (rev 05)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]
02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI
Express-to-PCI Bridge [10b5:8112] (rev aa)
03:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc
CMI8788 [Oxygen HD Audio] [13f6:8788]
04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial
ATA Controller [1b21:0612] (rev 01)
05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB
3.0 Host Controller [1912:0014] (rev 03)
0b:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI
Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01)
0c:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments
XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
[104c:823f] (rev 01)
0d:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit
Network Connection [8086:1533] (rev 03)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-04 19:26 ` Dana Goyette
@ 2014-01-04 20:22 ` Alex Williamson
2014-01-04 21:11 ` Dana Goyette
0 siblings, 1 reply; 15+ messages in thread
From: Alex Williamson @ 2014-01-04 20:22 UTC (permalink / raw)
To: Dana Goyette; +Cc: kvm
On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote:
> On 01/03/2014 04:03 PM, Alex Williamson wrote:
> > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> >> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>>>> soon decide which one to keep.
> >>>>>>
> >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
> >>>>>> downstream of the PLX switch to be in separate groups.
> >>>>>>
> >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>>>> applied and set for the Intel root ports, the devices behind the switch
> >>>>>> remain stuck in the same group.
> >>>>>>
> >>>>>> In terms of passing devices to different VMs, which is better: all
> >>>>>> devices on different root ports, or all devices behind the one
> >>>>>> ACS-supporting switch?
> >>>>> Can you provide lspci -vvv info? If you're getting that for groups
> >>>>> either the switch has ACS capabilities, but doesn't support the features
> >>>>> we need or we're doing something wrong. Thanks,
> >>>>>
> >>>> I initially tried attaching the output as a .txt file, but it's too
> >>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
> >>>> moved the Radeon to a different slot).
> >>> Well, something seems amiss since the downstream switch ports all seem
> >>> to support and enable the correct set of ACS capabilities. I'm tending
> >>> to suspect something wrong with the ACS override patch or how it's being
> >>> used since your IOMMU group is still based at the root port. Each root
> >>> port is isolated from the other root ports though, so something is
> >>> happening with the override patch. Can you provide the kernel command
> >>> line you use to enable ACS overrides and the override patch you're
> >>> using, as it applies to 3.13-rc5? Thanks,
> >>>
> >>> Alex
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>
> >> I'm using the original acs-override patch from this post:
> >> https://lkml.org/lkml/2013/5/30/513
> >>
> >> Kernel parameter is:
> >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
Actually, you're not:
pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606
And we register all of them:
[ 0.000000] PCIe ACS bypass added for 8086:8c10
[ 0.000000] PCIe ACS bypass added for 8086:8c16
[ 0.000000] PCIe ACS bypass added for 8086:8c18
[ 0.000000] PCIe ACS bypass added for 8086:ac1a
[ 0.000000] PCIe ACS bypass added for 8086:8c1c
[ 0.000000] PCIe ACS bypass added for 8086:8c1e
[ 0.000000] PCIe ACS bypass added for 10b5:8606
However, note that the root port causing you trouble is 8086:8c12, which
isn't provided as an override, therefore the code is doing the right
thing and grouping all devices behind that root port together.
> >>
> >> When booting a kernel without the override patch, the following devices
> >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> >> controller; Intel I210 Ethernet controller.
> > Could you please try the patch below and send dmesg for the system once
> > booted. This applies directly to upstream and includes the acs override
> > patch. Thanks,
>
> (removed patch from quote.)
>
> Here's the complete dmesg, with pcie_acs_override still set:
>
> http://pastebin.com/YHuKnrTb
>
> Most relevant section:
>
> [ 0.524362] DMAR: No ATSR found
> [ 0.524386] IOMMU 1 0xfed91000: using Queued invalidation
> [ 0.524389] IOMMU: Setting RMRR:
> [ 0.524398] IOMMU: Setting identity map for device 0000:00:1d.0
> [0x7bea1000 - 0x7beaffff]
> [ 0.524423] IOMMU: Setting identity map for device 0000:00:1a.0
> [0x7bea1000 - 0x7beaffff]
> [ 0.524441] IOMMU: Setting identity map for device 0000:00:14.0
> [0x7bea1000 - 0x7beaffff]
> [ 0.524454] IOMMU: Prepare 0-16MiB unity mapping for LPC
> [ 0.524461] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0
> - 0xffffff]
> [ 0.524548] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
> [ 0.524551] intel_iommu_add_device(0000:00:00.0)
> [ 0.524552] dma_pdev #1: 0000:00:00.0
> [ 0.524553] dma_pdev #2: 0000:00:00.0
> [ 0.524554] dma_pdev #3: 0000:00:00.0
> [ 0.524554] dma_pdev #4: 0000:00:00.0
> [ 0.524565] intel_iommu_add_device(0000:00:01.0)
> [ 0.524566] dma_pdev #1: 0000:00:01.0
> [ 0.524567] dma_pdev #2: 0000:00:01.0
> [ 0.524569] pci_acs_enabled(0000:00:01.0, 001d)
> [ 0.524572] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
> [ 0.524573] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
> [ 0.524574] -> false
> [ 0.524575] pci_acs_enabled(0000:00:01.0, 001d)
> [ 0.524577] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
> [ 0.524578] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
> [ 0.524579] -> false
> [ 0.524580] dma_pdev #3: 0000:00:01.0
> [ 0.524581] dma_pdev #4: 0000:00:01.0
> [ 0.524584] intel_iommu_add_device(0000:00:01.1)
> [ 0.524586] dma_pdev #1: 0000:00:01.1
> [ 0.524586] dma_pdev #2: 0000:00:01.1
> [ 0.524587] pci_acs_enabled(0000:00:01.1, 001d)
> [ 0.524589] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
> [ 0.524590] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
> [ 0.524591] -> false
> [ 0.524592] pci_acs_enabled(0000:00:01.0, 001d)
> [ 0.524593] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
> [ 0.524595] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
> [ 0.524596] -> false
> [ 0.524596] dma_pdev #3: 0000:00:01.0
> [ 0.524597] dma_pdev #4: 0000:00:01.0
> [ 0.524599] intel_iommu_add_device(0000:00:02.0)
> [ 0.524601] dma_pdev #1: 0000:00:02.0
> [ 0.524601] dma_pdev #2: 0000:00:02.0
> [ 0.524602] dma_pdev #3: 0000:00:02.0
> [ 0.524603] dma_pdev #4: 0000:00:02.0
> [ 0.524606] intel_iommu_add_device(0000:00:03.0)
> [ 0.524607] dma_pdev #1: 0000:00:03.0
> [ 0.524608] dma_pdev #2: 0000:00:03.0
> [ 0.524609] dma_pdev #3: 0000:00:03.0
> [ 0.524610] dma_pdev #4: 0000:00:03.0
> [ 0.524613] intel_iommu_add_device(0000:00:14.0)
> [ 0.524614] dma_pdev #1: 0000:00:14.0
> [ 0.524615] dma_pdev #2: 0000:00:14.0
> [ 0.524616] dma_pdev #3: 0000:00:14.0
> [ 0.524617] dma_pdev #4: 0000:00:14.0
> [ 0.524620] intel_iommu_add_device(0000:00:16.0)
> [ 0.524621] dma_pdev #1: 0000:00:16.0
> [ 0.524622] dma_pdev #2: 0000:00:16.0
> [ 0.524623] pci_acs_enabled(0000:00:16.0, 001d)
> [ 0.524624] -> false
> [ 0.524625] pci_acs_enabled(0000:00:16.0, 001d)
> [ 0.524626] -> false
> [ 0.524626] dma_pdev #3: 0000:00:16.0
> [ 0.524627] dma_pdev #4: 0000:00:16.0
> [ 0.524630] intel_iommu_add_device(0000:00:16.3)
> [ 0.524631] dma_pdev #1: 0000:00:16.3
> [ 0.524632] dma_pdev #2: 0000:00:16.3
> [ 0.524633] pci_acs_enabled(0000:00:16.3, 001d)
> [ 0.524634] -> false
> [ 0.524635] pci_acs_enabled(0000:00:16.0, 001d)
> [ 0.524636] -> false
> [ 0.524637] dma_pdev #3: 0000:00:16.0
> [ 0.524638] dma_pdev #4: 0000:00:16.0
> [ 0.524640] intel_iommu_add_device(0000:00:19.0)
> [ 0.524641] dma_pdev #1: 0000:00:19.0
> [ 0.524642] dma_pdev #2: 0000:00:19.0
> [ 0.524643] dma_pdev #3: 0000:00:19.0
> [ 0.524643] dma_pdev #4: 0000:00:19.0
> [ 0.524646] intel_iommu_add_device(0000:00:1a.0)
> [ 0.524647] dma_pdev #1: 0000:00:1a.0
> [ 0.524648] dma_pdev #2: 0000:00:1a.0
> [ 0.524649] dma_pdev #3: 0000:00:1a.0
> [ 0.524650] dma_pdev #4: 0000:00:1a.0
> [ 0.524654] intel_iommu_add_device(0000:00:1b.0)
> [ 0.524655] dma_pdev #1: 0000:00:1b.0
> [ 0.524656] dma_pdev #2: 0000:00:1b.0
> [ 0.524657] dma_pdev #3: 0000:00:1b.0
> [ 0.524658] dma_pdev #4: 0000:00:1b.0
> [ 0.524661] intel_iommu_add_device(0000:00:1c.0)
> [ 0.524662] dma_pdev #1: 0000:00:1c.0
> [ 0.524663] dma_pdev #2: 0000:00:1c.0
> [ 0.524664] pci_acs_enabled(0000:00:1c.0, 001d)
> [ 0.524665] PCIe ACS override match for 0000:00:1c.0
> [ 0.524666] -> true
> [ 0.524667] dma_pdev #3: 0000:00:1c.0
> [ 0.524668] dma_pdev #4: 0000:00:1c.0
> [ 0.524671] intel_iommu_add_device(0000:00:1c.1)
> [ 0.524672] dma_pdev #1: 0000:00:1c.1
> [ 0.524673] dma_pdev #2: 0000:00:1c.1
> [ 0.524674] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524681] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524682] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524683] -> false
> [ 0.524684] pci_acs_enabled(0000:00:1c.0, 001d)
> [ 0.524685] PCIe ACS override match for 0000:00:1c.0
> [ 0.524686] -> true
> [ 0.524687] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524694] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524695] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524696] -> false
> [ 0.524697] dma_pdev #3: 0000:00:1c.1
> [ 0.524698] dma_pdev #4: 0000:00:1c.1
> [ 0.524700] intel_iommu_add_device(0000:00:1c.3)
> [ 0.524701] dma_pdev #1: 0000:00:1c.3
> [ 0.524702] dma_pdev #2: 0000:00:1c.3
> [ 0.524703] pci_acs_enabled(0000:00:1c.3, 001d)
> [ 0.524704] PCIe ACS override match for 0000:00:1c.3
> [ 0.524705] -> true
> [ 0.524706] dma_pdev #3: 0000:00:1c.3
> [ 0.524707] dma_pdev #4: 0000:00:1c.3
> [ 0.524710] intel_iommu_add_device(0000:00:1c.4)
> [ 0.524711] dma_pdev #1: 0000:00:1c.4
> [ 0.524712] dma_pdev #2: 0000:00:1c.4
> [ 0.524713] pci_acs_enabled(0000:00:1c.4, 001d)
> [ 0.524714] PCIe ACS override match for 0000:00:1c.4
> [ 0.524715] -> true
> [ 0.524715] dma_pdev #3: 0000:00:1c.4
> [ 0.524716] dma_pdev #4: 0000:00:1c.4
> [ 0.524719] intel_iommu_add_device(0000:00:1d.0)
> [ 0.524720] dma_pdev #1: 0000:00:1d.0
> [ 0.524721] dma_pdev #2: 0000:00:1d.0
> [ 0.524722] dma_pdev #3: 0000:00:1d.0
> [ 0.524723] dma_pdev #4: 0000:00:1d.0
> [ 0.524726] intel_iommu_add_device(0000:00:1f.0)
> [ 0.524727] dma_pdev #1: 0000:00:1f.0
> [ 0.524728] dma_pdev #2: 0000:00:1f.0
> [ 0.524728] pci_acs_enabled(0000:00:1f.0, 001d)
> [ 0.524729] -> false
> [ 0.524730] pci_acs_enabled(0000:00:1f.0, 001d)
> [ 0.524731] -> false
> [ 0.524732] dma_pdev #3: 0000:00:1f.0
> [ 0.524733] dma_pdev #4: 0000:00:1f.0
> [ 0.524736] intel_iommu_add_device(0000:00:1f.2)
> [ 0.524737] dma_pdev #1: 0000:00:1f.2
> [ 0.524738] dma_pdev #2: 0000:00:1f.2
> [ 0.524739] pci_acs_enabled(0000:00:1f.2, 001d)
> [ 0.524740] -> false
> [ 0.524741] pci_acs_enabled(0000:00:1f.0, 001d)
> [ 0.524741] -> false
> [ 0.524742] dma_pdev #3: 0000:00:1f.0
> [ 0.524743] dma_pdev #4: 0000:00:1f.0
> [ 0.524745] intel_iommu_add_device(0000:00:1f.3)
> [ 0.524746] dma_pdev #1: 0000:00:1f.3
> [ 0.524747] dma_pdev #2: 0000:00:1f.3
> [ 0.524748] pci_acs_enabled(0000:00:1f.3, 001d)
> [ 0.524749] -> false
> [ 0.524750] pci_acs_enabled(0000:00:1f.0, 001d)
> [ 0.524751] -> false
> [ 0.524752] dma_pdev #3: 0000:00:1f.0
> [ 0.524752] dma_pdev #4: 0000:00:1f.0
> [ 0.524754] intel_iommu_add_device(0000:00:1f.6)
> [ 0.524755] dma_pdev #1: 0000:00:1f.6
> [ 0.524756] dma_pdev #2: 0000:00:1f.6
> [ 0.524757] pci_acs_enabled(0000:00:1f.6, 001d)
> [ 0.524758] -> false
> [ 0.524759] pci_acs_enabled(0000:00:1f.0, 001d)
> [ 0.524760] -> false
> [ 0.524761] dma_pdev #3: 0000:00:1f.0
> [ 0.524762] dma_pdev #4: 0000:00:1f.0
> [ 0.524765] intel_iommu_add_device(0000:01:00.0)
> [ 0.524766] dma_pdev #1: 0000:01:00.0
> [ 0.524767] dma_pdev #2: 0000:01:00.0
> [ 0.524768] pci_acs_enabled(0000:01:00.0, 001d)
> [ 0.524771] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
> [ 0.524772] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
> [ 0.524773] -> false
> [ 0.524774] pci_acs_enabled(0000:01:00.0, 001d)
> [ 0.524776] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
> [ 0.524777] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
> [ 0.524778] -> false
> [ 0.524779] dma_pdev #3: 0000:01:00.0
> [ 0.524780] pci_acs_path_enabled(0000:00:01.0, NULL, 001d)
> [ 0.524781] pci_acs_enabled(0000:00:01.0, 001d)
> [ 0.524783] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
> [ 0.524784] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
> [ 0.524785] -> false
> [ 0.524786] -> false
> [ 0.524787] dma_pdev #4: 0000:00:01.0
> [ 0.524789] intel_iommu_add_device(0000:01:00.1)
> [ 0.524790] dma_pdev #1: 0000:01:00.1
> [ 0.524791] dma_pdev #2: 0000:01:00.1
> [ 0.524792] pci_acs_enabled(0000:01:00.1, 001d)
> [ 0.524794] pci_acs_flags_enabled no ACS capability on 0000:01:00.1
> [ 0.524795] pci_acs_flags_enabled(0000:01:00.1, 001d) -> false
> [ 0.524796] -> false
> [ 0.524797] pci_acs_enabled(0000:01:00.0, 001d)
> [ 0.524799] pci_acs_flags_enabled no ACS capability on 0000:01:00.0
> [ 0.524800] pci_acs_flags_enabled(0000:01:00.0, 001d) -> false
> [ 0.524801] -> false
> [ 0.524802] dma_pdev #3: 0000:01:00.0
> [ 0.524803] pci_acs_path_enabled(0000:00:01.0, NULL, 001d)
> [ 0.524804] pci_acs_enabled(0000:00:01.0, 001d)
> [ 0.524806] pci_acs_flags_enabled no ACS capability on 0000:00:01.0
> [ 0.524807] pci_acs_flags_enabled(0000:00:01.0, 001d) -> false
> [ 0.524808] -> false
> [ 0.524809] -> false
> [ 0.524809] dma_pdev #4: 0000:00:01.0
> [ 0.524811] intel_iommu_add_device(0000:02:00.0)
> [ 0.524813] dma_pdev #1: 0000:02:00.0
> [ 0.524813] dma_pdev #2: 0000:02:00.0
> [ 0.524814] dma_pdev #3: 0000:02:00.0
> [ 0.524815] pci_acs_path_enabled(0000:00:01.1, NULL, 001d)
> [ 0.524816] pci_acs_enabled(0000:00:01.1, 001d)
> [ 0.524818] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
> [ 0.524819] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
> [ 0.524820] -> false
> [ 0.524821] -> false
> [ 0.524822] dma_pdev #4: 0000:00:01.1
> [ 0.524824] intel_iommu_add_device(0000:03:04.0)
> [ 0.524825] Upstream bridge 0000:02:00.0
> [ 0.524830] dma_pdev #1: 0000:02:00.0
> [ 0.524831] dma_pdev #2: 0000:02:00.0
> [ 0.524832] dma_pdev #3: 0000:02:00.0
> [ 0.524833] pci_acs_path_enabled(0000:00:01.1, NULL, 001d)
> [ 0.524834] pci_acs_enabled(0000:00:01.1, 001d)
> [ 0.524835] pci_acs_flags_enabled no ACS capability on 0000:00:01.1
> [ 0.524837] pci_acs_flags_enabled(0000:00:01.1, 001d) -> false
> [ 0.524837] -> false
> [ 0.524838] -> false
> [ 0.524839] dma_pdev #4: 0000:00:01.1
> [ 0.524841] intel_iommu_add_device(0000:04:00.0)
> [ 0.524842] dma_pdev #1: 0000:04:00.0
> [ 0.524843] dma_pdev #2: 0000:04:00.0
> [ 0.524844] dma_pdev #3: 0000:04:00.0
> [ 0.524845] pci_acs_path_enabled(0000:00:1c.0, NULL, 001d)
> [ 0.524846] pci_acs_enabled(0000:00:1c.0, 001d)
> [ 0.524847] PCIe ACS override match for 0000:00:1c.0
> [ 0.524848] -> true
> [ 0.524849] -> true
> [ 0.524849] dma_pdev #4: 0000:04:00.0
> [ 0.524852] intel_iommu_add_device(0000:05:00.0)
> [ 0.524853] dma_pdev #1: 0000:05:00.0
> [ 0.524854] dma_pdev #2: 0000:05:00.0
> [ 0.524855] dma_pdev #3: 0000:05:00.0
> [ 0.524856] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.524857] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524865] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524866] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524867] -> false
> [ 0.524868] -> false
> [ 0.524869] dma_pdev #4: 0000:00:1c.1
> [ 0.524871] intel_iommu_add_device(0000:06:01.0)
> [ 0.524872] dma_pdev #1: 0000:06:01.0
> [ 0.524872] dma_pdev #2: 0000:06:01.0
> [ 0.524873] dma_pdev #3: 0000:06:01.0
> [ 0.524874] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.524875] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.524876] PCIe ACS override match for 0000:05:00.0
> [ 0.524877] -> true
> [ 0.524878] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524886] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524887] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524888] -> false
> [ 0.524888] -> false
> [ 0.524889] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.524891] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524898] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524899] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524900] -> false
> [ 0.524901] -> false
> [ 0.524901] dma_pdev #4: 0000:00:1c.1
> [ 0.524903] intel_iommu_add_device(0000:06:04.0)
> [ 0.524904] dma_pdev #1: 0000:06:04.0
> [ 0.524905] dma_pdev #2: 0000:06:04.0
> [ 0.524906] dma_pdev #3: 0000:06:04.0
> [ 0.524907] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.524908] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.524909] PCIe ACS override match for 0000:05:00.0
> [ 0.524910] -> true
> [ 0.524911] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524913] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524914] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524915] -> false
> [ 0.524916] -> false
> [ 0.524917] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.524918] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524925] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524926] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524927] -> false
> [ 0.524928] -> false
> [ 0.524929] dma_pdev #4: 0000:00:1c.1
> [ 0.524949] intel_iommu_add_device(0000:06:05.0)
> [ 0.524951] dma_pdev #1: 0000:06:05.0
> [ 0.524951] dma_pdev #2: 0000:06:05.0
> [ 0.524952] dma_pdev #3: 0000:06:05.0
> [ 0.524953] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.524954] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.524955] PCIe ACS override match for 0000:05:00.0
> [ 0.524957] -> true
> [ 0.524958] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524964] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524965] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524966] -> false
> [ 0.524967] -> false
> [ 0.524968] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.524969] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524976] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524977] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524978] -> false
> [ 0.524979] -> false
> [ 0.524979] dma_pdev #4: 0000:00:1c.1
> [ 0.524983] intel_iommu_add_device(0000:06:07.0)
> [ 0.524984] dma_pdev #1: 0000:06:07.0
> [ 0.524985] dma_pdev #2: 0000:06:07.0
> [ 0.524986] dma_pdev #3: 0000:06:07.0
> [ 0.524987] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.524988] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.524989] PCIe ACS override match for 0000:05:00.0
> [ 0.524990] -> true
> [ 0.524991] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.524997] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.524998] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.524999] -> false
> [ 0.525000] -> false
> [ 0.525001] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.525002] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525004] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525006] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525007] -> false
> [ 0.525007] -> false
> [ 0.525008] dma_pdev #4: 0000:00:1c.1
> [ 0.525010] intel_iommu_add_device(0000:06:09.0)
> [ 0.525011] dma_pdev #1: 0000:06:09.0
> [ 0.525012] dma_pdev #2: 0000:06:09.0
> [ 0.525013] dma_pdev #3: 0000:06:09.0
> [ 0.525014] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.525015] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525016] PCIe ACS override match for 0000:05:00.0
> [ 0.525017] -> true
> [ 0.525018] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525025] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525026] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525027] -> false
> [ 0.525028] -> false
> [ 0.525029] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.525030] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525037] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525038] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525039] -> false
> [ 0.525040] -> false
> [ 0.525041] dma_pdev #4: 0000:00:1c.1
> [ 0.525043] intel_iommu_add_device(0000:0a:00.0)
> [ 0.525044] dma_pdev #1: 0000:0a:00.0
> [ 0.525044] dma_pdev #2: 0000:0a:00.0
> [ 0.525045] dma_pdev #3: 0000:0a:00.0
> [ 0.525046] pci_acs_path_enabled(0000:06:07.0, NULL, 001d)
> [ 0.525048] pci_acs_enabled(0000:06:07.0, 001d)
> [ 0.525048] PCIe ACS override match for 0000:06:07.0
> [ 0.525049] -> true
> [ 0.525050] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525051] PCIe ACS override match for 0000:05:00.0
> [ 0.525052] -> true
> [ 0.525053] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525060] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525062] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525062] -> false
> [ 0.525063] -> false
> [ 0.525064] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.525065] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525066] PCIe ACS override match for 0000:05:00.0
> [ 0.525067] -> true
> [ 0.525068] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525075] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525076] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525077] -> false
> [ 0.525078] -> false
> [ 0.525079] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.525080] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525087] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525088] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525089] -> false
> [ 0.525090] -> false
> [ 0.525091] dma_pdev #4: 0000:00:1c.1
> [ 0.525093] intel_iommu_add_device(0000:0b:00.0)
> [ 0.525094] dma_pdev #1: 0000:0b:00.0
> [ 0.525095] dma_pdev #2: 0000:0b:00.0
> [ 0.525096] dma_pdev #3: 0000:0b:00.0
> [ 0.525097] pci_acs_path_enabled(0000:06:09.0, NULL, 001d)
> [ 0.525098] pci_acs_enabled(0000:06:09.0, 001d)
> [ 0.525099] PCIe ACS override match for 0000:06:09.0
> [ 0.525100] -> true
> [ 0.525101] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525102] PCIe ACS override match for 0000:05:00.0
> [ 0.525103] -> true
> [ 0.525103] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525111] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525112] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525113] -> false
> [ 0.525114] -> false
> [ 0.525115] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.525116] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525117] PCIe ACS override match for 0000:05:00.0
> [ 0.525118] -> true
> [ 0.525118] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525125] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525126] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525127] -> false
> [ 0.525128] -> false
> [ 0.525129] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.525130] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525137] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525138] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525139] -> false
> [ 0.525140] -> false
> [ 0.525141] dma_pdev #4: 0000:00:1c.1
> [ 0.525143] intel_iommu_add_device(0000:0c:00.0)
> [ 0.525144] Upstream bridge 0000:0b:00.0
> [ 0.525148] dma_pdev #1: 0000:0c:00.0
> [ 0.525149] dma_pdev #2: 0000:0c:00.0
> [ 0.525150] dma_pdev #3: 0000:0c:00.0
> [ 0.525151] pci_acs_path_enabled(0000:0b:00.0, NULL, 001d)
> [ 0.525152] pci_acs_enabled(0000:0b:00.0, 001d)
> [ 0.525153] -> false
> [ 0.525154] -> false
> [ 0.525155] pci_acs_path_enabled(0000:06:09.0, NULL, 001d)
> [ 0.525156] pci_acs_enabled(0000:06:09.0, 001d)
> [ 0.525157] PCIe ACS override match for 0000:06:09.0
> [ 0.525158] -> true
> [ 0.525158] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525159] PCIe ACS override match for 0000:05:00.0
> [ 0.525160] -> true
> [ 0.525161] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525168] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525169] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525170] -> false
> [ 0.525171] -> false
> [ 0.525172] pci_acs_path_enabled(0000:05:00.0, NULL, 001d)
> [ 0.525173] pci_acs_enabled(0000:05:00.0, 001d)
> [ 0.525174] PCIe ACS override match for 0000:05:00.0
> [ 0.525175] -> true
> [ 0.525175] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525178] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525179] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525180] -> false
> [ 0.525181] -> false
> [ 0.525182] pci_acs_path_enabled(0000:00:1c.1, NULL, 001d)
> [ 0.525183] pci_acs_enabled(0000:00:1c.1, 001d)
> [ 0.525190] pci_acs_flags_enabled no ACS capability on 0000:00:1c.1
> [ 0.525191] pci_acs_flags_enabled(0000:00:1c.1, 001d) -> false
> [ 0.525192] -> false
> [ 0.525193] -> false
> [ 0.525193] dma_pdev #4: 0000:00:1c.1
> [ 0.525195] intel_iommu_add_device(0000:0d:00.0)
> [ 0.525196] dma_pdev #1: 0000:0d:00.0
> [ 0.525197] dma_pdev #2: 0000:0d:00.0
> [ 0.525198] dma_pdev #3: 0000:0d:00.0
> [ 0.525199] pci_acs_path_enabled(0000:00:1c.3, NULL, 001d)
> [ 0.525200] pci_acs_enabled(0000:00:1c.3, 001d)
> [ 0.525201] PCIe ACS override match for 0000:00:1c.3
> [ 0.525202] -> true
> [ 0.525203] -> true
> [ 0.525204] dma_pdev #4: 0000:0d:00.0
>
> lspci as slots are now (since Radeon is back in first x16 slot):
>
> 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor
> DRAM Controller [8086:0c08] (rev 06)
> 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
> 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
> 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200
> v3 Processor Integrated Graphics Controller [8086:041a] (rev 06)
> 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen
> Core Processor HD Audio Controller [8086:0c0c] (rev 06)
> 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
> Chipset Family USB xHCI [8086:8c31] (rev 05)
> 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220
> Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
> 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series
> Chipset Family KT Controller [8086:8c3d] (rev 04)
> 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet
> Connection I217-LM [8086:153a] (rev 05)
> 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series
> Chipset High Definition Audio Controller [8086:8c20] (rev 05)
> 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
> Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
> 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
> Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5)
> 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
> Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
> 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series
> Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
> 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series
> Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
> 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family
> Server Advanced SKU LPC Controller [8086:8c56] (rev 05)
> 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series
> Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
> 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset
> Family SMBus Controller [8086:8c22] (rev 05)
> 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series
> Chipset Family Thermal Management Controller [8086:8c24] (rev 05)
> 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
> [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be]
> 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
> Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]
> 02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI
> Express-to-PCI Bridge [10b5:8112] (rev aa)
> 03:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc
> CMI8788 [Oxygen HD Audio] [13f6:8788]
> 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial
> ATA Controller [1b21:0612] (rev 01)
> 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port
> PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
> 0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB
> 3.0 Host Controller [1912:0014] (rev 03)
> 0b:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI
> Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01)
> 0c:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments
> XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
> [104c:823f] (rev 01)
> 0d:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit
> Network Connection [8086:1533] (rev 03)
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: IOMMU groups ... PEX8606 switch?
2014-01-04 20:22 ` Alex Williamson
@ 2014-01-04 21:11 ` Dana Goyette
0 siblings, 0 replies; 15+ messages in thread
From: Dana Goyette @ 2014-01-04 21:11 UTC (permalink / raw)
To: kvm
On 01/04/2014 12:22 PM, Alex Williamson wrote:
> On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote:
>> On 01/03/2014 04:03 PM, Alex Williamson wrote:
>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>>>> soon decide which one to keep.
>>>>>>>>
>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>> PEX8066 switch, which claims to support ACS. I'd expect the devices
>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>
>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>>>> remain stuck in the same group.
>>>>>>>>
>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>> ACS-supporting switch?
>>>>>>> Can you provide lspci -vvv info? If you're getting that for groups
>>>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>>>> we need or we're doing something wrong. Thanks,
>>>>>>>
>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>> large. Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>>>> moved the Radeon to a different slot).
>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>> to support and enable the correct set of ACS capabilities. I'm tending
>>>>> to suspect something wrong with the ACS override patch or how it's being
>>>>> used since your IOMMU group is still based at the root port. Each root
>>>>> port is isolated from the other root ports though, so something is
>>>>> happening with the override patch. Can you provide the kernel command
>>>>> line you use to enable ACS overrides and the override patch you're
>>>>> using, as it applies to 3.13-rc5? Thanks,
>>>>>
>>>>> Alex
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>> I'm using the original acs-override patch from this post:
>>>> https://lkml.org/lkml/2013/5/30/513
>>>>
>>>> Kernel parameter is:
>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>
> Actually, you're not:
>
> pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606
>
> And we register all of them:
>
> [ 0.000000] PCIe ACS bypass added for 8086:8c10
> [ 0.000000] PCIe ACS bypass added for 8086:8c16
> [ 0.000000] PCIe ACS bypass added for 8086:8c18
> [ 0.000000] PCIe ACS bypass added for 8086:ac1a
> [ 0.000000] PCIe ACS bypass added for 8086:8c1c
> [ 0.000000] PCIe ACS bypass added for 8086:8c1e
> [ 0.000000] PCIe ACS bypass added for 10b5:8606
>
> However, note that the root port causing you trouble is 8086:8c12, which
> isn't provided as an override, therefore the code is doing the right
> thing and grouping all devices behind that root port together.
>
Thanks for catching that -- I certainly missed it!
I've added the override for that root port and removed the override for
the PLX switch; now all the ports are indeed in separate groups.
Do we yet know if it'll be possible to properly isolate the Intel root
ports, without this ACS override?
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2014-01-04 21:11 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-29 2:31 IOMMU groups: better with Intel root ports, or with PEX8606 switch? Dana Goyette
2013-12-29 3:23 ` Alex Williamson
2013-12-29 7:32 ` Dana Goyette
2013-12-30 4:16 ` Alex Williamson
2013-12-31 0:13 ` IOMMU groups ... " Dana Goyette
2014-01-02 19:36 ` Alex Williamson
2014-01-02 21:01 ` Dana Goyette
2014-01-02 21:14 ` Alex Williamson
2014-01-02 21:15 ` Dana Goyette
2014-01-02 21:22 ` Alex Williamson
2014-01-02 21:25 ` Dana Goyette
2014-01-04 0:03 ` Alex Williamson
2014-01-04 19:26 ` Dana Goyette
2014-01-04 20:22 ` Alex Williamson
2014-01-04 21:11 ` Dana Goyette
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox