Re: IOMMU groups ... PEX8606 switch?

Re: IOMMU groups ... PEX8606 switch?

[Alex Williamson]

On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> > On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>> soon decide which one to keep.
> >>>>
> >>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
> >>>> downstream of the PLX switch to be in separate groups.
> >>>>
> >>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>> applied and set for the Intel root ports, the devices behind the switch
> >>>> remain stuck in the same group.
> >>>>
> >>>> In terms of passing devices to different VMs, which is better: all
> >>>> devices on different root ports, or all devices behind the one
> >>>> ACS-supporting switch?
> >>>
> >>> Can you provide lspci -vvv info?  If you're getting that for groups
> >>> either the switch has ACS capabilities, but doesn't support the features
> >>> we need or we're doing something wrong.  Thanks,
> >>>
> >> I initially tried attaching the output as a .txt file, but it's too
> >> large.  Anyway, here's the output of lspci -nnvvv (you may notice that I
> >> moved the Radeon to a different slot).
> >
> > Well, something seems amiss since the downstream switch ports all seem
> > to support and enable the correct set of ACS capabilities.  I'm tending
> > to suspect something wrong with the ACS override patch or how it's being
> > used since your IOMMU group is still based at the root port.  Each root
> > port is isolated from the other root ports though, so something is
> > happening with the override patch.  Can you provide the kernel command
> > line you use to enable ACS overrides and the override patch you're
> > using, as it applies to 3.13-rc5?  Thanks,
> >
> > Alex
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> I'm using the original acs-override patch from this post:
> https://lkml.org/lkml/2013/5/30/513
> 
> Kernel parameter is:
> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
> 
> When booting a kernel without the override patch, the following devices 
> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA 
> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire 
> controller; Intel I210 Ethernet controller.

Ok, here's my shot in the dark; we must be detecting something about the
upstream switch port to make it fail the ACS test and the only thing I
can find that might do this is if the PCI config header on the upstream
switch reported itself as a multifunction device.  Multifunction
upstream switch ports do need ACS capabilities to make sure that traffic
isn't routed back through other functions.  Single function devices do
not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
looking to see whether the byte at 0xe has the MSB set.  If it does, it
lies that it's a multifunction device.  If it doesn't I'll have to get
the dart board back out.

FWIW, you should be able to work around this by adding id:10b5:8606 to
your list of overrides.  Long term, if this is the problem, we'll want
to add a quirk to sanitize the multifunction device flag.  Thanks,

Alex

Re: IOMMU groups ... PEX8606 switch?

[Dana Goyette]

On 01/02/2014 11:36 AM, Alex Williamson wrote:
> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>> soon decide which one to keep.
>>>>>>
>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>
>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>> remain stuck in the same group.
>>>>>>
>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>> devices on different root ports, or all devices behind the one
>>>>>> ACS-supporting switch?
>>>>>
>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>> we need or we're doing something wrong.  Thanks,
>>>>>
>>>> I initially tried attaching the output as a .txt file, but it's too
>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>> moved the Radeon to a different slot).
>>>
>>> Well, something seems amiss since the downstream switch ports all seem
>>> to support and enable the correct set of ACS capabilities.  I'm tending
>>> to suspect something wrong with the ACS override patch or how it's being
>>> used since your IOMMU group is still based at the root port.  Each root
>>> port is isolated from the other root ports though, so something is
>>> happening with the override patch.  Can you provide the kernel command
>>> line you use to enable ACS overrides and the override patch you're
>>> using, as it applies to 3.13-rc5?  Thanks,
>>>
>>> Alex
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> I'm using the original acs-override patch from this post:
>> https://lkml.org/lkml/2013/5/30/513
>>
>> Kernel parameter is:
>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>
>> When booting a kernel without the override patch, the following devices
>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>> controller; Intel I210 Ethernet controller.
>
> Ok, here's my shot in the dark; we must be detecting something about the
> upstream switch port to make it fail the ACS test and the only thing I
> can find that might do this is if the PCI config header on the upstream
> switch reported itself as a multifunction device.  Multifunction
> upstream switch ports do need ACS capabilities to make sure that traffic
> isn't routed back through other functions.  Single function devices do
> not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
> looking to see whether the byte at 0xe has the MSB set.  If it does, it
> lies that it's a multifunction device.  If it doesn't I'll have to get
> the dart board back out.
>
> FWIW, you should be able to work around this by adding id:10b5:8606 to
> your list of overrides.  Long term, if this is the problem, we'll want
> to add a quirk to sanitize the multifunction device flag.  Thanks,
>
> Alex
>

04:00.0 is the ASMedia SATA controller; I'll assume you meant the 
upstream port of the PLX switch.

05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI 
Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
	Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
	Memory behind bridge: ee800000-eeafffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Upstream Port, MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 
Port PCI Express Gen 2 (5.0 GT/s) Switch
	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [148] Virtual Channel
	Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?>
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00

Re: IOMMU groups ... PEX8606 switch?

[Alex Williamson]

On Thu, 2014-01-02 at 13:01 -0800, Dana Goyette wrote:
> On 01/02/2014 11:36 AM, Alex Williamson wrote:
> > On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> >> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> >>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
> >>>>>> soon decide which one to keep.
> >>>>>>
> >>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
> >>>>>> downstream of the PLX switch to be in separate groups.
> >>>>>>
> >>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
> >>>>>> applied and set for the Intel root ports, the devices behind the switch
> >>>>>> remain stuck in the same group.
> >>>>>>
> >>>>>> In terms of passing devices to different VMs, which is better: all
> >>>>>> devices on different root ports, or all devices behind the one
> >>>>>> ACS-supporting switch?
> >>>>>
> >>>>> Can you provide lspci -vvv info?  If you're getting that for groups
> >>>>> either the switch has ACS capabilities, but doesn't support the features
> >>>>> we need or we're doing something wrong.  Thanks,
> >>>>>
> >>>> I initially tried attaching the output as a .txt file, but it's too
> >>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice that I
> >>>> moved the Radeon to a different slot).
> >>>
> >>> Well, something seems amiss since the downstream switch ports all seem
> >>> to support and enable the correct set of ACS capabilities.  I'm tending
> >>> to suspect something wrong with the ACS override patch or how it's being
> >>> used since your IOMMU group is still based at the root port.  Each root
> >>> port is isolated from the other root ports though, so something is
> >>> happening with the override patch.  Can you provide the kernel command
> >>> line you use to enable ACS overrides and the override patch you're
> >>> using, as it applies to 3.13-rc5?  Thanks,
> >>>
> >>> Alex
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >> I'm using the original acs-override patch from this post:
> >> https://lkml.org/lkml/2013/5/30/513
> >>
> >> Kernel parameter is:
> >> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
> >>
> >> When booting a kernel without the override patch, the following devices
> >> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> >> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> >> controller; Intel I210 Ethernet controller.
> >
> > Ok, here's my shot in the dark; we must be detecting something about the
> > upstream switch port to make it fail the ACS test and the only thing I
> > can find that might do this is if the PCI config header on the upstream
> > switch reported itself as a multifunction device.  Multifunction
> > upstream switch ports do need ACS capabilities to make sure that traffic
> > isn't routed back through other functions.  Single function devices do
> > not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
> > looking to see whether the byte at 0xe has the MSB set.  If it does, it
> > lies that it's a multifunction device.  If it doesn't I'll have to get
> > the dart board back out.
> >
> > FWIW, you should be able to work around this by adding id:10b5:8606 to
> > your list of overrides.  Long term, if this is the problem, we'll want
> > to add a quirk to sanitize the multifunction device flag.  Thanks,
> >
> > Alex
> >
> 
> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the 
> upstream port of the PLX switch.

Yes, it was 04:00.0 in the previous lspci listing.

> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI 
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> 	Flags: bus master, fast devsel, latency 0
> 	Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
> 	Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
> 	Memory behind bridge: ee800000-eeafffff
> 	Capabilities: [40] Power Management version 3
> 	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> 	Capabilities: [68] Express Upstream Port, MSI 00
> 	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 
> Port PCI Express Gen 2 (5.0 GT/s) Switch
> 	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> 	Capabilities: [fb4] Advanced Error Reporting
> 	Capabilities: [138] Power Budgeting <?>
> 	Capabilities: [148] Virtual Channel
> 	Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?>
> 	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
> 	Kernel driver in use: pcieport
> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00

Well, that wasn't it.  I'll keep looking and probably send you a patch
to figure out what's going wrong.  Thanks,

Alex

Re: IOMMU groups ... PEX8606 switch?

[Alex Williamson]

On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote:
> On 01/02/2014 01:01 PM, Dana Goyette wrote:
> > On 01/02/2014 11:36 AM, Alex Williamson wrote:
> >> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
> >>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
> >>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
> >>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
> >>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
> >>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
> >>>>>>> need to
> >>>>>>> soon decide which one to keep.
> >>>>>>>
> >>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
> >>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
> >>>>>>> downstream of the PLX switch to be in separate groups.
> >>>>>>>
> >>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
> >>>>>>> capabilities"
> >>>>>>> applied and set for the Intel root ports, the devices behind the
> >>>>>>> switch
> >>>>>>> remain stuck in the same group.
> >>>>>>>
> >>>>>>> In terms of passing devices to different VMs, which is better: all
> >>>>>>> devices on different root ports, or all devices behind the one
> >>>>>>> ACS-supporting switch?
> >>>>>>
> >>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
> >>>>>> either the switch has ACS capabilities, but doesn't support the
> >>>>>> features
> >>>>>> we need or we're doing something wrong.  Thanks,
> >>>>>>
> >>>>> I initially tried attaching the output as a .txt file, but it's too
> >>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice
> >>>>> that I
> >>>>> moved the Radeon to a different slot).
> >>>>
> >>>> Well, something seems amiss since the downstream switch ports all seem
> >>>> to support and enable the correct set of ACS capabilities.  I'm tending
> >>>> to suspect something wrong with the ACS override patch or how it's
> >>>> being
> >>>> used since your IOMMU group is still based at the root port.  Each root
> >>>> port is isolated from the other root ports though, so something is
> >>>> happening with the override patch.  Can you provide the kernel command
> >>>> line you use to enable ACS overrides and the override patch you're
> >>>> using, as it applies to 3.13-rc5?  Thanks,
> >>>>
> >>>> Alex
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>
> >>> I'm using the original acs-override patch from this post:
> >>> https://lkml.org/lkml/2013/5/30/513
> >>>
> >>> Kernel parameter is:
> >>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
> >>>
> >>> When booting a kernel without the override patch, the following devices
> >>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
> >>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
> >>> controller; Intel I210 Ethernet controller.
> >>
> >> Ok, here's my shot in the dark; we must be detecting something about the
> >> upstream switch port to make it fail the ACS test and the only thing I
> >> can find that might do this is if the PCI config header on the upstream
> >> switch reported itself as a multifunction device.  Multifunction
> >> upstream switch ports do need ACS capabilities to make sure that traffic
> >> isn't routed back through other functions.  Single function devices do
> >> not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
> >> looking to see whether the byte at 0xe has the MSB set.  If it does, it
> >> lies that it's a multifunction device.  If it doesn't I'll have to get
> >> the dart board back out.
> >>
> >> FWIW, you should be able to work around this by adding id:10b5:8606 to
> >> your list of overrides.  Long term, if this is the problem, we'll want
> >> to add a quirk to sanitize the multifunction device flag.  Thanks,
> >>
> >> Alex
> >>
> >
> > 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
> > upstream port of the PLX switch.
> >
> > 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
> > Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> >      Flags: bus master, fast devsel, latency 0
> >      Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
> >      Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
> >      Memory behind bridge: ee800000-eeafffff
> >      Capabilities: [40] Power Management version 3
> >      Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> >      Capabilities: [68] Express Upstream Port, MSI 00
> >      Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
> > 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
> >      Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> >      Capabilities: [fb4] Advanced Error Reporting
> >      Capabilities: [138] Power Budgeting <?>
> >      Capabilities: [148] Virtual Channel
> >      Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
> > Len=0cc <?>
> >      Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
> > Len=010 <?>
> >      Kernel driver in use: pcieport
> > 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
> > 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
> > 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
> > 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> Just in case:
> 
> The root port:
> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset 
> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
> 	Flags: bus master, fast devsel, latency 0
> 	Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
> 	Memory behind bridge: ee800000-eebfffff
> 	Capabilities: [40] Express Root Port (Slot+), MSI 00
> 	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
> 	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
> 	Capabilities: [a0] Power Management version 3
> 	Kernel driver in use: pcieport
> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
> 
> The first downstream port:
> 
> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI 
> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
> 	Flags: bus master, fast devsel, latency 0
> 	Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
> 	Capabilities: [40] Power Management version 3
> 	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
> 	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
> 	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 
> Port PCI Express Gen 2 (5.0 GT/s) Switch
> 	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
> 	Capabilities: [fb4] Advanced Error Reporting
> 	Capabilities: [148] Virtual Channel
> 	Capabilities: [520] Access Control Services
> 	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
> 	Kernel driver in use: pcieport
> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00

Thanks, can you confirm whether adding id:10b5:8606 to the overrides
works?

Alex

Re: IOMMU groups ... PEX8606 switch?

[Dana Goyette]

On 01/02/2014 01:22 PM, Alex Williamson wrote:
> On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote:
>> On 01/02/2014 01:01 PM, Dana Goyette wrote:
>>> On 01/02/2014 11:36 AM, Alex Williamson wrote:
>>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
>>>>>>>>> need to
>>>>>>>>> soon decide which one to keep.
>>>>>>>>>
>>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
>>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>>
>>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
>>>>>>>>> capabilities"
>>>>>>>>> applied and set for the Intel root ports, the devices behind the
>>>>>>>>> switch
>>>>>>>>> remain stuck in the same group.
>>>>>>>>>
>>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>>> ACS-supporting switch?
>>>>>>>>
>>>>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
>>>>>>>> either the switch has ACS capabilities, but doesn't support the
>>>>>>>> features
>>>>>>>> we need or we're doing something wrong.  Thanks,
>>>>>>>>
>>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice
>>>>>>> that I
>>>>>>> moved the Radeon to a different slot).
>>>>>>
>>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>>> to support and enable the correct set of ACS capabilities.  I'm tending
>>>>>> to suspect something wrong with the ACS override patch or how it's
>>>>>> being
>>>>>> used since your IOMMU group is still based at the root port.  Each root
>>>>>> port is isolated from the other root ports though, so something is
>>>>>> happening with the override patch.  Can you provide the kernel command
>>>>>> line you use to enable ACS overrides and the override patch you're
>>>>>> using, as it applies to 3.13-rc5?  Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>> I'm using the original acs-override patch from this post:
>>>>> https://lkml.org/lkml/2013/5/30/513
>>>>>
>>>>> Kernel parameter is:
>>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>>>>
>>>>> When booting a kernel without the override patch, the following devices
>>>>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>>>>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>>>>> controller; Intel I210 Ethernet controller.
>>>>
>>>> Ok, here's my shot in the dark; we must be detecting something about the
>>>> upstream switch port to make it fail the ACS test and the only thing I
>>>> can find that might do this is if the PCI config header on the upstream
>>>> switch reported itself as a multifunction device.  Multifunction
>>>> upstream switch ports do need ACS capabilities to make sure that traffic
>>>> isn't routed back through other functions.  Single function devices do
>>>> not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
>>>> looking to see whether the byte at 0xe has the MSB set.  If it does, it
>>>> lies that it's a multifunction device.  If it doesn't I'll have to get
>>>> the dart board back out.
>>>>
>>>> FWIW, you should be able to work around this by adding id:10b5:8606 to
>>>> your list of overrides.  Long term, if this is the problem, we'll want
>>>> to add a quirk to sanitize the multifunction device flag.  Thanks,
>>>>
>>>> Alex
>>>>
>>>
>>> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
>>> upstream port of the PLX switch.
>>>
>>> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>>>       Flags: bus master, fast devsel, latency 0
>>>       Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
>>>       Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
>>>       Memory behind bridge: ee800000-eeafffff
>>>       Capabilities: [40] Power Management version 3
>>>       Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>>>       Capabilities: [68] Express Upstream Port, MSI 00
>>>       Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
>>> 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
>>>       Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>>>       Capabilities: [fb4] Advanced Error Reporting
>>>       Capabilities: [138] Power Budgeting <?>
>>>       Capabilities: [148] Virtual Channel
>>>       Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
>>> Len=0cc <?>
>>>       Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
>>> Len=010 <?>
>>>       Kernel driver in use: pcieport
>>> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
>>> 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
>>> 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> Just in case:
>>
>> The root port:
>> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
>> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
>> 	Flags: bus master, fast devsel, latency 0
>> 	Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
>> 	Memory behind bridge: ee800000-eebfffff
>> 	Capabilities: [40] Express Root Port (Slot+), MSI 00
>> 	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
>> 	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
>> 	Capabilities: [a0] Power Management version 3
>> 	Kernel driver in use: pcieport
>> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
>> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
>> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
>>
>> The first downstream port:
>>
>> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>> 	Flags: bus master, fast devsel, latency 0
>> 	Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
>> 	Capabilities: [40] Power Management version 3
>> 	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>> 	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
>> 	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
>> Port PCI Express Gen 2 (5.0 GT/s) Switch
>> 	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>> 	Capabilities: [fb4] Advanced Error Reporting
>> 	Capabilities: [148] Virtual Channel
>> 	Capabilities: [520] Access Control Services
>> 	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
>> 	Kernel driver in use: pcieport
>> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
>> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
>> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>
> Thanks, can you confirm whether adding id:10b5:8606 to the overrides
> works?
>
> Alex
>

Oddly, adding that ID doesn't seem to change anything.

\boot\vmlinuz-3.13.0-rc5 ro 
root=UUID=8414610a-91ff-41b9-80c2-cc49ee34cce6 console=ttyS4,115200n8 
console=tty0 intel_iommu=on,igfx_off 
pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 
initrd=boot\initrd.img-3.13.0-rc5

### Group 10 ###
     00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset 
Family PCI Express Root Port #2 (rev d5)
     05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     0a:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 
Host Controller (rev 03)
     0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI 
Express to PCI Bridge [Cheetah Express] (rev 01)
     0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 
IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)

Re: IOMMU groups ... PEX8606 switch?

[Dana Goyette]

(Re-sending to gmane.linux.kernel.pci so the solution ends up there, as 
  well.)

On 01/04/2014 12:22 PM, Alex Williamson wrote:
> On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote:
>> On 01/03/2014 04:03 PM, Alex Williamson wrote:
>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>>>> soon decide which one to keep.
>>>>>>>>
>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>
>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>>>> remain stuck in the same group.
>>>>>>>>
>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>> ACS-supporting switch?
>>>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
>>>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>>>> we need or we're doing something wrong.  Thanks,
>>>>>>>
>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>>>> moved the Radeon to a different slot).
>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>> to support and enable the correct set of ACS capabilities.  I'm tending
>>>>> to suspect something wrong with the ACS override patch or how it's being
>>>>> used since your IOMMU group is still based at the root port.  Each root
>>>>> port is isolated from the other root ports though, so something is
>>>>> happening with the override patch.  Can you provide the kernel command
>>>>> line you use to enable ACS overrides and the override patch you're
>>>>> using, as it applies to 3.13-rc5?  Thanks,
>>>>>
>>>>> Alex
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>> I'm using the original acs-override patch from this post:
>>>> https://lkml.org/lkml/2013/5/30/513
>>>>
>>>> Kernel parameter is:
>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>
> Actually, you're not:
>
> pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606
>
> And we register all of them:
>
> [    0.000000] PCIe ACS bypass added for 8086:8c10
> [    0.000000] PCIe ACS bypass added for 8086:8c16
> [    0.000000] PCIe ACS bypass added for 8086:8c18
> [    0.000000] PCIe ACS bypass added for 8086:ac1a
> [    0.000000] PCIe ACS bypass added for 8086:8c1c
> [    0.000000] PCIe ACS bypass added for 8086:8c1e
> [    0.000000] PCIe ACS bypass added for 10b5:8606
>
> However, note that the root port causing you trouble is 8086:8c12, which
> isn't provided as an override, therefore the code is doing the right
> thing and grouping all devices behind that root port together.
>

Thanks for catching that -- I certainly missed it!
I've added the override for that root port and removed the override for 
the PLX switch; now all the ports are indeed in separate groups.

Do we yet know if it'll be possible to properly isolate the Intel root 
ports, without this ACS override?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html