Re: IOMMU groups ... PEX8606 switch?

[Dana Goyette <DanaGoyette@gmail.com>]

On 01/02/2014 01:22 PM, Alex Williamson wrote:
> On Thu, 2014-01-02 at 13:15 -0800, Dana Goyette wrote:
>> On 01/02/2014 01:01 PM, Dana Goyette wrote:
>>> On 01/02/2014 11:36 AM, Alex Williamson wrote:
>>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I
>>>>>>>>> need to
>>>>>>>>> soon decide which one to keep.
>>>>>>>>>
>>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
>>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>>
>>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS
>>>>>>>>> capabilities"
>>>>>>>>> applied and set for the Intel root ports, the devices behind the
>>>>>>>>> switch
>>>>>>>>> remain stuck in the same group.
>>>>>>>>>
>>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>>> ACS-supporting switch?
>>>>>>>>
>>>>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
>>>>>>>> either the switch has ACS capabilities, but doesn't support the
>>>>>>>> features
>>>>>>>> we need or we're doing something wrong.  Thanks,
>>>>>>>>
>>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice
>>>>>>> that I
>>>>>>> moved the Radeon to a different slot).
>>>>>>
>>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>>> to support and enable the correct set of ACS capabilities.  I'm tending
>>>>>> to suspect something wrong with the ACS override patch or how it's
>>>>>> being
>>>>>> used since your IOMMU group is still based at the root port.  Each root
>>>>>> port is isolated from the other root ports though, so something is
>>>>>> happening with the override patch.  Can you provide the kernel command
>>>>>> line you use to enable ACS overrides and the override patch you're
>>>>>> using, as it applies to 3.13-rc5?  Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>> I'm using the original acs-override patch from this post:
>>>>> https://lkml.org/lkml/2013/5/30/513
>>>>>
>>>>> Kernel parameter is:
>>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>>>>>
>>>>> When booting a kernel without the override patch, the following devices
>>>>> are all in the same group: Intel Root Ports 1, 2, 4, 5; ASMedia SATA
>>>>> controller; PLX PEX8606 switch; Renesas USB controller; TI Firewire
>>>>> controller; Intel I210 Ethernet controller.
>>>>
>>>> Ok, here's my shot in the dark; we must be detecting something about the
>>>> upstream switch port to make it fail the ACS test and the only thing I
>>>> can find that might do this is if the PCI config header on the upstream
>>>> switch reported itself as a multifunction device.  Multifunction
>>>> upstream switch ports do need ACS capabilities to make sure that traffic
>>>> isn't routed back through other functions.  Single function devices do
>>>> not.  To test that theory, please provide 'lspci -vxs 4:00.0'.  We're
>>>> looking to see whether the byte at 0xe has the MSB set.  If it does, it
>>>> lies that it's a multifunction device.  If it doesn't I'll have to get
>>>> the dart board back out.
>>>>
>>>> FWIW, you should be able to work around this by adding id:10b5:8606 to
>>>> your list of overrides.  Long term, if this is the problem, we'll want
>>>> to add a quirk to sanitize the multifunction device flag.  Thanks,
>>>>
>>>> Alex
>>>>
>>>
>>> 04:00.0 is the ASMedia SATA controller; I'll assume you meant the
>>> upstream port of the PLX switch.
>>>
>>> 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>>>       Flags: bus master, fast devsel, latency 0
>>>       Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
>>>       Bus: primary=05, secondary=06, subordinate=0c, sec-latency=0
>>>       Memory behind bridge: ee800000-eeafffff
>>>       Capabilities: [40] Power Management version 3
>>>       Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>>>       Capabilities: [68] Express Upstream Port, MSI 00
>>>       Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane,
>>> 6 Port PCI Express Gen 2 (5.0 GT/s) Switch
>>>       Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>>>       Capabilities: [fb4] Advanced Error Reporting
>>>       Capabilities: [138] Power Budgeting <?>
>>>       Capabilities: [148] Virtual Channel
>>>       Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0
>>> Len=0cc <?>
>>>       Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0
>>> Len=010 <?>
>>>       Kernel driver in use: pcieport
>>> 00: b5 10 06 86 47 05 10 40 ba 00 04 06 10 00 01 00
>>> 10: 00 00 b0 ee 00 00 00 00 05 06 0c 00 f1 01 00 00
>>> 20: 80 ee a0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> Just in case:
>>
>> The root port:
>> 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
>> Family PCI Express Root Port #2 (rev d5) (prog-if 00 [Normal decode])
>> 	Flags: bus master, fast devsel, latency 0
>> 	Bus: primary=00, secondary=05, subordinate=0c, sec-latency=0
>> 	Memory behind bridge: ee800000-eebfffff
>> 	Capabilities: [40] Express Root Port (Slot+), MSI 00
>> 	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
>> 	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0818
>> 	Capabilities: [a0] Power Management version 3
>> 	Kernel driver in use: pcieport
>> 00: 86 80 12 8c 47 01 10 00 d5 00 04 06 10 00 81 00
>> 10: 00 00 00 00 00 00 00 00 00 05 0c 00 f0 00 00 20
>> 20: 80 ee b0 ee f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 02 13 00
>>
>> The first downstream port:
>>
>> 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI
>> Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode])
>> 	Flags: bus master, fast devsel, latency 0
>> 	Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
>> 	Capabilities: [40] Power Management version 3
>> 	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
>> 	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
>> 	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6
>> Port PCI Express Gen 2 (5.0 GT/s) Switch
>> 	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
>> 	Capabilities: [fb4] Advanced Error Reporting
>> 	Capabilities: [148] Virtual Channel
>> 	Capabilities: [520] Access Control Services
>> 	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
>> 	Kernel driver in use: pcieport
>> 00: b5 10 06 86 47 05 10 00 ba 00 04 06 10 00 01 00
>> 10: 00 00 00 00 00 00 00 00 06 07 07 00 f1 01 00 00
>> 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00
>> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 13 00
>
> Thanks, can you confirm whether adding id:10b5:8606 to the overrides
> works?
>
> Alex
>

Oddly, adding that ID doesn't seem to change anything.

\boot\vmlinuz-3.13.0-rc5 ro 
root=UUID=8414610a-91ff-41b9-80c2-cc49ee34cce6 console=ttyS4,115200n8 
console=tty0 intel_iommu=on,igfx_off 
pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606 
initrd=boot\initrd.img-3.13.0-rc5

### Group 10 ###
     00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset 
Family PCI Express Root Port #2 (rev d5)
     05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     06:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port 
PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
     0a:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 
Host Controller (rev 03)
     0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI 
Express to PCI Bridge [Cheetah Express] (rev 01)
     0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 
IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)