Re: IOMMU groups ... PEX8606 switch?

[Dana Goyette <DanaGoyette@gmail.com>]

On 01/04/2014 12:22 PM, Alex Williamson wrote:
> On Sat, 2014-01-04 at 11:26 -0800, Dana Goyette wrote:
>> On 01/03/2014 04:03 PM, Alex Williamson wrote:
>>> On Mon, 2013-12-30 at 16:13 -0800, Dana Goyette wrote:
>>>> On 12/29/2013 08:16 PM, Alex Williamson wrote:
>>>>> On Sat, 2013-12-28 at 23:32 -0800, Dana Goyette wrote:
>>>>>> On 12/28/2013 7:23 PM, Alex Williamson wrote:
>>>>>>> On Sat, 2013-12-28 at 18:31 -0800, Dana Goyette wrote:
>>>>>>>> I have purchased both a SuperMicro X10SAE and an X10SAT, and I need to
>>>>>>>> soon decide which one to keep.
>>>>>>>>
>>>>>>>> The SuperMicro X10SAT has all the PCIe x1 slots hidden behind a PLX
>>>>>>>> PEX8066 switch, which claims to support ACS.  I'd expect the devices
>>>>>>>> downstream of the PLX switch to be in separate groups.
>>>>>>>>
>>>>>>>> With Linux 3.13-rc5 and "enable overrides for missing ACS capabilities"
>>>>>>>> applied and set for the Intel root ports, the devices behind the switch
>>>>>>>> remain stuck in the same group.
>>>>>>>>
>>>>>>>> In terms of passing devices to different VMs, which is better: all
>>>>>>>> devices on different root ports, or all devices behind the one
>>>>>>>> ACS-supporting switch?
>>>>>>> Can you provide lspci -vvv info?  If you're getting that for groups
>>>>>>> either the switch has ACS capabilities, but doesn't support the features
>>>>>>> we need or we're doing something wrong.  Thanks,
>>>>>>>
>>>>>> I initially tried attaching the output as a .txt file, but it's too
>>>>>> large.  Anyway, here's the output of lspci -nnvvv (you may notice that I
>>>>>> moved the Radeon to a different slot).
>>>>> Well, something seems amiss since the downstream switch ports all seem
>>>>> to support and enable the correct set of ACS capabilities.  I'm tending
>>>>> to suspect something wrong with the ACS override patch or how it's being
>>>>> used since your IOMMU group is still based at the root port.  Each root
>>>>> port is isolated from the other root ports though, so something is
>>>>> happening with the override patch.  Can you provide the kernel command
>>>>> line you use to enable ACS overrides and the override patch you're
>>>>> using, as it applies to 3.13-rc5?  Thanks,
>>>>>
>>>>> Alex
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>> I'm using the original acs-override patch from this post:
>>>> https://lkml.org/lkml/2013/5/30/513
>>>>
>>>> Kernel parameter is:
>>>> pcie_acs_override=id:8086:8c10,id:8086:8c12,id:8086:8c16,id:8086:8c18
>
> Actually, you're not:
>
> pcie_acs_override=id:8086:8c10,id:8086:8c16,id:8086:8c18,id:8086:ac1a,id:8086:8c1c,id:8086:8c1e,id:10b5:8606
>
> And we register all of them:
>
> [    0.000000] PCIe ACS bypass added for 8086:8c10
> [    0.000000] PCIe ACS bypass added for 8086:8c16
> [    0.000000] PCIe ACS bypass added for 8086:8c18
> [    0.000000] PCIe ACS bypass added for 8086:ac1a
> [    0.000000] PCIe ACS bypass added for 8086:8c1c
> [    0.000000] PCIe ACS bypass added for 8086:8c1e
> [    0.000000] PCIe ACS bypass added for 10b5:8606
>
> However, note that the root port causing you trouble is 8086:8c12, which
> isn't provided as an override, therefore the code is doing the right
> thing and grouping all devices behind that root port together.
>

Thanks for catching that -- I certainly missed it!
I've added the override for that root port and removed the override for 
the PLX switch; now all the ports are indeed in separate groups.

Do we yet know if it'll be possible to properly isolate the Intel root 
ports, without this ACS override?