Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Gordan Bobic <gordan@bobich.net>]

On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
>> Jan Beulich wrote on 2013-09-11:
>>>>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
>>>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>>>> <JBeulich@suse.com>
>>>>   wrote:
>>>>>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
>>>>>> <JBeulich@suse.com>
>>>>>>   wrote:
>>>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>>>>>>>>   dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>>>>>>
>>>>>>>>   I'll try adding one of my LSI cards and see the comparative
>>>>>>>> behaviour. Right now I don't even know if the phantom device  is
>>>>>>>> on the SAS card or the motherboard.
>>>>>>>
>>>>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>>>>> likely that this other device also is on that card.
>>>>>>>
>>>>>>> I guess the issue is mainly because the device itself is a PCI
>>>>>>> one, while the immediately upstream bridge (where I mean only the
>>>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>>>>> them. And as long as firmware doesn't know about that bridge and
>>>>>>> the bridge doesn't properly handle config space accesses to it,
>>>>>>> such a device just can't be used with an IOMMU (without some yet
>>>>>>> to be invented workaround).
>>>>>>>
>>>>>>   I'm actually thinking about Konrad's proposed hack in that
>>>>>> thread from 3 years ago. If the device IDs are parameterized  out
>>>>>> rather than hard-coded, then this could work in nearly the  same
>>>>>> was as xen-pciback in terms of usage. Pass the phantom  device IDs
>>>>>> as parameters to the module. Done that way it  might even be
>>>>>> considered clean enough to be fit for public  consumption.
>>>>>
>>>>> Except that, short of being able to determine it via config space
>>>>> reads, we also need the resulting command line option to tell us
>>>>> that what kind of device that is.
>>>>>
>>>>   Not sure I follow. Why do we need to know the device type?
>>>
>>> Just look at set_msi_source_id() as well as
>>> domain_context_{mapping,unmap}() (just the most prominent
>>> examples): Behavior here heavily depends on the type of the device
>>> itself _and_ that of the upstream bridge(s).
>> Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.
>>
>> As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.
>>
>
> Interestingly enough I just hit this with my brand-new Haswell CPU and
> new motherboard when passing in a capture card. It shows:
>
>      +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
>             |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
>             |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
>             |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>
> And Xen says:
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> (XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> (XEN)     root_entry = ffff83083d47e000
> (XEN)     root_entry[8] = 72569a001
> (XEN)     context = ffff83072569a000
> (XEN)     context[0] = 0_0
> (XEN)     ctxt_entry[0] not present
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
>
>
> Oddly enough it was working fine in a box with an AMD IOMMU. But
> to be fair - that machine was running with Xen 4.1.
>
> The hack I developed: http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> ends up with this:
>
> (XEN) alloc_pdev: unknown type: 0000:08:00.0
> (XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> (XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>
> (FYI, this Xen 4.3.1)
>
> Let me retry on the AMD box with the same version of Xen.

I may be wrong, but this doesn't look like the same problem (phantom PCI 
device on the bus). Or am I missing something?

As far as I can tell, the original problem was arising on cards that are 
PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge. Xen 
wasn't the only thing affected in my case - bare metal Linux kernel was 
also having problems with intel-iommu=1 in the kernel boot parameters. 
If might be worth trying that with your card to see what happens. If 
bare metal Linux with intel-iommu=1 works for your card, it's probably 
not the same problem (of course it could be similar/related).

Out of interest, I noticed recently there is a xen parameter 
"pci-phantom", but I haven't been able to find documentation for it. Can 
you point me in the right direction? Does it, perchance, allow 
specifying the PCI slot ID of a phantom device so that IOMMU doesn't 
freak out when a seemingly non-existant device starts trying to do DMA?

Gordan