Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> >On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
> >>Jan Beulich wrote on 2013-09-11:
> >>>>>>On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
> >>>><JBeulich@suse.com>
> >>>>  wrote:
> >>>>>>>>On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>>>On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
> >>>>>><JBeulich@suse.com>
> >>>>>>  wrote:
> >>>>>>>>>>On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
> >>>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
> >>>>>>>>
> >>>>>>>>  I'll try adding one of my LSI cards and see the comparative
> >>>>>>>>behaviour. Right now I don't even know if the phantom device  is
> >>>>>>>>on the SAS card or the motherboard.
> >>>>>>>
> >>>>>>>The Adaptec card being the only thing on bus 0f makes it pretty
> >>>>>>>likely that this other device also is on that card.
> >>>>>>>
> >>>>>>>I guess the issue is mainly because the device itself is a PCI
> >>>>>>>one, while the immediately upstream bridge (where I mean only the
> >>>>>>>visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> >>>>>>>them. And as long as firmware doesn't know about that bridge and
> >>>>>>>the bridge doesn't properly handle config space accesses to it,
> >>>>>>>such a device just can't be used with an IOMMU (without some yet
> >>>>>>>to be invented workaround).
> >>>>>>>
> >>>>>>  I'm actually thinking about Konrad's proposed hack in that
> >>>>>>thread from 3 years ago. If the device IDs are parameterized  out
> >>>>>>rather than hard-coded, then this could work in nearly the  same
> >>>>>>was as xen-pciback in terms of usage. Pass the phantom  device IDs
> >>>>>>as parameters to the module. Done that way it  might even be
> >>>>>>considered clean enough to be fit for public  consumption.
> >>>>>
> >>>>>Except that, short of being able to determine it via config space
> >>>>>reads, we also need the resulting command line option to tell us
> >>>>>that what kind of device that is.
> >>>>>
> >>>>  Not sure I follow. Why do we need to know the device type?
> >>>
> >>>Just look at set_msi_source_id() as well as
> >>>domain_context_{mapping,unmap}() (just the most prominent
> >>>examples): Behavior here heavily depends on the type of the device
> >>>itself _and_ that of the upstream bridge(s).
> >>Looks like there are many devices are failed to work. I wonder whether the PCI/PCIe specification tells how to detect the hidden device behind those devices (Like detection of phantom device). If not, I think those devices are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since VT-d only covers the PCI/PCIe device, it's reasonable that non-PCI/PCIe device failed to work under VT-d.
> >>
> >>As Jan's suggestion, we need the user to tell us whether there is a hidden device or BDF behind anther device that the OS is unaware. We need to pass that info to Xen before pass-thought the device.
> >>
> >
> >Interestingly enough I just hit this with my brand-new Haswell CPU and
> >new motherboard when passing in a capture card. It shows:
> >
> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-08.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-09.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-09.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0a.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-0a.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0b.0  Brooktree Corporation Bt878 Video Capture
> >            |                               |            \-0b.1  Brooktree Corporation Bt878 Audio Capture
> >            |                               \-03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >
> >And Xen says:
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> >(XEN)     root_entry = ffff83083d47e000
> >(XEN)     root_entry[8] = 72569a001
> >(XEN)     context = ffff83072569a000
> >(XEN)     context[0] = 0_0
> >(XEN)     ctxt_entry[0] not present
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >
> >
> >Oddly enough it was working fine in a box with an AMD IOMMU. But
> >to be fair - that machine was running with Xen 4.1.
> >
> >The hack I developed: http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> >ends up with this:
> >
> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
> >
> >(FYI, this Xen 4.3.1)
> >
> >Let me retry on the AMD box with the same version of Xen.
> 
> I may be wrong, but this doesn't look like the same problem (phantom
> PCI device on the bus). Or am I missing something?

It is. A phantom device as well.
> 
> As far as I can tell, the original problem was arising on cards that
> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
> Xen wasn't the only thing affected in my case - bare metal Linux
> kernel was also having problems with intel-iommu=1 in the kernel
> boot parameters. If might be worth trying that with your card to see
> what happens. If bare metal Linux with intel-iommu=1 works for your
> card, it's probably not the same problem (of course it could be
> similar/related).

That is a similar problem here. Except that I have a PCI devices and
it goes over an PCIe bridge (I think).
> 
> Out of interest, I noticed recently there is a xen parameter
> "pci-phantom", but I haven't been able to find documentation for it.
> Can you point me in the right direction? Does it, perchance, allow
> specifying the PCI slot ID of a phantom device so that IOMMU doesn't
> freak out when a seemingly non-existant device starts trying to do
> DMA?

I forgot about it!

t 4e3c592c93d7dbe02ca36878457515d30fe931d2
Author: Jan Beulich <jbeulich@suse.com>
Date:   Mon Jan 7 12:58:09 2013 +0100

    IOMMU: add option to specify devices behaving like ones using phantom functions
    
    At least certain Marvell SATA controllers are known to issue bus master
    requests with a non-zero function as origin, despite themselves being
    single function devices.

Here is what the manpage says:

+### pci-phantom
+> `=[<seg>:]<bus>:<device>,<stride>`
+
+Mark a group of PCI devices as using phantom functions without actually
+advertising so, so the IOMMU can create translation contexts for them.
+
+All numbers specified must be hexadecimal ones.
+
+This option can be specified more than once (up to 8 times at present).
+

Hm, time to try it out.
> 
> Gordan