Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Gordan Bobic <gordan@bobich.net>]

On 2014-01-07 10:38, Andrew Cooper wrote:
> On 07/01/14 10:35, Gordan Bobic wrote:
>> On 2014-01-07 03:17, Zhang, Yang Z wrote:
>>> Konrad Rzeszutek Wilk wrote on 2014-01-07:
>>>>> Which would look like this:
>>>>> 
>>>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs)
>>>>> on the card
>>>>>           \--------------> IEEE-1394a
>>>>> 
>>>>> I am actually wondering if this 07:00.0 device is the one that
>>>>> reports itself as 08:00.0 (which I think is what you alluding to
>>>>> Jan)
>>>>> 
>>>> 
>>>> And to double check that theory I decided to pass in the IEEE-1394a
>>>> to a guest:
>>>> 
>>>>            +-1c.5-[07-08]----00.0-[08]----03.0  Texas Instruments
>>>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>>>> 
>>>> 
>>>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN)
>>>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN)
>>>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] 
>>>> fault
>>>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason
>>>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries:
>>>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN)     
>>>> root_entry
>>>> = ffff83083d47f000 (XEN)     root_entry[8] = 72569b001 (XEN)
>>>> context
>>>> = ffff83072569b000 (XEN)     context[0] = 0_0 (XEN)     
>>>> ctxt_entry[0]
>>>> not present
>>>> 
>>>> So, capture card OK - Likely the Tundra bridge has an issue:
>>>> 
>>>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>>>> (prog-if 01 [Subtractive decode])
>>>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>>>         ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 
>>>> 66MHz-
>>>>         UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+
>>>>         >SERR- <PERR- INTx- Latency: 0 Bus: primary=07, 
>>>> secondary=08,
>>>>         subordinate=08, sec-latency=32 Memory behind bridge:
>>>>         f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr-
>>>>         DEVSEL=medium TAbort- <TAbort- <MAbort+ <SERR- <PERR-
>>>> BridgeCtl:
>>>>         Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>>>         Capabilities: [60] Subsystem: Super Micro Computer Inc
>>>> Device 0805
>>>>         Capabilities: [a0] Power Management version 3
>>>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>>                 PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 
>>>> NoSoftRst+
>>>>                 PME-Enable- DSel=0 DScale=0 PME-
>>>> 
>>>> or there is some unknown bridge in the motherboard.
>>> 
>>> According your description above, the upstream Linux should also have
>>> the same problem. Did you see it with upstream Linux?
>> 
>> The problem I was seeing with LSI cards (phantom device doing DMA)
>> does, indeed, also occur in upstream Linux. If I enable intel-iommu on
>> bare metal Linux, the same problem occurs as with Xen.
>> 
>>> There may be some buggy device that generate DMA request with 
>>> internal
>>> BDF but it didn't expose it(not like Phantom device). For those
>>> devices, I think we need to setup the VT-d page table manually.
>> 
>> I think what is needed is a pci-phantom style override that tells the
>> hypervisor to tell the IOMMU to allow DMA traffic from a specific
>> invisible device ID.
>> 
>> Gordan
> 
> There is.  See "pci-phantom" in
> http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

I thought this was only applicable to phantom _functions_ (number after 
the
dot) rather than whole phantom _devices_. Is that not the case?