From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gordan Bobic Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Date: Tue, 07 Jan 2014 10:35:49 +0000 Message-ID: <1e555bfbd9f1c2eccfb4721d90782d12@mail.shatteredsilicon.net> References: "\"<52308E1402000078000F2748@nat28.tlf.novell.com> <20131211183233.GA2760@phenom.dumpdata.com> <52A8D5E5.2030902@bobich.net> <20131211213025.GA8283@phenom.dumpdata.com> <52AAF9D7020000780010CF2C@nat28.tlf.novell.com> <20131213144319.GK2923@phenom.dumpdata.com> <52AB2E18020000780010D13B@nat28.tlf.novell.com> <52AB275D.2010401@bobich.net>" <20140106202621.GA30667@phenom.dumpdata.com>" <20140106214527.GA31147@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 2014-01-07 03:17, Zhang, Yang Z wrote: > Konrad Rzeszutek Wilk wrote on 2014-01-07: >>> Which would look like this: >>> >>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) on >>> the card >>> \--------------> IEEE-1394a >>> >>> I am actually wondering if this 07:00.0 device is the one that >>> reports itself as 08:00.0 (which I think is what you alluding to >>> Jan) >>> >> >> And to double check that theory I decided to pass in the IEEE-1394a to >> a guest: >> >> +-1c.5-[07-08]----00.0-[08]----03.0 Texas Instruments >> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] >> >> >> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN) >> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) >> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault >> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason >> 02h] Present bit in context entry is clear (XEN) print_vtd_entries: >> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN) >> root_entry >> = ffff83083d47f000 (XEN) root_entry[8] = 72569b001 (XEN) >> context >> = ffff83072569b000 (XEN) context[0] = 0_0 (XEN) ctxt_entry[0] >> not present >> >> So, capture card OK - Likely the Tundra bridge has an issue: >> >> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) >> (prog-if 01 [Subtractive decode]) >> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >> ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- >> UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > >SERR- > subordinate=08, sec-latency=32 Memory behind bridge: >> f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr- >> DEVSEL=medium TAbort- > BridgeCtl: >> Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- >> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- >> Capabilities: [60] Subsystem: Super Micro Computer Inc Device >> 0805 >> Capabilities: [a0] Power Management version 3 >> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA >> PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ >> PME-Enable- DSel=0 DScale=0 PME- >> >> or there is some unknown bridge in the motherboard. > > According your description above, the upstream Linux should also have > the same problem. Did you see it with upstream Linux? The problem I was seeing with LSI cards (phantom device doing DMA) does, indeed, also occur in upstream Linux. If I enable intel-iommu on bare metal Linux, the same problem occurs as with Xen. > There may be some buggy device that generate DMA request with internal > BDF but it didn't expose it(not like Phantom device). For those > devices, I think we need to setup the VT-d page table manually. I think what is needed is a pci-phantom style override that tells the hypervisor to tell the IOMMU to allow DMA traffic from a specific invisible device ID. Gordan