From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Date: Tue, 7 Jan 2014 10:38:39 +0000 Message-ID: <52CBD92F.3050301@citrix.com> References: "\"<52308E1402000078000F2748@nat28.tlf.novell.com> <20131211183233.GA2760@phenom.dumpdata.com> <52A8D5E5.2030902@bobich.net> <20131211213025.GA8283@phenom.dumpdata.com> <52AAF9D7020000780010CF2C@nat28.tlf.novell.com> <20131213144319.GK2923@phenom.dumpdata.com> <52AB2E18020000780010D13B@nat28.tlf.novell.com> <52AB275D.2010401@bobich.net>" <20140106202621.GA30667@phenom.dumpdata.com>" <20140106214527.GA31147@phenom.dumpdata.com> <1e555bfbd9f1c2eccfb4721d90782d12@mail.shatteredsilicon.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1e555bfbd9f1c2eccfb4721d90782d12@mail.shatteredsilicon.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Gordan Bobic Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 07/01/14 10:35, Gordan Bobic wrote: > On 2014-01-07 03:17, Zhang, Yang Z wrote: >> Konrad Rzeszutek Wilk wrote on 2014-01-07: >>>> Which would look like this: >>>> >>>> C220 ---> Tundra Bridge -----> (HB6 PCI bridge -> Brooktree BDFs) >>>> on the card >>>> \--------------> IEEE-1394a >>>> >>>> I am actually wondering if this 07:00.0 device is the one that >>>> reports itself as 08:00.0 (which I think is what you alluding to >>>> Jan) >>>> >>> >>> And to double check that theory I decided to pass in the IEEE-1394a >>> to a guest: >>> >>> +-1c.5-[07-08]----00.0-[08]----03.0 Texas Instruments >>> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] >>> >>> >>> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow (XEN) >>> [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) >>> [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault >>> addr 370f1000, iommu reg = ffff82c3ffd53000 (XEN) DMAR:[fault reason >>> 02h] Present bit in context entry is clear (XEN) print_vtd_entries: >>> iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 370f1 (XEN) root_entry >>> = ffff83083d47f000 (XEN) root_entry[8] = 72569b001 (XEN) >>> context >>> = ffff83072569b000 (XEN) context[0] = 0_0 (XEN) ctxt_entry[0] >>> not present >>> >>> So, capture card OK - Likely the Tundra bridge has an issue: >>> >>> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) >>> (prog-if 01 [Subtractive decode]) >>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>> ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- >>> UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >> >SERR- >> subordinate=08, sec-latency=32 Memory behind bridge: >>> f0600000-f06fffff Secondary status: 66MHz+ FastB2B+ ParErr- >>> DEVSEL=medium TAbort- >> BridgeCtl: >>> Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- >>> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- >>> Capabilities: [60] Subsystem: Super Micro Computer Inc >>> Device 0805 >>> Capabilities: [a0] Power Management version 3 >>> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA >>> PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ >>> PME-Enable- DSel=0 DScale=0 PME- >>> >>> or there is some unknown bridge in the motherboard. >> >> According your description above, the upstream Linux should also have >> the same problem. Did you see it with upstream Linux? > > The problem I was seeing with LSI cards (phantom device doing DMA) > does, indeed, also occur in upstream Linux. If I enable intel-iommu on > bare metal Linux, the same problem occurs as with Xen. > >> There may be some buggy device that generate DMA request with internal >> BDF but it didn't expose it(not like Phantom device). For those >> devices, I think we need to setup the VT-d page table manually. > > I think what is needed is a pci-phantom style override that tells the > hypervisor to tell the IOMMU to allow DMA traffic from a specific > invisible device ID. > > Gordan There is. See "pci-phantom" in http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html ~Andrew