From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gordan Bobic Subject: Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)) Date: Wed, 11 Sep 2013 13:14:22 +0100 Message-ID: <9bd7c39084e6b264dda5b6ac256ea97b@mail.shatteredsilicon.net> References: <523075C502000078000F2617@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VJjJR-000413-Cl for xen-devel@lists.xenproject.org; Wed, 11 Sep 2013 12:14:25 +0000 Received: from mail.shatteredsilicon.net (localhost [127.0.0.1]) by external.sentinel2 (Postfix) with ESMTP id 110A1220294 for ; Wed, 11 Sep 2013 13:14:23 +0100 (BST) In-Reply-To: <523075C502000078000F2617@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xenproject.org List-Id: xen-devel@lists.xenproject.org On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" wrote: >>>> On 11.09.13 at 13:05, Gordan Bobic wrote: >> I found this: >> >> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html >> >> while looking for a solution to a similar problem. I am >> facing a similar issue with LSI (8408E, 3081E-R) and >> Adaptec (31605) SAS cards. Was there ever a proper, more general >> fix or workaround for this issue? >> >> These SAS cards experience these problems in dom0. When running >> a vanilla kernel on bare metal, they work OK without intel_iommu >> set. As soon as I set intel_iommu, the same thing happens (on >> bare metal, not dom0). >> >> Clearly there is something badly broken with multiple layers >> of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe >> root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller) > > The link above has some (hackish) workarounds - did you try > them? Not yet. The thing that bothers me is that he workaround involves hard-coding the PCI device ID which is _nasty_ and unstable. > The link above, however, doesn't indicate any relationship to > multiple bridges being in between, so it may not match what > you're observing. The impression I got was that the "invisible" devices the previous thread was referring to were bridges on the SAS card. But I may have misunderstood. > In any event, seeing a hypervisor log with "iommu=debug" might > shed further light on this: For one, we might be able to see which > exact devices are present in the ACPI tables. And we would see > which device(s) eventual faults originate from. The thing that bothers me is that this happens in dom0 even with iommu=dom0-passthrough being set. iommu=dom0-passthrough,workaround_bios_bug doesn't help, either And lo and behold, I do have phantom PCI devices after all! lspci shows no device with ID 0000:0f:01.0 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 857f15000, iommu reg = ffff82c3ffd54000 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 857f15 (XEN) root_entry = ffff83043ffe5000 (XEN) root_entry[f] = e6f7001 (XEN) context = ffff83000e6f7000 (XEN) context[8] = 0_0 (XEN) ctxt_entry[8] not present (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 858a35000, iommu reg = ffff82c3ffd54000 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 858a35 (XEN) root_entry = ffff83043ffe5000 (XEN) root_entry[f] = e6f7001 (XEN) context = ffff83000e6f7000 (XEN) context[8] = 0_0 (XEN) ctxt_entry[8] not present (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 471df8000, iommu reg = ffff82c3ffd54000 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 471df8 (XEN) root_entry = ffff83043ffe5000 (XEN) root_entry[f] = e6f7001 (XEN) context = ffff83000e6f7000 (XEN) context[8] = 0_0 (XEN) ctxt_entry[8] not present (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] fault addr 46fc22000, iommu reg = ffff82c3ffd54000 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 46fc22 (XEN) root_entry = ffff83043ffe5000 (XEN) root_entry[f] = e6f7001 (XEN) context = ffff83000e6f7000 (XEN) context[8] = 0_0 (XEN) ctxt_entry[8] not present Gordan