Re: Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

[Gordan Bobic <gordan@bobich.net>]

 On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
>> I found this:
>>
>>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>
>>  while looking for a solution to a similar problem. I am
>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>  fix or workaround for this issue?
>>
>>  These SAS cards experience these problems in dom0. When running
>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>  set. As soon as I set intel_iommu, the same thing happens (on
>>  bare metal, not dom0).
>>
>>  Clearly there is something badly broken with multiple layers
>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>
> The link above has some (hackish) workarounds - did you try
> them?

 Not yet. The thing that bothers me is that he workaround
 involves hard-coding the PCI device ID which is _nasty_
 and unstable.

> The link above, however, doesn't indicate any relationship to
> multiple bridges being in between, so it may not match what
> you're observing.

 The impression I got was that the "invisible" devices the
 previous thread was referring to were bridges on the SAS
 card. But I may have misunderstood.

> In any event, seeing a hypervisor log with "iommu=debug" might
> shed further light on this: For one, we might be able to see which
> exact devices are present in the ACPI tables. And we would see
> which device(s) eventual faults originate from.

 The thing that bothers me is that this happens in dom0 even
 with iommu=dom0-passthrough being set.
 iommu=dom0-passthrough,workaround_bios_bug doesn't help,
 either

 And lo and behold, I do have phantom PCI devices after all!
 lspci shows no device with ID 0000:0f:01.0


 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 857f15000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 857f15
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 858a35000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 858a35
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 471df8000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 471df8
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 46fc22000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 46fc22
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present

 Gordan