Re: Dealing with non-existent BDF devices in VT-d and in the hardware.

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 11/03/14 17:30, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> I am one of those lucky folks who had purchased a motherboard that has bugs.

You say this as if you expect someone has managed to find a bugfree
motherboard :)

>
> I figured I would post this email as way for a starting point
> for some discussion on this - and perhaps have a similar as 'pci-phantom'
> way of instructing the hypervisor what to do with them.
>
> The problem I am seeing is that this device:
>
> 08:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>
> Can't be passed in the guest. Or rather it can - but everytime
> the guest (or domain0) tries to access I see:
>
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:08:00.0] fault addr 0, iommu reg = ffff82c3ffd53000
> (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> (XEN) print_vtd_entries: iommu ffff83043dca99b0 dev 0000:08:00.0 gmfn 0
> (XEN)     root_entry = ffff83043dc6b000
> (XEN)     root_entry[8] = 3326b5001
> (XEN)     context = ffff8303326b5000
> (XEN)     context[0] = 0_0
> (XEN)     ctxt_entry[0] not present
>
>
> Of course the '08:00.0' device does not exist. It is rather this chipset:
> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
>
> that is buggy and using the wrong BDF when forwarding DMA requests from
> devices underneath it (like this Firewire chip).
>
> The hack I came up with was to create in the Xen code that deals with
> PCI passthrough a copy of the bridge (so 07:00.0) but with a new
> BDF: 08:00.0. And link it to the PCI device that I am passing to the
> guest (so 08:03.0).
>
> The end result is that when loading the driver (hack.c) one should
> see:
>
> (XEN) 0000:08:00.0 linked with 08:03.0
> (XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:08:00.0
> (XEN) [VT-D]iommu.c:1476: d0:PCI: map 0000:08:03.0
> (XEN) PCI add link 0000:08:00.0
>
> And when launching a guest with the BDF:
> pci = ["08:03.0"]
>
> the hypervisor will automatically also create an VT-d context for the
> 08:00.0 device.
>
> To use this hack, apply the 0001-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
> to your hypervisor, compile and install.
>
> And also compile the 'hack.c' module. There is an attached 'Makefile'
> that will do it for you. Make sure you edit it to set the right BDF
> entries in it.
>
> Once done install your new hypervisor, and insmod ./hack.ko and try
> passing in the device to your guest (or use it normally). The
> 'DMAR:[DMA Write]' error should go away.
>
> This should be generic enough for most devices. It needn't be a bridge
> that is spewing out these DMAR errors.


Do you have an lspci -tv for the system?

It is genuinely the case that the bridge doesn't exist, or simply that
it is not correctly attributed in the DMAR table?

If the latter, it Xen can probably gain some DMAR[$FOO]=$BAR command
line workarounds similar to the IVRS ones for AMD systems.

~Andrew