Re: VT-d faults with Integrated Intel graphics on 4.6

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 26/08/15 18:28, Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 26, 2015 at 06:20:34PM +0100, Andrew Cooper wrote:
>> On 26/08/15 18:15, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Aug 26, 2015 at 05:19:39PM +0100, Malcolm Crossley wrote:
>>>> On 25/08/15 15:43, Konrad Rzeszutek Wilk wrote:
>>>>> On Tue, Aug 25, 2015 at 02:55:31PM +0800, Chen, Tiejun wrote:
>>>>>> On 8/25/2015 8:19 AM, Tamas K Lengyel wrote:
>>>>>>> Hi everyone,
>>>>>>> I saw some people passingly mention this on the list before but just in
>>>>>>> case it has been missed, my serial is also being spammed with the following
>>>>>>> printouts with both Xen 4.6 RC1 and the latest staging build:
>>>>>>>
>>>>>>> ...
>>>>>>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
>>>>>>> 33487d7000, iommu reg = ffff82c000201000
>>>>>>> (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
>>>>>>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
>>>>>>> 33487d7000, iommu reg = ffff82c000201000
>>>>>>> (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
>>>>>>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
>>>>>>> 33487d7000, iommu reg = ffff82c000201000
>>>>>>> (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
>>>>>>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
>>>>>>> 33487d7000, iommu reg = ffff82c000201000
>>>>>>> (XEN) [VT-D]DMAR: reason 06 - PTE Read access is not set
>>>>>>> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr
>>>>>>> 2610742000, iommu reg = ffff82c000201000
>>>>>>> (XEN) [VT-D]DMAR: reason 07 - Next page table ptr is invalid
>>>>>>> ...
>>>>>>>
>>>> I think this problem is caused by missing IOMMU mappings for the RMRR regions
>>>> if the domain does not have shared EPT enabled. This includes PV Dom 0.
>>>>
>>>> I have posted a patch to fix the issue:
>>>>
>>>> http://lists.xen.org/archives/html/xen-devel/2015-08/msg02090.html
>>> No dice. Still seeing the same problem :-(
>> Does your BIOS provide suitable extra RMRR regions when AMT is in use? 
>> IIRC, AMT uses the on-chip GPU as part of its VNC handling.

> (XEN)  0000000000000000 - 000000000009a800 (usable)
> (XEN)  000000000009a800 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 0000000020000000 (usable)
> (XEN)  0000000020000000 - 0000000020200000 (reserved)
> (XEN)  0000000020200000 - 0000000040000000 (usable)
> (XEN)  0000000040000000 - 0000000040200000 (reserved)
> (XEN)  0000000040200000 - 000000009e856000 (usable)
> (XEN)  000000009e856000 - 000000009e85f000 (ACPI data)
> (XEN)  000000009e85f000 - 000000009e8aa000 (ACPI NVS)
> (XEN)  000000009e8aa000 - 000000009e8b2000 (usable)
> (XEN)  000000009e8b2000 - 000000009e9a5000 (reserved)
> (XEN)  000000009e9a5000 - 000000009e9a7000 (usable)
> (XEN)  000000009e9a7000 - 000000009ebc6000 (reserved)
> (XEN)  000000009ebc6000 - 000000009ebc7000 (usable)
> (XEN)  000000009ebc7000 - 000000009ebd7000 (reserved)
> (XEN)  000000009ebd7000 - 000000009ebf5000 (ACPI NVS)
> (XEN)  000000009ebf5000 - 000000009ec19000 (reserved)
> (XEN)  000000009ec19000 - 000000009ec5c000 (ACPI NVS)
> (XEN)  000000009ec5c000 - 000000009ee7c000 (reserved)
> (XEN)  000000009ee7c000 - 000000009f000000 (usable)
> (XEN)  000000009f800000 - 00000000bfa00000 (reserved)
> (XEN)  00000000fed1c000 - 00000000fed40000 (reserved)
> (XEN)  00000000ff000000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 000000043e600000 (usable)
<snip>
> (XEN) [VT-D]dmar.c:809: Host address width 36

<snip>

> (XEN) [VT-D]dmar.c:823: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:485:   dmaru->address = fed90000
> (XEN) [VT-D]iommu.c:1149: drhd->address = fed90000 iommu->reg = ffff82c000201000
> (XEN) [VT-D]iommu.c:1151: cap = c0000020e60262 ecap = f0101a
> (XEN) [VT-D]dmar.c:396:  endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:823: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:485:   dmaru->address = fed91000
> (XEN) [VT-D]iommu.c:1149: drhd->address = fed91000 iommu->reg = ffff82c000203000
> (XEN) [VT-D]iommu.c:1151: cap = c9008020660262 ecap = f0105a
> (XEN) [VT-D]dmar.c:410:  IOAPIC: 0000:f0:1f.0
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.0
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.1
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.2
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.3
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.4
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.5
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.6
> (XEN) [VT-D]dmar.c:374:  MSI HPET: 0000:f0:0f.7
> (XEN) [VT-D]dmar.c:499:   flags: INCLUDE_ALL
> (XEN) [VT-D]dmar.c:828: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:396:  endpoint: 0000:00:1d.0
> (XEN) [VT-D]dmar.c:396:  endpoint: 0000:00:1a.0
> (XEN) [VT-D]dmar.c:694:   RMRR region: base_addr 9ebad000 end_address 9ebbbfff
> (XEN) [VT-D]dmar.c:828: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:396:  endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:694:   RMRR region: base_addr 9f800000 end_address bf9fffff
>
> There ^^^^ That is the device.
>
<snip> (although, that is a large RMRR...)
> (XEN) [VT-D]iommu.c:873: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:875: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]DMAR:[DMA Read] Request device [0000:00:02.0] fault addr 49c9090000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 07 - Next page table ptr is invalid
> (XEN) print_vtd_entries: iommu ffff83042f6f7640 dev 0000:00:02.0 gmfn 49c9090

This faulting address is higher than top of ram as reported in the
e820.  Xen has legitimately not created IOMMU pagetables to map that
physical address.

Looks like something has programmed the GPU with a duff GTT.  I suspect
a firmware bug.

~Andrew