Re: [PATCH] amd iommu: Dump flags of IO page faults (off topic - pci devices)

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 07/09/12 08:32, Sander Eikelenboom wrote:
> Thursday, September 6, 2012, 5:03:05 PM, you wrote:
>
>> On 09/06/2012 03:50 PM, Sander Eikelenboom wrote:
>>> Thursday, September 6, 2012, 3:32:51 PM, you wrote:
>>>
>>>> On 09/06/2012 12:59 AM, Sander Eikelenboom wrote:
>>>>> Wednesday, September 5, 2012, 4:42:42 PM, you wrote:
>>>>>
>>>>>> Hi Jan,
>>>>>> Attached patch dumps io page fault flags. The flags show the reason of
>>>>>> the fault and tell us if this is an unmapped interrupt fault or a DMA fault.
>>>>>> Thanks,
>>>>>> Wei
>>>>>> signed-off-by: Wei Wang<wei.wang2@amd.com>
>>>>>
>>>>> I have applied the patch and the flags seem to differ between the faults:
>>>>>
>>>>> AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0x0a06, fault address = 0xc2c2c2c0, flags = 0x000
>>>>> (XEN) [2012-09-05 20:54:16] AMD-Vi: IO_PAGE_FAULT: domain = 0, device id = 0x0a06, fault address = 0xc2c2c2c0, flags = 0x000
>>>>> (XEN) [2012-09-05 20:54:16] AMD-Vi: IO_PAGE_FAULT: domain = 14, device id = 0x0700, fault address = 0xa8d339e0, flags = 0x020
>>>>> (XEN) [2012-09-05 20:54:16] AMD-Vi: IO_PAGE_FAULT: domain = 14, device id = 0x0700, fault address = 0xa8d33a40, flags = 0x020
>>>> OK, so they are not interrupt requests. I guess further information from
>>>> your system would be helpful to debug this issue:
>>>> 1) xl info
>>>> 2) xl list
>>>> 3) lscpi -vvv (NOTE: not in dom0 but in your guest)
>>>> 4) cat /proc/iomem (in both dom0 and your hvm guest)
>>> dom14 is not a HVM guest,it's a PV guest.
>> Ah, I see. PV guest is quite different than hvm, it does use p2m tables
>> as io page tables. So no-sharept option does not work in this case. PV
>> guests always use separated io page tables. There might be some
>> incorrect mappings on the page tables. I will check this on my side.
> I have reverted the machine to xen-4.1.4-pre (changeset 23353) and kept everything else the same.
> I haven't seen any IO PAGE FAULTS after that.
>
> I did spot some differences in the output from lspci between xen 4.1 and 4.2, related to MSI enabled or not for the IOMMU device.
> Have attached the xl/xm dmesg and lspci from booting with both versions.
>
> lspci:
>
> 00:00.2 Generic system peripheral [0806]: ATI Technologies Inc RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
>         Subsystem: ATI Technologies Inc RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 10
>         Capabilities: [40] Secure device <?>
> 4.1:    Capabilities: [54] MSI: Enable- Count=1/1 Maskable- 64bit+
> 4.2:    Capabilities: [54] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 00000000fee0100c  Data: 4128
>         Capabilities: [64] HyperTransport: MSI Mapping Enable+ Fixed+
>
> Although it seems enabled, shouldn't the IRQ number used be much higher than 10 for MSI interrupts ?

For compatibility reasons, all real PCI devices have to have the ability
to fall back to legacy line level interrupts.  This is the IRQ10 which
you see, which is #INTA in perhaps more recognizable notation.  The line
interrupt(s) will only be used if MSI and MSI-x interrupts are disabled.

You should find that all devices in lspci show between 1 and 4 #INTs (a
thru d), with the exception of SRIOV virtual function which are
specified to only support MSI/MSI-x

~Andrew

>
> There is another difference in the bridge device that's in front of the  0a:00.6 device that faults before the kernel is even booted.
>
> 00:03.0 PCI bridge [0604]: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port C) [1002:5a17] (prog-if 00 [Normal decode])
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
> 4.1:    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 4.2:    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort+ <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
>         I/O behind bridge: 0000f000-00000fff
>         Memory behind bridge: f9f00000-f9ffffff
>         Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
> 4.1:    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> 4.2:    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort+ <TAbort- <MAbort- <SERR- <PERR-
>         BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [50] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
>                         ExtTag+ RBE+ FLReset-
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
>                 LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <1us, L1 <8us
>                         ClockPM- Surprise- LLActRep+ BwNot+
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
>                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
>                         Slot #3, PowerLimit 10.000W; Interlock- NoCompl+
>                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>                         Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
>                 SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>                         Changed: MRL- PresDet+ LinkState+
>
> serveerstertje:~# lspci -t
> -[0000:00]-+-00.0
>            +-00.2
>            +-02.0-[0b]----00.0
>            +-03.0-[0a]--+-00.0
>            |            +-00.1
>            |            +-00.2
>            |            +-00.3
>            |            +-00.4
>            |            +-00.5
>            |            +-00.6
>            |            \-00.7
>            +-05.0-[09]----00.0
>            +-06.0-[08]----00.0
>            +-0a.0-[07]----00.0
>            +-0b.0-[06]--+-00.0
>            |            \-00.1
>            +-0c.0-[05]----00.0
>            +-0d.0-[04]--+-00.0
>            |            +-00.1
>            |            +-00.2
>            |            +-00.3
>            |            +-00.4
>            |            +-00.5
>            |            +-00.6
>            |            \-00.7
>            +-11.0
>            +-12.0
>            +-12.2
>            +-13.0
>            +-13.2
>            +-14.0
>            +-14.3
>            +-14.4-[03]----06.0
>            +-14.5
>            +-15.0-[02]--
>            +-16.0
>            +-16.2
>            +-18.0
>            +-18.1
>            +-18.2
>            +-18.3
>            \-18.4
>
>
>
>
>
>> Thanks,
>> Wei
>>> I will try to make a complete package, and try with one pv domain only where the devices are being passed through just to simplify the setup.
>>>
>>>
>>>> * I would also like to know the symptoms of device 0x0700 when IO_PF
>>>> happened. Did it stop working?
>>> Yes it stops working, the video capture just freezes, but the driver doesn't bail out.
>>> For the USB controller (0x0a06) it starts to give errors for usbdev_open in the guest.
>>>
>>>> (BTW: I copied a few options from your boot cmd line and it worked with
>>>> my RD890 system
>>>> dom0_mem=1024M,max:1024M loglvl=all loglvl_guest=all console_timestamps
>>>> cpuidle cpufreq=xen noreboot debug lapic=debug apic_verbosity=debug
>>>> apic=debug iommu=on,verbose,debug,no-sharept
>>>> * so, what OEM board you have?)
>>> MSI 890FXA-GD70
>>>
>>>> Also from your log, these lines looks very strange:
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xd5, mfn=0xa4a11
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xd7, mfn=0xa4a0f
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xd9, mfn=0xa4a0d
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xdb, mfn=0xa4a0b
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xdd, mfn=0xa4a09
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xdf, mfn=0xa4a07
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xe1, mfn=0xa4a05
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xe3, mfn=0xa4a03
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xe5, mfn=0xa4a01
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xe7, mfn=0xa463f
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xe9, mfn=0xa463d
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xeb, mfn=0xa463b
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xed, mfn=0xa4639
>>>> (XEN) [2012-09-04 15:54:35] hvm.c:2435:d15 guest attempted write to
>>>> read-only memory page. gfn=0xef, mfn=0xa4637
>>>> (XEN) [2012-09-04 16:13:56] AMD-Vi: IO_PAGE_FAULT: domain = 0, device id
>>>> = 0x0a06, fault address = 0xc2c2c2c0
>>>> (XEN) [2012-09-04 16:13:56] AMD-Vi: IO_PAGE_FAULT: domain = 14, device
>>>> id = 0x0700, fault address = 0xa90f8300
>>>> (XEN) [2012-09-04 16:13:56] AMD-Vi: IO_PAGE_FAULT: domain = 14, device
>>>> id = 0x0700, fault address = 0xa90f8340
>>>> (XEN) [2012-09-04 16:13:56] AMD-Vi: IO_PAGE_FAULT: domain = 14, device
>>>> id = 0x0700, fault address = 0xa90f8380
>>>> (XEN) [2012-09-04 16:13:56] AMD-Vi: IO_PAGE_FAULT: domain = 14, device
>>>> id = 0x0700, fault address = 0xa90f83c0
>>>> * they are just followed by the IO PAGE fault. Do you know where are
>>>> they from? Your video card driver maybe?
>>>  From a HVM domain with a old (3.0.3) kernel, but the faults also occur without this domain being started.
>>>
>>>
>>>> Thanks,
>>>> Wei
>>>
>>>>> Complete xl dmesg and lspci -vvvknn attached.
>>>>>
>>>>> Thx
>>>>>
>>>>> --
>>>>> Sander
>>>
>>>
>>>
>>>
>

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com