Re: Regression compared to Xen 4.3, Xen 4.4-rc2 - pci_prepare_msix+0xb1/0x12 - BOOM

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 21/01/2014 21:54, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> I hadn't done yet any diagnosis to figure out exactly which
> PCI device is at fault here. But this is regression compared
> to Xen 4.3 which boots just fine (see logs). The xen-syms
> is at: http://darnok.org/xen/xen-syms.gz
>
> I used idential kernel for Xen 4.3 and it booted nicely.
>
> My next step is to instrument the do_physdev_op to figure out which
> of the PCI devices is triggering this, but that will have to wait
> till later this week.
>
> What I get is this when booting Xen 4.4:
>
>
> [   15.927480] xen: registering gsi 19 triggering 0 polarity 1
> [   15.933039] Already setup the GSI :19
> (XEN) [2014-01-22 05:38:00] ----[ Xen-4.4-rc2  x86_64  debug=y  Tainted:    C ]----
> (XEN) [2014-01-22 05:38:00] CPU:    0
> (XEN) [2014-01-22 05:38:00] RIP:    e008:[<ffff82d080168d51>] pci_prepare_msix+0xb1/0x128
> (XEN) [2014-01-22 05:38:00] RFLAGS: 0000000000010246   CONTEXT: hypervisor
> (XEN) [2014-01-22 05:38:00] rax: 0000000000000000   rbx: 00000000fffffff0   rcx: 0000000000000000
> (XEN) [2014-01-22 05:38:00] rdx: ffff830239463b70   rsi: 0000000000000000   rdi: 0000000000000000
> (XEN) [2014-01-22 05:38:00] rbp: ffff82d0802cfe48   rsp: ffff82d0802cfe08   r8:  0000000000000000
> (XEN) [2014-01-22 05:38:00] r9:  00000000deadbeef   r10: ffff82d080238f20   r11: 0000000000000202
> (XEN) [2014-01-22 05:38:00] r12: ffff830239466700   r13: 0000000000000005   r14: 0000000000000000
> (XEN) [2014-01-22 05:38:00] r15: 0000000000000005   cr0: 0000000080050033   cr4: 00000000001526f0
> (XEN) [2014-01-22 05:38:00] cr3: 000000022dc0c000   cr2: 0000000000000004
> (XEN) [2014-01-22 05:38:00] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) [2014-01-22 05:38:00] Xen stack trace from rsp=ffff82d0802cfe08:
> (XEN) [2014-01-22 05:38:00]    00000070b7313060 0000000000310f00 ffff82d0802cfe68 000000000000001e
> (XEN) [2014-01-22 05:38:00]    ffff880078623e28 ffff8300b7313000 ffff880078716898 0000000000000000
> (XEN) [2014-01-22 05:38:00]    ffff82d0802cfef8 ffff82d08017fede ffff82d08012a25f 0000000000000000
> (XEN) [2014-01-22 05:38:00]    ffff82d000050000 ffff82d08018cdc8 ffff82d080310f00 ffff82d0802cff18
> (XEN) [2014-01-22 05:38:00]    ffff82d0802cfef8 ffff82d08021d98c 0000000000040004 0000000000000246
> (XEN) [2014-01-22 05:38:00]    ffffffff8100122a 0000000000000000 ffffffff8100122a 000000000000e030
> (XEN) [2014-01-22 05:38:00]    0000000000000246 ffff8300b7313000 ffff880070fe2780 0000000000000000
> (XEN) [2014-01-22 05:38:00]    ffff880078716898 0000000000000000 00007d2f7fd300c7 ffff82d08022231b
> (XEN) [2014-01-22 05:38:00]    ffffffff8100142a 0000000000000021 ffff88007f60e0e0 0000000000000000
> (XEN) [2014-01-22 05:38:00]    000000000007e8b5 00000003b5ef9df9 ffff880078623e58 ffff880078716800
> (XEN) [2014-01-22 05:38:00]    0000000000000202 0000000000000594 0000000000000006 0000000000000000
> (XEN) [2014-01-22 05:38:00]    0000000000000021 ffffffff8100142a 0000000000000000 ffff880078623e28
> (XEN) [2014-01-22 05:38:00]    000000000000001e 0001010000000000 ffffffff8100142a 000000000000e033
> (XEN) [2014-01-22 05:38:00]    0000000000000202 ffff880078623e10 000000000000e02b 0000000000000000
> (XEN) [2014-01-22 05:38:00]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) [2014-01-22 05:38:00]    ffff8300b7313000 0000000000000000 0000000000000000
> (XEN) [2014-01-22 05:38:00] Xen call trace:
> (XEN) [2014-01-22 05:38:00]    [<ffff82d080168d51>] pci_prepare_msix+0xb1/0x128
> (XEN) [2014-01-22 05:38:00]    [<ffff82d08017fede>] do_physdev_op+0xd10/0x119e
> (XEN) [2014-01-22 05:38:00]    [<ffff82d08022231b>] syscall_enter+0xeb/0x145
> (XEN) [2014-01-22 05:38:00] 
> (XEN) [2014-01-22 05:38:00] Pagetable walk from 0000000000000004:
> (XEN) [2014-01-22 05:38:00]  L4[0x000] = 0000000000000000 ffffffffffffffff
> (XEN) [2014-01-22 05:38:00] 
> (XEN) [2014-01-22 05:38:00] ****************************************
> (XEN) [2014-01-22 05:38:00] Panic on CPU 0:
> (XEN) [2014-01-22 05:38:00] FATAL PAGE FAULT
> (XEN) [2014-01-22 05:38:00] [error_code=0000]
> (XEN) [2014-01-22 05:38:00] Faulting linear address: 0000000000000004
> (XEN) [2014-01-22 05:38:00] ****************************************
> (XEN) [2014-01-22 05:38:00] 
> (XEN) [2014-01-22 05:38:00] Manual reset required ('noreboot' specified)

This is breakage, caused by 1035bb64fd7fd9f05c510466d98566fd82e37ad9
"PCI: break MSI-X data out of struct pci_dev_info", which made it valid
for a PCI device to not have an associated arch_msix structure.

In pci_prepare_msix(), there is a logic chain

    pdev = pci_get_pdev(seg, bus, devfn);
    if ( !pdev )
        rc = -ENODEV;
    else if ( pdev->msix->used_entries != !!off )
...

which dereferences this optional pointer without first checking whether
the guest-provided PCI device is actually MSI-X capable.

Therefore, dom0 is issuing PHYSDEVOP_prepare_msix hypercalls on PCI
devices Xen believes to be incapable of MSI-X.

~Andrew