Re: e008:[<ffff82c480122353>] check_lock+0x1b/0x45

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 19/06/13 17:06, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> It looks like one of the patches that went in Xen 4.1 blows up.
>
> This used to work over the weekend, so I can only surmise
> it is one the three patches. This is with a debug=y build
> (which has been working nicely for the last month or more).
>
> It looks like one of the patches that went in Xen 4.1 blows up.
>
> This used to work a day ago - that is c/s 23551 worked nicely.
>
> ----- Forwarded message from konrad.wilk@oracle.com -----
>
> Date: Wed, 19 Jun 2013 03:55:10 -0400
> From: konrad.wilk@oracle.com
> To: konrad.wilk@oracle.com
> Subject: FAILURE 3.10.0-rc6upstream-00061-g752bf7d(x86_64) 3.10.0-rc6upstream-00061-g752bf7d(i386): 2013-06-19 (tst007)
>
> --------------------------
> FAILED:  tst007(XEN-x86_64) tst007(XEN-i386):
> --------------------------
> tst007 (XEN-x86_64): FAILED to bootup
> tst007 (XEN-i386): FAILED to bootup
> tst007 (BAREMETAL-x86_64): 3.10.0-rc6upstream-00061-g752bf7d ON
> tst007 (BAREMETAL-i386): 3.10.0-rc6upstream-00061-g752bf7d ON
> --------------------------
> (tst007-XEN) SERIAL
> --------------------------
> ��\x01��\x03��\x03��PXELINUX 3.82 2009-06-09  Copyright (C) 1994-2009 H. Peter Anvin et al
> Loading latest/xen.gz... ok
> Loading latest/vmlinuz... ok
> Loading latest/initramfs.cpio.gz... ok
>  __  __            _  _    _   __
>  \ \/ /___ _ __   | || |  / | / /_     _ __  _ __ ___
>   \  // _ \ '_ \  | || |_ | || '_ \ __| '_ \| '__/ _ \
>   /  \  __/ | | | |__   _|| || (_) |__| |_) | | |  __/
>  /_/\_\___|_| |_|    |_|(_)_(_)___/   | .__/|_|  \___|
>                                       |_|
> (XEN) Xen version 4.1.6-pre (konrad@dumpdata.com) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) Tue Jun 18 18:09:23 EDT 2013
> (XEN) Latest ChangeSet: Mon Jun 17 11:12:41 2013 +0200 23554:2da9f4421c9a
> (XEN) Bootloader: unknown
> (XEN) Command line: com1=115200,8n1 console=com1,vga guest_loglvl=all tmem tmem_compress tmem_dedup dom0_mem=999M,max:1232M dom0_max_vcpus=2 cpufreq=xen:performance,verbose loglvl=all apic=debug
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 1 MBR signatures
> (XEN)  Found 1 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 000000000009ec00 (usable)
> (XEN)  000000000009ec00 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 0000000020000000 (usable)
> (XEN)  0000000020000000 - 0000000020200000 (reserved)
> (XEN)  0000000020200000 - 0000000040000000 (usable)
> (XEN)  0000000040000000 - 0000000040200000 (reserved)
> (XEN)  0000000040200000 - 00000000bad80000 (usable)
> (XEN)  00000000bad80000 - 00000000badc9000 (ACPI NVS)
> (XEN)  00000000badc9000 - 00000000badd1000 (ACPI data)
> (XEN)  00000000badd1000 - 00000000badf4000 (reserved)
> (XEN)  00000000badf4000 - 00000000badf6000 (usable)
> (XEN)  00000000badf6000 - 00000000bae06000 (reserved)
> (XEN)  00000000bae06000 - 00000000bae14000 (ACPI NVS)
> (XEN)  00000000bae14000 - 00000000bae3c000 (reserved)
> (XEN)  00000000bae3c000 - 00000000bae7f000 (ACPI NVS)
> (XEN)  00000000bae7f000 - 00000000bb000000 (usable)
> (XEN)  00000000bb800000 - 00000000bfa00000 (reserved)
> (XEN)  00000000fed1c000 - 00000000fed40000 (reserved)
> (XEN)  00000000ff000000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 000000023fe00000 (usable)
> (XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)
> (XEN) ACPI: XSDT BADC9068, 0054 (r1 ALASKA    A M I  1072009 AMI     10013)
> (XEN)      15 INTL 20051117)
> (XEN) ACPI: FACS BAE0BF80, 0040
> (XEN) ACPI: APIC BADD0400, 0072 (r3 ALASKA    A M I  1072009 AMI     10013)
> (XEN) ACPI: SSDT BADD0478, 0102 (r1 AMICPU     PROC        1 MSFT  3000001)
> (XEN) ACPI: MCFG BADD0580, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
> (XEN) ACPI: HPET BADD05C0, 0038 (r1 ALASKA    A M I  1072009 AMI.        4)
> (XEN) ACPI: ASF! BADD05F8, 00A0 (r32 INTEL       HCG        1 TFSM    F4240)
> (XEN) System RAM: 8104MB (8299140kB)
> (XEN) No NUMA configuration found
> (XEN) Faking a node at 0000000000000000-000000023fe00000
> (XEN) Domain heap initialised
> (XEN) found SMP MP-table at 000fcde0
> (XEN) DMI 2.7 present.
> (XEN) Using APIC driver default
> (XI: 32/64X FACS address mismatch in FADT - bae0bf80/0000000000000000, using 32
> (XEN) ACPI:                  wakeup_vec[bae0bf8c], vec_size[20]
> (XEN) ACPI: Local APIC address 0xfee00000
> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> (XEN) Processor #0 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> (XEN) Processor #2 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
> (XEN) Processor #1 6:10 APIC version 21
> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
> (XEN) Processor #3 6:10 APIC version 21
> (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
> (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> (XEN) PCI: Not using MMCONFIG.
> (XEN) ERST table was not found
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 3093.046 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) mce_intel.c:1162: MCA Capability: BCAST 1 SER 0 CMCine check reporting enabled
> (XEN) I/O virtualisation disabled
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using old ACK method
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) TSC deadline timer enabled
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Allocated console ring of 32 KiB.
> (XEN) VMX: Supported advanced features:
> (XEN)  - Extended Page Tables (EPT)
> (XEN)  - Virtual-Processor Identifiers (VPID)
> (XEN)  - Virtual NMI
> (XEN)  - MSR direct-access bitmap
> (XEN)  - Unrestricted Guest
> (XEN) HVM: ASIDs enabled.
> (XEN) HVM: VMX enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB
> (XEN) Brought up 4 CPUs
> (XEN) tmem: initialized comp=1 dedup=1 tze=0 global-lock=0
> (XEN) ACPI sleep modes: S3
> (XEN) mcheck_poll: Machine check polling(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0x9ef000
> (XEN) elf_parse_binary: phdr: paddr=0x1a00000 memsz=0xae0f0
> (XEN)_O= "linux"
> (XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
> (XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
> (XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
> (XEN) elf_xen_parse_note: ENTRY = 0xffffffff81ac41e0
> (XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
> (XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
> (XEN) elf_xen_parse_note: PAE_MODE = "yes"
> (XEN) elf_xen_parse_note: LOADER = "generic"
> (XEN) elf_xen_parse_note: unknown xen elf note (0xd)
> (XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
> (XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
> (XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
> (XEN) elf_xen_addr_calc_check: addresses:
> (XEN)     virt_base        = 0xffffffff80000000
> (XEN)     elf_paddr_offset = 0x0
> (XEN)     virt_offset      = 0xffffffff80000000
> (XEN)     virt_kstart      = 0xffffffff81000000
> (XEN)     virt_kend        = 0xffffffff82197000
> (XEN)     virt_entry       = 0xffffffff81ac41e0
> (XEN)     p2m_base         = 0xffffffffffffffff
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2197000
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN)  Dom0 alloc.:   0000000224000000->0000000228000000 (182927 pages to be allocated)
> (XEN)  Init. ramdisk: 000000023218f000->000000023fe00000
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN)  Loaded kernel: ffffffff81000000->ffffffff82197000
> (XEN)  Init. ramdisk: ffffffff82197000->ffffffff8fe08000
> (XEN)  Phys-Mach map: ffffffff8fe08000->ffffffff8fffb800
> (XEN)  Start info:    ffffffff8fffc000->ffffffff8fffc4b4
> (XEN)  Page tables:   ffffffff8fffd000->ffffffff90082000
> (XEN)  Boot stack:    ffffffff90082000->ffffffff90083000
> (XEN)  TOTAL:         ffffffff80000000->ffffffff90400000
> (XEN)  ENTRY ADDRESS: ffffffff81ac41e0
> (XEN) Dom0 has maximum 2 VCPUs
> (XEN) elf_load_binary: phdr 0 at 0x18446744071578845184 -> 0x18446744071589261312
> (XEN) elf_load_binary: phdr 1 at 0x18446744071589330944 -> 0x18446744071590043888
> (XEN) elf_load_binary: phdr 2 at 0x18446744071590047744 -> 0x18446744071590133248
> (XEN) elf_load_binary: phdr 3 at 0x18446744071590133760 -> 0x18446744071591841792
> (XEN) Scrubbing Free RAM: .....................................................................done.
> (XEN) Xen trace buffers: disabled
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) Xen is relinquishing VGA co
> (XEN) irq.c:1639: dom0: pirq 9 or irq 9 already mapped
> (XEN) irq.c:1639: dom0: pirq 1 or irq 1 already mapped

This is because of a map_domain_pirq hypercall.  The implication is that
there are multiple redundant hypercalls being made.

> (XEN) ----[ Xen-4.1.6-pre  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c480122353>] check_lock+0x1b/0x45
> (XEN) RFLAGS: 0000000000010002   CONTEXT: hypervisor
> (XEN) rax: 0000000000000001   rbx: ffff8312a707a4b4   rcx: 0000000000000000
> (XEN) rdx: 0000000000000001   rsi: 0000000000000001   rdi: ffff8312a707a4b8
> (XEN) rbp: ffff82c480297dc8   rsp: ffff82c480297dc8   r8:  00000000deadbeef
> (XEN) r9:  00000000deadbeef   r10: ffff82c4802177a0   r11: 0000000000000246
> (XEN) r12: 0000000000000286   r13: ffff82c480297e40   r14: ffff8312a707a480
> (XEN) r15: 0000000020ec8348   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 0000000225a0c000   cr2: ffff8312a707a4b8
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=ffff82c480297dc8:
> (XEN)    ffff82c480297de8 ffff82c48012272a ffff830230c0a000 0000000000000004
> (XEN)    ffff8312a707a4b4 ffff82c48015ef3d ffff82c480297e48 0000000000000000
> (XEN)    ffff82c480297ef8 0000000000000001 ffffffff81a01d88 ffff8300baff4000
> (XEN)    ffffffff81996456 ffff82c48015ef81 ffffffff81a01d88 ffff8300baff4000
> (XEN)    0000000000000005 ffff82c480173625 0000000000000002 ffff82c480297e78
> (XEN)    ffff82c480122798 ffff82c480258160 ffff82c480297e98 ffff82c48012407d
> (XEN)    ffff82c4802d55e0 0000000000000000 0000000100000001 ffff82c48012411e
> (XEN)    ffff82c4802b0880 00000000ffffffff ffff82c480297ef8 ffff82c480121fba
> (XEN)    ffff82c48015854f ffff8300baff4000 0000000000000001 ffff88003dc07200
> (XEN)    0000000000000001 ffffffff81996456 00007d3b7fd680c7 ffff82c4802056d8
> (XEN)    ffffffff8100142a 0000000000000021 ffffffff81996456 0000000000000001
> (XEN)    ffff88003dc07200 0000000000000001 ffffffff81a01da8 ffff88003dc07200
> (XEN)    0000000000000246 ffff88003dc07200 0000000000000000 00000000a58856ce
> (XEN)    0000000000000021 ffffffff8100142a ffff88003dc05600 ffffffff81a01d88
> (XEN)    0000000000000005 0000010000000000 ffffffff8100142a 000000000000e033

This 0x0000010000000000 means TRAP_syscall

The 0xffffffff8100142a means hypercall 33, __HYPERVISOR_physdev_op

> (XEN)    0000000000000246 ffffffff81a01d70 000000000000e02b 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    ffff8300baff4000 0000000000000000 0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<ffff82c480122353>] check_lock+0x1b/0x45
> (XEN)    [<ffff82c48012272a>] _spin_lock_irqsave+0x21/0x67
> (XEN)    [<ffff82c48015ef3d>] domain_spin_lock_irq_desc+0x6d/0xa0
> (XEN)
> (XEN) Pagetable walk from ffff8312a707a4b8:
> (XEN)  L4[0x106] = 00000000baaa1027 5555555555555555
> (XEN)  L3[0x04a] = 0000000000000000 ffffffffffffffff
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) FATAL PAGE FAULT
> (XEN) [error_code=0000]
> (XEN) Faulting linear address: ffff8312a707a4b8
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
>
> PXELINUX 3.82 2009-06-09  Copyright (C) 1994-2009 H. Peter Anvin et al
>

But the stack trace looks wonky.

I suspect we have followed a bad irq_desc pointer.

~Andrew


<SNIPPING LOTS>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel