* Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
@ 2013-11-28 12:31 Andrew Cooper
2013-11-28 13:05 ` Dario Faggioli
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Andrew Cooper @ 2013-11-28 12:31 UTC (permalink / raw)
To: Xen-devel List, Dario Faggioli, George Dunlap; +Cc: Jan Beulich
[-- Attachment #1: Type: text/plain, Size: 515 bytes --]
Hello,
I have recently positivly identified
b54a623efbcf5bff25c55117add1b4427b4e2f1b as causing a boot failure.
Serial log is attached. The crash is completely deterministic, and is
from an IBM xSeries 3530 M4 server.
Given the crash and bad patch, I suspect it is more to do with the
NUMA/memory layout than the specifics of the server.
Dario: Being your patch, do you have any ideas?
George: Regarding the release, if a fix cant easily be found, it might
be worth considering reverting the change.
~Andrew
[-- Attachment #2: CA-122685-serial.log --]
[-- Type: text/x-log, Size: 18395 bytes --]
;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, visit that file with C-x C-f,
;; then enter the text in that file's own buffer.
__ __ _ _ _____ _
\ \/ /___ _ __ | || | |___ / / |
\ // _ \ '_ \ | || |_ |_ \ | |
/ \ __/ | | | |__ _| ___) || |
/_/\_\___|_| |_| |_|(_)____(_)_|
(XEN) Xen version 4.3.1 (andrewcoop@uk.xensource.com) (x86_64-linux-gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46)) debug=y Thu Nov 28 06:08:49 EST 2013
(XEN) Latest ChangeSet: 27324:91d760a1fef1, pq 164:f208f8d47dcd
(XEN) Bootloader: PXELINUX 4.04 2011-04-18
(XEN) Command line: watchdog com1=115200,8n1 console=com1 noreboot
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 2 MBR signatures
(XEN) Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009b800 (usable)
(XEN) 000000000009b800 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000673a7000 (usable)
(XEN) 00000000673a7000 - 00000000673a8000 (ACPI data)
(XEN) 00000000673a8000 - 0000000069c05000 (usable)
(XEN) 0000000069c05000 - 0000000069cc9000 (reserved)
(XEN) 0000000069cc9000 - 000000006ad8b000 (usable)
(XEN) 000000006ad8b000 - 000000006ad92000 (ACPI data)
(XEN) 000000006ad92000 - 000000006ad95000 (usable)
(XEN) 000000006ad95000 - 000000006ad96000 (ACPI data)
(XEN) 000000006ad96000 - 000000006ad98000 (usable)
(XEN) 000000006ad98000 - 000000006ad9f000 (ACPI data)
(XEN) 000000006ad9f000 - 000000006adb1000 (usable)
(XEN) 000000006adb1000 - 000000006adb4000 (ACPI data)
(XEN) 000000006adb4000 - 000000006adb5000 (usable)
(XEN) 000000006adb5000 - 000000006adb8000 (ACPI data)
(XEN) 000000006adb8000 - 000000006adbc000 (usable)
(XEN) 000000006adbc000 - 000000006adbf000 (ACPI data)
(XEN) 000000006adbf000 - 000000006adc7000 (usable)
(XEN) 000000006adc7000 - 000000006adcc000 (ACPI data)
(XEN) 000000006adcc000 - 000000006b6b3000 (usable)
(XEN) 000000006b6b3000 - 000000006b6b4000 (ACPI data)
(XEN) 000000006b6b4000 - 000000006b98d000 (usable)
(XEN) 000000006b98d000 - 000000006b98e000 (ACPI data)
(XEN) 000000006b98e000 - 000000007e78a000 (usable)
(XEN) 000000007e78a000 - 000000007e893000 (reserved)
(XEN) 000000007e893000 - 000000007ebb5000 (ACPI NVS)
(XEN) 000000007ebb5000 - 000000007ec02000 (usable)
(XEN) 000000007f000000 - 000000007fc00000 (reserved)
(XEN) 0000000080000000 - 0000000090000000 (reserved)
(XEN) 00000000fed1c000 - 00000000fed20000 (reserved)
(XEN) 00000000ff800000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000c80000000 (usable)
(XEN) ACPI: RSDP 000FDFD0, 0024 (r2 IBM )
(XEN) ACPI: XSDT 6B98D1C0, 00C4 (r1 IBM SYSTEM_X 0 1000013)
(XEN) ACPI: FACP 6ADCA000, 00F4 (r4 IBM SYSTEM_X 0 MSFT 1000019)
(XEN) ACPI: DSDT 6AD9A000, 486F (r1 INTEL TIANO 3 MSFT 1000013)
(XEN) ACPI: FACS 7E91A000, 0040
(XEN) ACPI: TCPA 6B6B3000, 0064 (r0 0 0)
(XEN) ACPI: ERST 6ADCB000, 0230 (r1 IBM SYSTEM_X 1 MSFT 100001B)
(XEN) ACPI: HEST 6ADC9000, 00F4 (r1 IBM SYSTEM_X 1 MSFT 100001E)
(XEN) ACPI: HPET 6ADC8000, 0038 (r1 IBM SYSTEM_X 1 MSFT 1000017)
(XEN) ACPI: APIC 6ADC7000, 012A (r3 IBM SYSTEM_X 0 MSFT 1000014)
(XEN) ACPI: MCFG 6ADBE000, 003C (r1 IBM SYSTEM_X 1 MSFT 1000018)
(XEN) ACPI: OEM0 6ADB7000, 0268 (r3 IBM XSECSRAT 100 MSFT 1000022)
(XEN) ACPI: OEM1 6ADBD000, 0030 (r1 IBM IBMERROR 1 MSFT 1000013)
(XEN) ACPI: SLIT 6ADBC000, 0030 (r1 IBM SYSTEM_X 1 MSFT 1000020)
(XEN) ACPI: SRAT 6ADB6000, 0228 (r3 IBM SYSTEM_X 1 MSFT 100001A)
(XEN) ACPI: SLIC 6ADB5000, 0176 (r1 IBM SYSTEM_X 0 MSFT 100001F)
(XEN) ACPI: SSDT 6ADB2000, 0527 (r2 IBM CPUSCOPE 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6ADB1000, 052E (r2 IBM CPUWYVRN 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6AD8B000, 6144 (r2 IBM PSTATEPM 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6ADB3000, 0BE2 (r2 IBM CPUCSTAT 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6AD99000, 0214 (r2 IBM WYVRNDEV 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6AD98000, 009E (r2 IBM WYVRNGPE 4000 MSFT 1000016)
(XEN) ACPI: SSDT 6AD95000, 06EF (r2 IBM CPUNOTFY 4000 MSFT 1000016)
(XEN) ACPI: DMAR 673A7000, 0158 (r1 IBM SYSTEM_X 1 MSFT 1000021)
(XEN) System RAM: 49126MB (50305592kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 6 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 8 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 10 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 32 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 34 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 36 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 38 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 40 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 42 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 7 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 9 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 11 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 33 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 35 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 37 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 39 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 41 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 43 -> Node 1
(XEN) SRAT: Node 0 PXM 0 0-80000000
(XEN) SRAT: Node 0 PXM 0 100000000-680000000
(XEN) SRAT: Node 1 PXM 1 680000000-c80000000
(XEN) NUMA: Using 19 for the hash shift.
(XEN) Domain heap initialised DMA width 32 bits
(XEN) found SMP MP-table at 0009b940
(XEN) DMI 2.7 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI: wakeup_vec[7e91a00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
(XEN) Processor #2 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
(XEN) Processor #4 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
(XEN) Processor #6 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
(XEN) Processor #8 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
(XEN) Processor #10 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled)
(XEN) Processor #32 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled)
(XEN) Processor #34 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled)
(XEN) Processor #36 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled)
(XEN) Processor #38 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled)
(XEN) Processor #40 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled)
(XEN) Processor #42 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x01] enabled)
(XEN) Processor #1 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x03] enabled)
(XEN) Processor #3 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x05] enabled)
(XEN) Processor #5 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled)
(XEN) Processor #7 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x09] enabled)
(XEN) Processor #9 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x0b] enabled)
(XEN) Processor #11 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x21] enabled)
(XEN) Processor #33 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x19] lapic_id[0x23] enabled)
(XEN) Processor #35 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x25] enabled)
(XEN) Processor #37 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x27] enabled)
(XEN) Processor #39 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x29] enabled)
(XEN) Processor #41 6:13 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x2b] enabled)
(XEN) Processor #43 6:13 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
(XEN) ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48])
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) ACPI: HPET id: 0x8086a701 base: 0xfed00000
(XEN) Invalid bit width in GAR
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 72 GSI, 4552 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 1900.046 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 1 CMCI 1 firstbank 0 extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base 80000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at 80000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) [2013-11-28 11:20:54] Platform timer is 14.318MHz HPET
(XEN) [2013-11-28 11:20:54] Defaulting to alternative key handling; send 'A' to switch to normal mode.
(XEN) [2013-11-28 11:20:54] Allocated console ring of 256 KiB.
(XEN) [2013-11-28 11:20:54] mwait-idle: MWAIT substates: 0x21120
(XEN) [2013-11-28 11:20:54] mwait-idle: v0.4 model 0x2d
(XEN) [2013-11-28 11:20:54] mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) [2013-11-28 11:20:54] VMX: Supported advanced features:
(XEN) [2013-11-28 11:20:54] - APIC MMIO access virtualisation
(XEN) [2013-11-28 11:20:54] - APIC TPR shadow
(XEN) [2013-11-28 11:20:54] - Extended Page Tables (EPT)
(XEN) [2013-11-28 11:20:54] - Virtual-Processor Identifiers (VPID)
(XEN) [2013-11-28 11:20:54] - Virtual NMI
(XEN) [2013-11-28 11:20:54] - MSR direct-access bitmap
(XEN) [2013-11-28 11:20:54] - Unrestricted Guest
(XEN) [2013-11-28 11:20:54] HVM: ASIDs enabled.
(XEN) [2013-11-28 11:20:54] HVM: VMX enabled
(XEN) [2013-11-28 11:20:54] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [2013-11-28 11:20:54] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [2013-11-28 11:20:54] Brought up 24 CPUs
(XEN) [2013-11-28 11:20:54] Testing NMI watchdog --- CPU#0 okay. CPU#1 okay. CPU#2 okay. CPU#3 okay. CPU#4 okay. CPU#5 okay. CPU#6 okay. CPU#7 okay. CPU#8 okay. CPU#9 okay. CPU#10 okay. CPU#11 okay. CPU#12 okay. CPU#13 okay. CPU#14 okay. CPU#15 okay. CPU#16 okay. CPU#17 okay. CPU#18 okay. CPU#19 okay. CPU#20 okay. CPU#21 okay. CPU#22 okay. CPU#23 okay.
(XEN) [2013-11-28 11:20:55] ACPI sleep modes: S3
(XEN) [2013-11-28 11:20:55] mcheck_poll: Machine check polling timer started.
(XEN) [2013-11-28 11:20:55] *** LOADING DOMAIN 0 ***
(XEN) [2013-11-28 11:20:55] elf_parse_binary: phdr: paddr=0x100000 memsz=0x3fd000
(XEN) [2013-11-28 11:20:55] elf_parse_binary: phdr: paddr=0x4fd000 memsz=0x28a000
(XEN) [2013-11-28 11:20:55] elf_parse_binary: memory: 0x100000 -> 0x787000
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: GUEST_OS = "linux"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: VIRT_BASE = 0xc0000000
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: ENTRY = 0xc0100000
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: HYPERCALL_PAGE = 0xc0101000
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: HV_START_LOW = 0xf5800000
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: FEATURES = "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: PAE_MODE = "yes"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: LOADER = "generic"
(XEN) [2013-11-28 11:20:55] elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) [2013-11-28 11:20:55] elf_xen_addr_calc_check: addresses:
(XEN) [2013-11-28 11:20:55] virt_base = 0xc0000000
(XEN) [2013-11-28 11:20:55] elf_paddr_offset = 0x0
(XEN) [2013-11-28 11:20:55] virt_offset = 0xc0000000
(XEN) [2013-11-28 11:20:55] virt_kstart = 0xc0100000
(XEN) [2013-11-28 11:20:55] virt_kend = 0xc0787000
(XEN) [2013-11-28 11:20:55] virt_entry = 0xc0100000
(XEN) [2013-11-28 11:20:55] p2m_base = 0xffffffffffffffff
(XEN) [2013-11-28 11:20:55] Xen kernel: 64-bit, lsb, compat32
(XEN) [2013-11-28 11:20:55] Dom0 kernel: 32-bit, PAE, lsb, paddr 0x100000 -> 0x787000
(XEN) [2013-11-28 11:20:55] ----[ Xen-4.3.1 x86_64 debug=y Not tainted ]----
(XEN) [2013-11-28 11:20:55] CPU: 0
(XEN) [2013-11-28 11:20:55] RIP: e008:[<ffff82c4c0118883>] assign_pages+0x1e8/0x238
(XEN) [2013-11-28 11:20:55] RFLAGS: 0000000000010a06 CONTEXT: hypervisor
(XEN) [2013-11-28 11:20:55] rax: 07fffc1700c5c000 rbx: 00007d2000000000 rcx: ffff82e018b80000
(XEN) [2013-11-28 11:20:55] rdx: 0000000000000000 rsi: ffff82e018b80000 rdi: ffff83043ff08020
(XEN) [2013-11-28 11:20:55] rbp: ffff82c4c02e7848 rsp: ffff82c4c02e7818 r8: 0000000000000000
(XEN) [2013-11-28 11:20:55] r9: ffff83043ff08020 r10: 000000000043ff08 r11: 0000000000004000
(XEN) [2013-11-28 11:20:55] r12: 00007d0000000000 r13: ffff82e018b80000 r14: ffff83043ff08000
(XEN) [2013-11-28 11:20:55] r15: 0000000000004000 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) [2013-11-28 11:20:55] cr3: 000000007e485000 cr2: 0000000000000000
(XEN) [2013-11-28 11:20:55] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) [2013-11-28 11:20:55] Xen stack trace from rsp=ffff82c4c02e7818:
(XEN) [2013-11-28 11:20:55] ffff830c7fab2074 0000000000000027 ffff82e018b80000 0000000000000000
(XEN) [2013-11-28 11:20:55] ffff83043ff08000 000000000000000e ffff82c4c02e7888 ffff82c4c0119dcc
(XEN) [2013-11-28 11:20:55] 0000000000000003 000000000000000e 0000000000000000 0000000000bdc912
(XEN) [2013-11-28 11:20:55] 0000000000000fff 0000000003800000 ffff82c4c02e7da8 ffff82c4c02ba2b1
(XEN) [2013-11-28 11:20:55] ffff82c4c012a591 ffff82c4c0252a35 ffff82c4c02e7990 0000000000000003
(XEN) [2013-11-28 11:20:55] ffff82c4c0334320 0000000000780000 ffff82c4c02e7908 ffff82c4c02a80d2
(XEN) [2013-11-28 11:20:55] ffff82c4c0334b20 ffff82c4c02e7908 ffff82c4c02a824f 0000000000000100
(XEN) [2013-11-28 11:20:55] ffff82c4c02d4c40 ffff82c4c02a8961 0000000000000000 ffff83043ff08000
(XEN) [2013-11-28 11:20:55] ffff82c4c02e7948 0000000000bfe68e 0000000000000001 00ff82c4c02e7948
(XEN) [2013-11-28 11:20:55] ffff82c432303534 0000000000000021 ffff82c4c02e7978 00ff82c4c0144d38
(XEN) [2013-11-28 11:20:55] 0000000032303534 ffff82c4c0252a34 ffff82c4c02877c0 ffff83006b98e000
(XEN) [2013-11-28 11:20:55] 0000000000000000 ffff82c4c012a9df ffff82c4c02e79c8 ffff82c4c018bb6b
(XEN) [2013-11-28 11:20:55] ffff83043b400400 00000000c0100000 00000000c0787000 00000000c0787000
(XEN) [2013-11-28 11:20:55] 00000000c0787000 00000000c0787000 00000000c36c1800 00000000c36c2000
(XEN) [2013-11-28 11:20:55] 00000000c36c24b4 00000000c36e5000 00000000c36c3000 00000000c36e4000
(XEN) [2013-11-28 11:20:55] 00000000c0000000 00000000c3800000 0000000000bcea00 00ff82c4c02e7cb8
(XEN) [2013-11-28 11:20:55] 0000000000000000 ffff82c4c02877c0 ffff82c4c02e7b58 ffff83043b400400
(XEN) [2013-11-28 11:20:55] ffff82c4c02e7a68 ffff82c4c02e7bea ffff82c4c02e7c50 00000000c02e7bf7
(XEN) [2013-11-28 11:20:55] 0000000000000001 0000000000000002 ffff82c4c02e7ac8 ffff82c4c02e7c1a
(XEN) [2013-11-28 11:20:55] 0000000000000000 ffff830c7fff4000 ffff830c7fff418c 0000000000000000
(XEN) [2013-11-28 11:20:55] Xen call trace:
(XEN) [2013-11-28 11:20:55] [<ffff82c4c0118883>] assign_pages+0x1e8/0x238
(XEN) [2013-11-28 11:20:55] [<ffff82c4c0119dcc>] alloc_domheap_pages+0x13a/0x170
(XEN) [2013-11-28 11:20:55] [<ffff82c4c02ba2b1>] construct_dom0+0x92d/0x35ce
(XEN) [2013-11-28 11:20:55] [<ffff82c4c02afa3b>] __start_xen+0x6922/0x6997
(XEN) [2013-11-28 11:20:55]
(XEN) [2013-11-28 11:20:55] Pagetable walk from 0000000000000000:
(XEN) [2013-11-28 11:20:55] L4[0x000] = 000000043fff4063 ffffffffffffffff
(XEN) [2013-11-28 11:20:55] L3[0x000] = 000000043fff3063 ffffffffffffffff
(XEN) [2013-11-28 11:20:55] L2[0x000] = 000000043fff2063 ffffffffffffffff
(XEN) [2013-11-28 11:20:55] L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN) [2013-11-28 11:20:55]
(XEN) [2013-11-28 11:20:55] ****************************************
(XEN) [2013-11-28 11:20:55] Panic on CPU 0:
(XEN) [2013-11-28 11:20:55] FATAL PAGE FAULT
(XEN) [2013-11-28 11:20:55] [error_code=0002]
(XEN) [2013-11-28 11:20:55] Faulting linear address: 0000000000000000
(XEN) [2013-11-28 11:20:55] ****************************************
(XEN) [2013-11-28 11:20:55]
(XEN) [2013-11-28 11:20:55] Manual reset required ('noreboot' specified)
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 12:31 Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode" Andrew Cooper
@ 2013-11-28 13:05 ` Dario Faggioli
2013-11-28 15:09 ` George Dunlap
2013-11-28 21:17 ` Andrew Cooper
2 siblings, 0 replies; 12+ messages in thread
From: Dario Faggioli @ 2013-11-28 13:05 UTC (permalink / raw)
To: Andrew Cooper; +Cc: George Dunlap, Jan Beulich, Xen-devel List
[-- Attachment #1.1.1: Type: text/plain, Size: 1061 bytes --]
On gio, 2013-11-28 at 12:31 +0000, Andrew Cooper wrote:
> Hello,
>
Hi,
> I have recently positivly identified
> b54a623efbcf5bff25c55117add1b4427b4e2f1b as causing a boot failure.
>
> Serial log is attached. The crash is completely deterministic, and is
> from an IBM xSeries 3530 M4 server.
>
> Given the crash and bad patch, I suspect it is more to do with the
> NUMA/memory layout than the specifics of the server.
>
> Dario: Being your patch, do you have any ideas?
>
Wow... Not out of the top of my head... Can you try (or tell me how to
do that on that box) the attached debug patch?
I know it's gross, but given where and how early the crash happens, it
shouldn't be too bad, and it would hopefully at least tell if
d->node_affinity contains garbage.
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.1.2: node_affinity.patch --]
[-- Type: text/x-patch, Size: 1085 bytes --]
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 2cbc489..06b099f 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -348,6 +348,13 @@ struct domain *domain_create(
return ERR_PTR(err);
}
+static void nodeset_print(char *set, int size, const nodemask_t *mask)
+{
+ *set++ = '[';
+ set += nodelist_scnprintf(set, size-2, mask);
+ *set++ = ']';
+ *set++ = '\0';
+}
void domain_update_node_affinity(struct domain *d)
{
@@ -357,6 +364,8 @@ void domain_update_node_affinity(struct domain *d)
struct vcpu *v;
unsigned int node;
+ char tmpstr[256];
+
if ( !zalloc_cpumask_var(&cpumask) )
return;
if ( !alloc_cpumask_var(&online_affinity) )
@@ -394,6 +403,9 @@ void domain_update_node_affinity(struct domain *d)
spin_unlock(&d->node_affinity_lock);
+ nodeset_print(tmpstr, sizeof(tmpstr), &d->node_affinity);
+ printk("--VCPU: %d, d->node_affinity: %s\n", v->vcpu_id, tmpstr);
+
free_cpumask_var(online_affinity);
free_cpumask_var(cpumask);
}
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 12:31 Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode" Andrew Cooper
2013-11-28 13:05 ` Dario Faggioli
@ 2013-11-28 15:09 ` George Dunlap
2013-11-28 15:14 ` Dario Faggioli
2013-11-28 21:17 ` Andrew Cooper
2 siblings, 1 reply; 12+ messages in thread
From: George Dunlap @ 2013-11-28 15:09 UTC (permalink / raw)
To: Andrew Cooper, Xen-devel List, Dario Faggioli; +Cc: Jan Beulich
On 11/28/2013 12:31 PM, Andrew Cooper wrote:
> Hello,
>
> I have recently positivly identified
> b54a623efbcf5bff25c55117add1b4427b4e2f1b as causing a boot failure.
>
> Serial log is attached. The crash is completely deterministic, and is
> from an IBM xSeries 3530 M4 server.
>
> Given the crash and bad patch, I suspect it is more to do with the
> NUMA/memory layout than the specifics of the server.
>
> Dario: Being your patch, do you have any ideas?
Do you have a xen-syms you can use to find out what line the crash
happened at?
Dom0 should have auto_node_affinity set at this point; so before this
patch you'd have:
nodemask = NODEMASK_MASK_NONE;
[set nodes in nodemask from cpumask]
d->node_affinity=nodemask
After, you have:
nodes_clear(d->node_affinity)
[set nodes in d->node_affinity from cpumask]
Everything looks like it should be the same.
Can you try just reverting what's in the positive side of the if()?
I.e., adding back in nodemask=NODE_MASK_NONE at the top, and the
nodemask copying, and see what happens?
-George
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 15:09 ` George Dunlap
@ 2013-11-28 15:14 ` Dario Faggioli
2013-11-28 15:16 ` Andrew Cooper
0 siblings, 1 reply; 12+ messages in thread
From: Dario Faggioli @ 2013-11-28 15:14 UTC (permalink / raw)
To: George Dunlap; +Cc: Andrew Cooper, Jan Beulich, Xen-devel List
[-- Attachment #1.1: Type: text/plain, Size: 1242 bytes --]
On gio, 2013-11-28 at 15:09 +0000, George Dunlap wrote:
> On 11/28/2013 12:31 PM, Andrew Cooper wrote:
> Do you have a xen-syms you can use to find out what line the crash
> happened at?
>
Yep, that would be helpful...
> Dom0 should have auto_node_affinity set at this point; so before this
> patch you'd have:
> nodemask = NODEMASK_MASK_NONE;
> [set nodes in nodemask from cpumask]
> d->node_affinity=nodemask
>
> After, you have:
> nodes_clear(d->node_affinity)
> [set nodes in d->node_affinity from cpumask]
>
> Everything looks like it should be the same.
>
Exactly! I really don't see what could be happening.
Anyway, I've now access to the machine and can try doing some
debugging...
> Can you try just reverting what's in the positive side of the if()?
> I.e., adding back in nodemask=NODE_MASK_NONE at the top, and the
> nodemask copying, and see what happens?
>
...Including this, and let you know.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 15:14 ` Dario Faggioli
@ 2013-11-28 15:16 ` Andrew Cooper
0 siblings, 0 replies; 12+ messages in thread
From: Andrew Cooper @ 2013-11-28 15:16 UTC (permalink / raw)
To: Dario Faggioli; +Cc: George Dunlap, Jan Beulich, Xen-devel List
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 28/11/13 15:14, Dario Faggioli wrote:
> On gio, 2013-11-28 at 15:09 +0000, George Dunlap wrote:
>> On 11/28/2013 12:31 PM, Andrew Cooper wrote:
>> Do you have a xen-syms you can use to find out what line the crash
>> happened at?
>>
> Yep, that would be helpful...
The exact location of the crash was xen/include/xen/mm.h:188 in
page_list_add_tail()
static inline void
page_list_add_tail(struct page_info *page, struct page_list_head *head)
{
page->list.next = PAGE_LIST_NULL;
if ( head->next )
{
page->list.prev = page_to_pdx(head->tail);
head->tail->list.next = page_to_pdx(page); <--- crash on this
line.
}
else
{
page->list.prev = PAGE_LIST_NULL;
head->next = page;
}
head->tail = page;
}
~Andrew
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/
iQIcBAEBAgAGBQJSl15AAAoJEGjbQLT+DFRo8tQP/1bVKK8bpwrWB1QU7Ka3KCiD
Are1jpklsPRVBLdTcYHlQ6nYGt+V7++QRhHhbBpEATdq5GCrlT3EacDGKNw9ECRt
aRVWGJXbUdgwGJxy3CSQbN4OwiefjowYJn9Za/MMNlXAxzMOaqf/duYak8gRtbeY
M5JRciDAjVrniMMqMem0dUXOaPUTvOPTleFlix9almYer9j6qPDF9ei7BcD9buLm
HgFAs3laNS4V90jqXIiK85GC5bSVTiAN8PJROGApfScp4DDRGC3m+Bg4JBi5cvMI
s0++BBm2LGKSJT/uAIU8z0+GJB8W+za766rY16bPc9km4EIeBgwsKEDU8SSIcRUV
WbUVtJKBb8W4mQFfInpx0e70nuwJyQB8jIbof+joaFxdrEIPsduqIVEYT2/Gmzc4
kjUcrzMRlspJQXJU7JgewC8niQQ7ro3+F2uvhLjT/LJza80vcrBvD68qN8hRHp0v
DsNXpdqimjRafMkpzzBftd5dBJR41AlOUQL2kwGbLBDFHMKexNCzQaq826gTFUCQ
maP/XzluTwW2U2eiHQmR+fdyoLSQlBZcZAm5yh22sZMGWHTHeVB/AOawJRdfxyzC
OW9fRFL0+SvYGCcGPnE2bU+ZKZtcqPNQjawph+HAQumagtJvnY35ABmWqcDP8/CQ
mzvibxQxVkdOPWfqulG6
=2Io0
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 12:31 Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode" Andrew Cooper
2013-11-28 13:05 ` Dario Faggioli
2013-11-28 15:09 ` George Dunlap
@ 2013-11-28 21:17 ` Andrew Cooper
2013-11-28 23:30 ` George Dunlap
2013-11-29 10:51 ` Ian Campbell
2 siblings, 2 replies; 12+ messages in thread
From: Andrew Cooper @ 2013-11-28 21:17 UTC (permalink / raw)
To: Xen-devel List, Dario Faggioli, George Dunlap; +Cc: Jan Beulich
On 28/11/13 12:31, Andrew Cooper wrote:
> Hello,
>
> I have recently positivly identified
> b54a623efbcf5bff25c55117add1b4427b4e2f1b as causing a boot failure.
>
> Serial log is attached. The crash is completely deterministic, and is
> from an IBM xSeries 3530 M4 server.
>
> Given the crash and bad patch, I suspect it is more to do with the
> NUMA/memory layout than the specifics of the server.
>
> Dario: Being your patch, do you have any ideas?
>
> George: Regarding the release, if a fix cant easily be found, it might
> be worth considering reverting the change.
>
> ~Andrew
Following some further debugging, this is rather more complicated than I
initially thought.
There is some form of memory corruption; depending on which exact
underlying changeset I base the XenServer patch queue on, or which pages
are present in the queue, I get crashes in different locations,
including faults from mis-aligned instructions including stack traces
which are completely bogus.
The saving grace is that the crashes appear to be completely
deterministic for a given binary. (although this sever is slower than
treacle to boot)
~Andrew
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 21:17 ` Andrew Cooper
@ 2013-11-28 23:30 ` George Dunlap
2013-11-29 10:51 ` Ian Campbell
1 sibling, 0 replies; 12+ messages in thread
From: George Dunlap @ 2013-11-28 23:30 UTC (permalink / raw)
To: Andrew Cooper, Xen-devel List, Dario Faggioli; +Cc: Jan Beulich
On 11/28/2013 09:17 PM, Andrew Cooper wrote:
> On 28/11/13 12:31, Andrew Cooper wrote:
>> Hello,
>>
>> I have recently positivly identified
>> b54a623efbcf5bff25c55117add1b4427b4e2f1b as causing a boot failure.
>>
>> Serial log is attached. The crash is completely deterministic, and is
>> from an IBM xSeries 3530 M4 server.
>>
>> Given the crash and bad patch, I suspect it is more to do with the
>> NUMA/memory layout than the specifics of the server.
>>
>> Dario: Being your patch, do you have any ideas?
>>
>> George: Regarding the release, if a fix cant easily be found, it might
>> be worth considering reverting the change.
>>
>> ~Andrew
>
> Following some further debugging, this is rather more complicated than I
> initially thought.
>
> There is some form of memory corruption; depending on which exact
> underlying changeset I base the XenServer patch queue on, or which pages
> are present in the queue, I get crashes in different locations,
> including faults from mis-aligned instructions including stack traces
> which are completely bogus.
>
> The saving grace is that the crashes appear to be completely
> deterministic for a given binary. (although this sever is slower than
> treacle to boot)
Well, one thing that patch certainly *does* do is remove a very large
chunk of zeroed bytes from the stack (doing the work directly in the
domain struct rather than doing it on the stack and then copying it in);
so it's possible you're got an uninitialized variable somewhere...
-George
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-28 21:17 ` Andrew Cooper
2013-11-28 23:30 ` George Dunlap
@ 2013-11-29 10:51 ` Ian Campbell
2013-11-29 11:04 ` Andrew Cooper
1 sibling, 1 reply; 12+ messages in thread
From: Ian Campbell @ 2013-11-29 10:51 UTC (permalink / raw)
To: Andrew Cooper; +Cc: George Dunlap, Dario Faggioli, Jan Beulich, Xen-devel List
On Thu, 2013-11-28 at 21:17 +0000, Andrew Cooper wrote:
> the XenServer patch queue on
Are you positive that the bug is in the underlying Xen tree and not some
interaction with a patch in your queue?
A boot time issue ought to be reasonably easy to test with a bare tree.
Ian.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-29 10:51 ` Ian Campbell
@ 2013-11-29 11:04 ` Andrew Cooper
2013-12-02 14:01 ` Andrew Cooper
0 siblings, 1 reply; 12+ messages in thread
From: Andrew Cooper @ 2013-11-29 11:04 UTC (permalink / raw)
To: Ian Campbell; +Cc: George Dunlap, Dario Faggioli, Jan Beulich, Xen-devel List
On 29/11/13 10:51, Ian Campbell wrote:
> On Thu, 2013-11-28 at 21:17 +0000, Andrew Cooper wrote:
>> the XenServer patch queue on
> Are you positive that the bug is in the underlying Xen tree and not some
> interaction with a patch in your queue?
>
> A boot time issue ought to be reasonably easy to test with a bare tree.
>
> Ian.
>
I am not sure of anything at the moment, although I have found one
instance of a crash with none of the XenServer patch queue whatsoever.
At the moment, I have narrowed the problem down to a handful of
instructions writing 0s into a well-formed region of the stack.
Clearly, this is not correct, and every tweak of the debugging causes
the problem to jump around.
~Andrew
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-11-29 11:04 ` Andrew Cooper
@ 2013-12-02 14:01 ` Andrew Cooper
2013-12-02 14:36 ` Jan Beulich
0 siblings, 1 reply; 12+ messages in thread
From: Andrew Cooper @ 2013-12-02 14:01 UTC (permalink / raw)
To: Ian Campbell; +Cc: George Dunlap, Dario Faggioli, Jan Beulich, Xen-devel List
On 29/11/13 11:04, Andrew Cooper wrote:
> On 29/11/13 10:51, Ian Campbell wrote:
>> On Thu, 2013-11-28 at 21:17 +0000, Andrew Cooper wrote:
>>> the XenServer patch queue on
>> Are you positive that the bug is in the underlying Xen tree and not some
>> interaction with a patch in your queue?
>>
>> A boot time issue ought to be reasonably easy to test with a bare tree.
>>
>> Ian.
>>
> I am not sure of anything at the moment, although I have found one
> instance of a crash with none of the XenServer patch queue whatsoever.
>
> At the moment, I have narrowed the problem down to a handful of
> instructions writing 0s into a well-formed region of the stack.
> Clearly, this is not correct, and every tweak of the debugging causes
> the problem to jump around.
>
> ~Andrew
After some more investigation, this is not a regression at all, although
the patch is directly relevant to identifying the problem.
PXELINUX 4.04 2011-04-18 Copyright (C) 1994-2011 H. Peter Anvin et al
boot:
Loading xenrt/xen-minnow.gz... ok
Loading xenrt/vmlinuz... ok
After multiboot magic check
Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
Before lret into trampoline
Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
After (failed) conditional jmp to start_secondary
Opcode from 0xffff830000105fef: 97 0e 86 00 49 8d be b0
__ __ _ _ _____ _
\ \/ /___ _ __ | || | |___ / / |
\ // _ \ '_ \ | || |_ |_ \ | |
Something between entering the trampoline and emerging in 64bit mode is
corrupting a single byte at phys 0x105ff1 from its correct value to a
value of 0x86.
The corruption disappears if the "no-real-mode" is used.
Currently the BIOS is trying to be updated, but the intersection of
operating systems which will successfully boot, and will successfully
run the IBM update tool is rather low.
~Andrew
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-12-02 14:01 ` Andrew Cooper
@ 2013-12-02 14:36 ` Jan Beulich
2013-12-03 19:53 ` Andrew Cooper
0 siblings, 1 reply; 12+ messages in thread
From: Jan Beulich @ 2013-12-02 14:36 UTC (permalink / raw)
To: Andrew Cooper; +Cc: George Dunlap, Dario Faggioli, Ian Campbell, Xen-devel List
>>> On 02.12.13 at 15:01, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> After some more investigation, this is not a regression at all, although
> the patch is directly relevant to identifying the problem.
>
> PXELINUX 4.04 2011-04-18 Copyright (C) 1994-2011 H. Peter Anvin et al
> boot:
> Loading xenrt/xen-minnow.gz... ok
> Loading xenrt/vmlinuz... ok
> After multiboot magic check
> Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
> Before lret into trampoline
> Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
> After (failed) conditional jmp to start_secondary
> Opcode from 0xffff830000105fef: 97 0e 86 00 49 8d be b0
> __ __ _ _ _____ _
> \ \/ /___ _ __ | || | |___ / / |
> \ // _ \ '_ \ | || |_ |_ \ | |
>
>
> Something between entering the trampoline and emerging in 64bit mode is
> corrupting a single byte at phys 0x105ff1 from its correct value to a
> value of 0x86.
>
> The corruption disappears if the "no-real-mode" is used.
And I'd say the primary suspect is
/*
* Declare that our target operating mode is long mode.
* Initialise 32-bit registers since some buggy BIOSes depend on it.
*/
movl $0xec00,%eax # declare target operating mode
movl $0x0002,%ebx # long mode
int $0x15
considering that 0x86 is a relatively common "function not
implemented" indicator for BIOS, namely INT 15, functions.
As a possible workaround I'd consider trying
a) zeroing %esp rather than just %sp a few lines up from the
above quoted code
b) zeroing the high halves of all registers
Jan
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode"
2013-12-02 14:36 ` Jan Beulich
@ 2013-12-03 19:53 ` Andrew Cooper
0 siblings, 0 replies; 12+ messages in thread
From: Andrew Cooper @ 2013-12-03 19:53 UTC (permalink / raw)
To: Jan Beulich; +Cc: George Dunlap, Dario Faggioli, Ian Campbell, Xen-devel List
On 02/12/13 14:36, Jan Beulich wrote:
>>>> On 02.12.13 at 15:01, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> After some more investigation, this is not a regression at all, although
>> the patch is directly relevant to identifying the problem.
>>
>> PXELINUX 4.04 2011-04-18 Copyright (C) 1994-2011 H. Peter Anvin et al
>> boot:
>> Loading xenrt/xen-minnow.gz... ok
>> Loading xenrt/vmlinuz... ok
>> After multiboot magic check
>> Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
>> Before lret into trampoline
>> Opcode from 0x105fef: 97 0e 00 00 49 8d be b0
>> After (failed) conditional jmp to start_secondary
>> Opcode from 0xffff830000105fef: 97 0e 86 00 49 8d be b0
>> __ __ _ _ _____ _
>> \ \/ /___ _ __ | || | |___ / / |
>> \ // _ \ '_ \ | || |_ |_ \ | |
>>
>>
>> Something between entering the trampoline and emerging in 64bit mode is
>> corrupting a single byte at phys 0x105ff1 from its correct value to a
>> value of 0x86.
>>
>> The corruption disappears if the "no-real-mode" is used.
> And I'd say the primary suspect is
>
> /*
> * Declare that our target operating mode is long mode.
> * Initialise 32-bit registers since some buggy BIOSes depend on it.
> */
> movl $0xec00,%eax # declare target operating mode
> movl $0x0002,%ebx # long mode
> int $0x15
>
> considering that 0x86 is a relatively common "function not
> implemented" indicator for BIOS, namely INT 15, functions.
>
> As a possible workaround I'd consider trying
> a) zeroing %esp rather than just %sp a few lines up from the
> above quoted code
> b) zeroing the high halves of all registers
>
> Jan
>
Your suspicion would be entirely correct. I have positively identified
this `int $0x15` call as corrupting the memory. The byte is fine
immediately before and bad immediately afterwards.
I have further confirmed that zeroing all 32bits of the GPRs before
entering the interrupt fixes the issue.
In an attempt to understand what is going on, I stuck in more debugging
for the entire register/selector state before and after, to see whether
anything looked like a smoking gun.
(XEN) Pre-state:
(XEN) eax 00007600 ebx 00000000 ecx 00000000 edx 00007600
(XEN) esi 0028b0c4 edi 00078a80 esp 00080000 ebp 00000000
(XEN) cs 7600 ds 7600 es 7600 fs 0028 gs 0028 ss 7600
If the GPRs are left as are the post state looks like:
(XEN) Post-state:
(XEN) eax 00008600 ebx 00000000 ecx 00000000 edx 00007600
(XEN) esi 0028b0c4 edi 00078a70 esp 00080000 ebp 00000000
(XEN) cs 7600 ds 7600 es 7600 fs 0028 gs 0028 ss 7600
If the GPRs are zeroed as much as possible, the post state looks like:
(XEN) Post-state:
(XEN) eax 00008600 ebx 00000000 ecx 00000000 edx 00000000
(XEN) esi 00000000 edi 00000000 esp 00000000 ebp 00000000
(XEN) cs 7600 ds 7600 es 7600 fs 0028 gs 0028 ss 7600
In both cases, the carry flag is set, which is consistent with the
return value of 0x86 is %ah.
I iterated through the registers, and proved that it was esp
specifically which was the problem.
I shall submit a patch against trampoline.S shortly.
~Andrew
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-12-03 19:53 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-11-28 12:31 Xen-4.3 and -unstable regression from changeset "numa-sched: leave node-affinity alone if not in 'auto' mode" Andrew Cooper
2013-11-28 13:05 ` Dario Faggioli
2013-11-28 15:09 ` George Dunlap
2013-11-28 15:14 ` Dario Faggioli
2013-11-28 15:16 ` Andrew Cooper
2013-11-28 21:17 ` Andrew Cooper
2013-11-28 23:30 ` George Dunlap
2013-11-29 10:51 ` Ian Campbell
2013-11-29 11:04 ` Andrew Cooper
2013-12-02 14:01 ` Andrew Cooper
2013-12-02 14:36 ` Jan Beulich
2013-12-03 19:53 ` Andrew Cooper
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).