From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Xen question Date: Wed, 02 Feb 2011 11:52:17 -0800 Message-ID: <4D49B5F1.2080208@goop.org> References: <20110201104929.390e8511@nehalam> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20110201104929.390e8511@nehalam> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Stephen Hemminger Cc: Chris Wright , "Xen-devel@lists.xensource.com" , Jeremy Fitzhardinge List-Id: xen-devel@lists.xenproject.org On 02/01/2011 10:49 AM, Stephen Hemminger wrote: > Something is wrong with the Vyatta Xen kernel. > It worked with 2.6.32, but something wrong with 2.6.35. > Have you seen this? > > -- > [ 0.000000] Reserving virtual address space above 0xf5800000 > [ 0.000000] Initializing cgroup subsys cpuset > [ 0.000000] Initializing cgroup subsys cpu > [ 0.000000] Linux version 2.6.35-1-586-vyatta-virt (root@vyatta) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Jan 28 16:40:27 PST 2011 > [ 0.000000] ACPI in unprivileged domain disabled > [ 0.000000] BIOS-provided physical RAM map: > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] Xen: 0000000000100000 - 0000000020000000 (usable) > [ 0.000000] NX (Execute Disable) protection: active > [ 0.000000] DMI not present or invalid. > [ 0.000000] last_pfn = 0x20000 max_arch_pfn = 0x1000000 > [ 0.000000] init_memory_mapping: 0000000000000000-0000000020000000 > [ 0.000000] RAMDISK: 01607000 - 03148000 > [ 0.000000] Node: 0, start_pfn: 0, end_pfn: 20000 > [ 0.000000] Reserving total of 600 pages for numa KVA remap > [ 0.000000] kva_start_pfn ~ 1f400 max_low_pfn ~ 20000 > [ 0.000000] max_pfn = 20000 > [ 0.000000] 0MB HIGHMEM available. > [ 0.000000] 512MB LOWMEM available. > [ 0.000000] 1 multicall(s) failed: cpu 0 > [ 0.000000] Pid: 0, comm: swapper Not tainted 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? xen_mc_flush+0x92/0x13d > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 This is failing during the initial mapping of memory, I think. Are there any messages on the Xen console? Is it completely deterministic? > [ 0.000000] ------------[ cut here ]------------ > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_issue+0x11/0x1f() > [ 0.000000] Modules linked in: > [ 0.000000] Pid: 0, comm: swapper Not tainted 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? warn_slowpath_common+0x6a/0x7b > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? warn_slowpath_null+0xd/0x10 > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 > [ 0.000000] ---[ end trace 4eaa2a86a8e2da22 ]--- > [ 0.000000] 1 multicall(s) failed: cpu 0 > [ 0.000000] Pid: 0, comm: swapper Tainted: G W 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? xen_mc_flush+0x92/0x13d > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 > [ 0.000000] ------------[ cut here ]------------ > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_issue+0x11/0x1f() > [ 0.000000] Modules linked in: > [ 0.000000] Pid: 0, comm: swapper Tainted: G W 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? warn_slowpath_common+0x6a/0x7b > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? warn_slowpath_null+0xd/0x10 > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 > [ 0.000000] ---[ end trace 4eaa2a86a8e2da23 ]--- > [ 0.000000] 1 multicall(s) failed: cpu 0 > [ 0.000000] Pid: 0, comm: swapper Tainted: G W 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? xen_mc_flush+0x92/0x13d > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 > [ 0.000000] ------------[ cut here ]------------ > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_issue+0x11/0x1f() > [ 0.000000] Modules linked in: > [ 0.000000] Pid: 0, comm: swapper Tainted: G W 2.6.35-1-586-vyatta-virt #1 > [ 0.000000] Call Trace: > [ 0.000000] [] ? warn_slowpath_common+0x6a/0x7b > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? warn_slowpath_null+0xd/0x10 > [ 0.000000] [] ? xen_mc_issue+0x11/0x1f > [ 0.000000] [] ? xen_set_pmd_hyper+0x55/0x5c > [ 0.000000] [] ? set_pmd_pfn+0xb4/0xc3 > [ 0.000000] [] ? initmem_init+0x4e9/0x5ea > [ 0.000000] [] ? printk+0xe/0x14 > [ 0.000000] [] ? setup_arch+0x8d1/0x989 > [ 0.000000] [] ? proc_caches_init+0x27/0xbf > [ 0.000000] [] ? start_kernel+0x86/0x330 > [ 0.000000] [] ? xen_start_kernel+0x4c0/0x4c7 > [ 0.000000] ---[ end trace 4eaa2a86a8e2da24 ]--- > [ 0.000000] mapped low ram: 0 - 20000000 > [ 0.000000] low ram: 0 - 20000000 > [ 0.000000] Zone PFN ranges: > [ 0.000000] DMA 0x00000001 -> 0x00001000 > [ 0.000000] Normal 0x00001000 -> 0x00020000 > [ 0.000000] HighMem empty > [ 0.000000] Movable zone start PFN for each node > [ 0.000000] early_node_map[2] active PFN ranges > [ 0.000000] 0: 0x00000001 -> 0x000000a0 > [ 0.000000] 0: 0x00000100 -> 0x0001fa00 > [ 0.000000] Using APIC driver default > [ 0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org > [ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs > [ 0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic" > [ 0.000000] APIC: disable apic facility > [ 0.000000] APIC: switched to apic NOOP > [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000 > [ 0.000000] Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000) > [ 0.000000] Booting paravirtualized kernel on Xen > [ 0.000000] Xen version: 4.0.1 (preserve-AD) > [ 0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:8 > [ 0.000000] PERCPU: Embedded 15 pages/cpu @c14e7000 s37504 r0 d23936 u65536 > [ 0.000000] pcpu-alloc: s37504 r0 d23936 u65536 alloc=16*4096 > [ 0.000000] pcpu-alloc: [0] 0 > [ 0.000000] Xen: using vcpu_info placement > [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 128427 > [ 0.000000] Policy zone: Normal > [ 0.000000] Kernel command line: root=/dev/xvda1 ro ip=:127.0.255.255::::eth0:dhcp > [ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes) > [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) > [ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) > [ 0.000000] Enabling fast FPU save and restore... done. > [ 0.000000] Enabling unmasked SIMD FPU exception support... done. > [ 0.000000] Initializing CPU#0 > [ 0.000000] Subtract (33 early reservations) > [ 0.000000] #1 [00031cb000 - 00031e7000] XEN PAGETABLES > [ 0.000000] #2 [0000001000 - 0000002000] EX TRAMPOLINE > [ 0.000000] #3 [0001000000 - 00014e15d0] TEXT DATA BSS > [ 0.000000] #4 [0001607000 - 0003148000] RAMDISK > [ 0.000000] #5 [0003148000 - 00031cb000] XEN START INFO > [ 0.000000] #6 [0000002000 - 0000003000] TRAMPOLINE > [ 0.000000] #7 [0000003000 - 0000007000] ACPI WAKEUP > [ 0.000000] #8 [0000100000 - 00001e4000] PGTABLE > [ 0.000000] #9 [001fa00000 - 0020000000] KVA RAM > [ 0.000000] #10 [001f400000 - 001fa00000] KVA PG > [ 0.000000] #11 [00014e2000 - 00014e3000] BOOTMEM > [ 0.000000] #12 [00014e1600 - 00014e1604] BOOTMEM > [ 0.000000] #13 [00014e1640 - 00014e1700] BOOTMEM > [ 0.000000] #14 [00014e1700 - 00014e1730] BOOTMEM > [ 0.000000] #15 [00014e3000 - 00014e4800] BOOTMEM > [ 0.000000] #16 [00014e1740 - 00014e17d0] BOOTMEM > [ 0.000000] #17 [00014e1800 - 00014e1840] BOOTMEM > [ 0.000000] #18 [00014e1840 - 00014e1880] BOOTMEM > [ 0.000000] #19 [00014e1880 - 00014e18c0] BOOTMEM > [ 0.000000] #20 [00014e18c0 - 00014e18d0] BOOTMEM > [ 0.000000] #21 [00014e1900 - 00014e1933] BOOTMEM > [ 0.000000] #22 [00014e1940 - 00014e1973] BOOTMEM > [ 0.000000] #23 [00014e7000 - 00014f6000] BOOTMEM > [ 0.000000] #24 [00014e1980 - 00014e1984] BOOTMEM > [ 0.000000] #25 [00014e19c0 - 00014e19c4] BOOTMEM > [ 0.000000] #26 [00014e1a00 - 00014e1a04] BOOTMEM > [ 0.000000] #27 [00014e1a40 - 00014e1a44] BOOTMEM > [ 0.000000] #28 [00014e1a80 - 00014e1b00] BOOTMEM > [ 0.000000] #29 [00014e1b00 - 00014e1b2c] BOOTMEM > [ 0.000000] #30 [00014e4800 - 00014e6800] BOOTMEM > [ 0.000000] #31 [00014f6000 - 0001536000] BOOTMEM > [ 0.000000] #32 [0001536000 - 0001556000] BOOTMEM > [ 0.000000] Initializing HighMem for node 0 (00000000:00000000) > [ 0.000000] Memory: 476668k/524288k available (2608k kernel code, 41088k reserved, 1281k data, 512k init, 0k highmem) > [ 0.000000] virtual kernel memory layout: > [ 0.000000] fixmap : 0xf5536000 - 0xf57ff000 (2852 kB) > [ 0.000000] pkmap : 0xf5200000 - 0xf5400000 (2048 kB) > [ 0.000000] vmalloc : 0xe0800000 - 0xf51fe000 ( 329 MB) > [ 0.000000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB) > [ 0.000000] .init : 0xc13cd000 - 0xc144d000 ( 512 kB) > [ 0.000000] .data : 0xc128c1b2 - 0xc13cc918 (1281 kB) > [ 0.000000] .text : 0xc1000000 - 0xc128c1b2 (2608 kB) > [ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=8 > [ 0.000000] Hierarchical RCU implementation. > [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. > [ 0.000000] RCU-based detection of stalled CPUs is disabled. > [ 0.000000] Verbose stalled-CPUs detection is disabled. > [ 0.000000] NR_IRQS:2304 nr_irqs:256 > [ 0.000000] Console: colour dummy device 80x25 > [ 0.000000] console [tty0] enabled > [ 0.000000] console [hvc0] enabled > [ 0.000000] installing Xen timer for CPU 0 > [ 0.000000] Detected 1994.378 MHz processor. > [ 0.010000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3988.75 BogoMIPS (lpj=19943780) > [ 0.010000] pid_max: default: 32768 minimum: 301 > [ 0.010000] Security Framework initialized > [ 0.010000] SELinux: Disabled at boot. > [ 0.010000] Mount-cache hash table entries: 512 > [ 0.010000] Initializing cgroup subsys ns > [ 0.010000] Initializing cgroup subsys cpuacct > [ 0.010000] Initializing cgroup subsys devices > [ 0.010000] Initializing cgroup subsys freezer > [ 0.010000] Performance Events: > [ 0.010000] no APIC, boot with the "lapic" boot parameter to force-enable it. > [ 0.010000] no hardware sampling interrupt available. > [ 0.010000] AMD PMU driver. > [ 0.010000] ... version: 0 > [ 0.010000] ... bit width: 48 > [ 0.010000] ... generic registers: 4 > [ 0.010000] ... value mask: 0000ffffffffffff > [ 0.010000] ... max period: 00007fffffffffff > [ 0.010000] ... fixed-purpose events: 0 > [ 0.010000] ... event mask: 000000000000000f > [ 0.010000] SMP alternatives: switching to UP code > [ 0.010000] Freeing SMP alternatives: 16k freed > [ 0.010121] cpu 0 spinlock event irq 1 > [ 0.010429] Brought up 1 CPUs > [ 0.010467] BUG: unable to handle kernel NULL pointer dereference at (null) > [ 0.010482] IP: [] cpu_attach_domain+0x6e/0x178 > [ 0.010503] *pdpt = 0000000001454027 *pde = 0000000000000000 > [ 0.010517] Oops: 0000 [#1] SMP > [ 0.010528] last sysfs file: > [ 0.010536] Modules linked in: > [ 0.010546] > [ 0.010554] Pid: 1, comm: swapper Tainted: G W 2.6.35-1-586-vyatta-virt #1 / > [ 0.010566] EIP: 0061:[] EFLAGS: 00010206 CPU: 0 > [ 0.010576] EIP is at cpu_attach_domain+0x6e/0x178 > [ 0.010585] EAX: 0000042f EBX: c14eb868 ECX: 00000000 EDX: c14eb940 > [ 0.010595] ESI: c14efb80 EDI: 00000000 EBP: c14eb9ac ESP: df035f40 > [ 0.010605] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021 > [ 0.010614] Process swapper (pid: 1, ti=df034000 task=df030000 task.ti=df034000) > [ 0.010625] Stack: > [ 0.010631] c14eb940 0000102f 0000042f df044000 df006058 df006058 00000000 c1447868 > [ 0.010655] <0> 00000000 c102f634 00000027 c14eba08 00000000 00000020 c14479ac 00000001 > [ 0.010683] <0> c1398564 00000000 c1398564 00000000 00000001 00000001 00000001 00000000 > [ 0.010713] Call Trace: > [ 0.010725] [] ? __build_sched_domains+0x650/0x697 This might relate to BIGSMP. Maybe something going wrong with the cpu topology info? What local changes do you have? What kernel last worked for you? Thanks, J > [ 0.010742] [] ? sched_init_smp+0x81/0x11f > [ 0.010754] [] ? kernel_init+0xfd/0x1ba > [ 0.010764] [] ? kernel_init+0x0/0x1ba > [ 0.010776] [] ? kernel_thread_helper+0x6/0x10 > [ 0.010784] Code: 42 38 89 44 24 04 8b 45 38 89 14 24 89 44 24 08 8d 45 5c e8 94 a7 ff ff 8b 14 24 48 74 4c 8b 45 38 a9 8f 02 00 00 74 0b 8b 4d 08 <3b> 09 0f 85 ef 00 00 00 a8 20 0f 85 e7 00 00 00 eb 2d 8b 4d 08 > [ 0.010938] EIP: [] cpu_attach_domain+0x6e/0x178 SS:ESP e021:df035f40 > [ 0.010955] CR2: 0000000000000000 > [ 0.010965] ---[ end trace 4eaa2a86a8e2da25 ]--- > [ 0.010988] Kernel panic - not syncing: Attempted to kill init! > [ 0.020000] Pid: 1, comm: swapper Tainted: G D W 2.6.35-1-586-vyatta-virt #1 > [ 0.020000] Call Trace: > [ 0.020000] [] ? panic+0x4a/0xbb > [ 0.020000] [] ? do_exit+0x53/0x646 > [ 0.020000] [] ? oops_end+0x7d/0x80 > [ 0.020000] [] ? no_context+0x13b/0x144 > [ 0.020000] [] ? do_page_fault+0x0/0x24b > [ 0.020000] [] ? bad_area_nosemaphore+0xa/0xc > [ 0.020000] [] ? error_code+0x66/0x6c > [ 0.020000] [] ? do_page_fault+0x0/0x24b > [ 0.020000] [] ? cpu_attach_domain+0x6e/0x178 > [ 0.020000] [] ? __build_sched_domains+0x650/0x697 > [ 0.020000] [] ? sched_init_smp+0x81/0x11f > [ 0.020000] [] ? kernel_init+0xfd/0x1ba > [ 0.020000] [] ? kernel_init+0x0/0x1ba > [ 0.020000] [] ? kernel_thread_helper+0x6/0x10