From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joanna Rutkowska Subject: Strange kernel BUG() on PV DomU boot Date: Fri, 22 Jun 2012 14:21:58 +0200 Message-ID: <4FE46366.8010104@invisiblethingslab.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0827160727928925549==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "xen-devel@lists.xensource.com" Cc: Marek Marczykowski List-Id: xen-devel@lists.xenproject.org This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --===============0827160727928925549== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig1AD961FB7F90801FC802C034" This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1AD961FB7F90801FC802C034 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello, =46rom time to time (every several weeks or even less) I run into a strange Dom0 kernel BUG() that manifests itself with the following message (see the end of the message). The Dom0 and VM kernels are 3.2.7 pvops, and the Xen hypervisor is 4.1.2 both with only some minor, irrelevant (I think) modifications for Qubes. The bug is very hard to reproduce, but once this BUG() starts being signaled, it consistently prevents me from starting any new VMs in the system (e.g. tried over a dozen of times now, and every time the VM boot fails). The following lines in the VM kernel are responsible for signaling the BUG(): if (HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, ctxt)) BUG(); =2E..yet, there is nothing in the xl dmesg that would provide more info why this hypercall fails. Ah, that's because there are not printk's in the hypercall code: case VCPUOP_initialise: if ( v->vcpu_info =3D=3D &dummy_vcpu_info ) return -EINVAL; if ( (ctxt =3D xmalloc(struct vcpu_guest_context)) =3D=3D NULL ) return -ENOMEM; if ( copy_from_guest(ctxt, arg, 1) ) { xfree(ctxt); return -EFAULT; } domain_lock(d); rc =3D -EEXIST; if ( !v->is_initialised ) rc =3D boot_vcpu(d, vcpuid, ctxt); domain_unlock(d); xfree(ctxt); break; So, looking at the above it seems like it might be failing because of xmalloc() fails, however Xen seems to have enough memory as reported by xl info: total_memory : 8074 free_memory : 66 free_cpus : 0 Any ideas what might be the cause? FWIW, below the actual oops message. Thanks, joanna. [ 0.004356] ------------[ cut here ]------------ [ 0.004361] kernel BUG at /home/user/qubes-src/kernel/kernel-3.2.7/linux-3.2.7/arch/x86/xen/smp.c:3= 22! [ 0.004366] invalid opcode: 0000 [#1] SMP [ 0.004370] CPU 0 [ 0.004372] Modules linked in: [ 0.004376] [ 0.004379] Pid: 1, comm: swapper/0 Not tainted 3.2.7-5.pvops.qubes.x86_64 #1 [ 0.004385] RIP: e030:[] [] cpu_initialize_context+0x263/0x280 [ 0.004396] RSP: e02b:ffff880018063e10 EFLAGS: 00010282 [ 0.004399] RAX: fffffffffffffff4 RBX: ffff8800180c0000 RCX: 0000000000000000 [ 0.004404] RDX: ffff8800180c0000 RSI: 0000000000000001 RDI: 0000000000000000 [ 0.004408] RBP: ffff880018063e50 R08: 00003ffffffff000 R09: ffff880000000000 [ 0.004412] R10: ffff8800180c0000 R11: 0000000000002000 R12: 0000000000000001 [ 0.004417] R13: ffff880018f82d30 R14: ffff88001806e0c0 R15: 00000000000a98ed [ 0.004429] FS: 0000000000000000(0000) GS:ffff880018f5c000(0000) knlGS:0000000000000000 [ 0.004436] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 0.004441] CR2: 0000000000000000 CR3: 0000000001805000 CR4: 0000000000002660 [ 0.004447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 0.004452] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 0.004459] Process swapper/0 (pid: 1, threadinfo ffff880018062000, task ffff880018060040) [ 0.004465] Stack: [ 0.004469] ffff88001806e0c0 0000000000018f7b ffffffff81866c80 0000000000000001 [ 0.004479] ffff88001806e0c0 0000000000000001 ffffffff81866c80 0000000000000001 [ 0.004490] ffff880018063e80 ffffffff8143a2e1 ffff880018063e70 0000000000000000 [ 0.004500] Call Trace: [ 0.004507] [] xen_cpu_up+0x9b/0x115 [ 0.004513] [] _cpu_up+0x9c/0x10e [ 0.004520] [] cpu_up+0x75/0x85 [ 0.004527] [] smp_init+0x46/0x9e [ 0.004533] [] kernel_init+0x89/0x142 [ 0.004541] [] kernel_thread_helper+0x4/0x10 [ 0.004549] [] ? int_ret_from_sys_call+0x7/0x1b [ 0.004558] [] ? retint_restore_args+0x5/0x6 [ 0.004565] [] ? gs_change+0x13/0x13 [ 0.004570] Code: 74 0d 48 ba ff ff ff ff ff ff ff 3f 48 21 d0 48 c1 e0 0c 31 ff 49 63 f4 48 89 83 90 13 00 00 48 89 da e8 db 70 bc ff 85 c0 74 04 <0f> 0b eb fe 48 89 df e8 db f6 ce ff 31 c0 48 83 c4 18 5b 41 5c [ 0.004653] RIP [] cpu_initialize_context+0x263/0x2= 80 [ 0.004661] RSP [ 0.004672] ---[ end trace 4eaa2a86a8e2da22 ]--- [ 0.004686] Kernel panic - not syncing: Attempted to kill init! [ 0.004692] Pid: 1, comm: swapper/0 Tainted: G D 3.2.7-5.pvops.qubes.x86_64 #1 [ 0.004698] Call Trace: [ 0.004704] [] panic+0x8c/0x1a2 [ 0.004712] [] ? enqueue_entity+0x74/0x2f0 [ 0.004719] [] forget_original_parent+0x34d/0x360 [ 0.004728] [] ? xen_restore_fl_direct_reloc+0x4/0x= 4 [ 0.004735] [] ? _raw_spin_unlock_irqrestore+0x11/0= x20 [ 0.004743] [] ? sched_move_task+0x93/0x150 [ 0.004750] [] exit_notify+0x12/0x190 [ 0.004756] [] do_exit+0x1ed/0x3e0 [ 0.004763] [] oops_end+0xa6/0xf0 [ 0.004770] [] die+0x56/0x90 [ 0.004776] [] do_trap+0xc4/0x170 [ 0.004783] [] do_invalid_op+0x90/0xb0 [ 0.004790] [] ? cpu_initialize_context+0x263/0x280= [ 0.004799] [] ? cache_grow.clone.0+0x2b4/0x3b0 [ 0.004805] [] ? xen_restore_fl_direct_reloc+0x4/0x= 4 [ 0.004812] [] ? pte_mfn_to_pfn+0x71/0xf0 [ 0.004820] [] invalid_op+0x1b/0x20 [ 0.004827] [] ? cpu_initialize_context+0x263/0x280= [ 0.004834] [] xen_cpu_up+0x9b/0x115 [ 0.004840] [] _cpu_up+0x9c/0x10e [ 0.004846] [] cpu_up+0x75/0x85 [ 0.004852] [] smp_init+0x46/0x9e [ 0.004858] [] kernel_init+0x89/0x142 [ 0.004864] [] kernel_thread_helper+0x4/0x10 [ 0.004871] [] ? int_ret_from_sys_call+0x7/0x1b [ 0.004878] [] ? retint_restore_args+0x5/0x6 [ 0.004885] [] ? gs_change+0x13/0x13 --------------enig1AD961FB7F90801FC802C034 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJP5GNmAAoJEDaIqHeRBUM01a8IAMEIVlHl7pQrhcEnFX4dEvqR my/Vx6GrmyFHa8JrvkbrmF7Vb4LizRQu5MV9MIYxW40Cl8iyBAryJ/QlZDH+Rq+M x6yoDiYvbjgsJftDVzQ2S9EPD988ms0wCH+7SxlFHgbi94m6VF3KUUT47vTP23UR +rk8wgGnTH14zqxDPmuw6VkRhzm8i2/6zyUmY//puPJif0+oIdIEmKp+LWUOxsU9 TKjlr5njhYegDG+pUzaR7a2jWm5IAdS2Rvex5pLFaJXC9Iu9XJDhWGmHxPkHTsiZ CKe7R/Wsgr8Dw4oGYA6ytVd69SZ73ZL0s1DyLw8JY7qSbAGOMxZUK+jxKrdhRoQ= =VByX -----END PGP SIGNATURE----- --------------enig1AD961FB7F90801FC802C034-- --===============0827160727928925549== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============0827160727928925549==--