From mboxrd@z Thu Jan 1 00:00:00 1970 From: Justin Dossey Subject: Re: Linux x86 guest panics in skb_copy_bits Date: Wed, 6 May 2009 16:35:24 -0700 Message-ID: References: <20090504224006.GB10616@amt.cnet> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE To: Marcelo Tosatti , kvm@vger.kernel.org Return-path: Received: from yw-out-2324.google.com ([74.125.46.29]:16624 "EHLO yw-out-2324.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751384AbZEFXfZ convert rfc822-to-8bit (ORCPT ); Wed, 6 May 2009 19:35:25 -0400 Received: by yw-out-2324.google.com with SMTP id 5so253287ywb.1 for ; Wed, 06 May 2009 16:35:25 -0700 (PDT) In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Tue, May 5, 2009 at 3:29 PM, Justin Dossey wr= ote: > On Mon, May 4, 2009 at 3:40 PM, Marcelo Tosatti = wrote: >> Justin, >> >> On Sun, May 03, 2009 at 11:40:47AM -0700, Justin Dossey wrote: > [snip] >> >> Seems to be an issue with paravirt mmu. Do you happen to have >> CONFIG_DEBUG_PAGEALLOC turned on your guests? > > I don't, as my VMs are in production use. =C2=A0To find the source of= this > issue, I can turn it on though. > > While I'm at it, are there any other kernel features I should enable? > I went ahead and recompiled with CONFIG_DEBUG_PAGEALLOC enabled. Here's the panic (77 seconds after boot!) [ 76.911884] BUG: unable to handle kernel paging request at f4d17000 [ 76.915076] IP: [] __slab_alloc+0x217/0x42f [ 76.917161] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 76.919309] last sysfs file: /sys/kernel/uevent_seqnum [ 76.920015] Modules linked in: [ 76.920015] [ 76.920015] Pid: 4632, comm: ruby18 Not tainted (2.6.28-gentoo-r4 #2= ) [ 76.920015] EIP: 0060:[] EFLAGS: 00210086 CPU: 0 [ 76.920015] EIP is at __slab_alloc+0x217/0x42f [ 76.920015] EAX: c0761564 EBX: c1abb740 ECX: c0761564 EDX: 00000000 [ 76.920015] ESI: f4d17800 EDI: f4d17000 EBP: f72e9b8c ESP: f72e9b6c [ 76.920015] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 76.920015] Process ruby18 (pid: 4632, ti=3Df72e8000 task=3Df721d8e0 task.ti=3Df72e8000) [ 76.920015] Stack: [ 76.920015] ffffffff 00000020 c0761564 00000000 f4d10000 00000000 00200282 c0761564 [ 76.920015] f72e9bb4 c02867a5 c04b6782 c2a202a8 c04b6782 00000020 00000800 f70583c0 [ 76.920015] 00000600 f70583c0 f72e9bd4 c04b5948 00000000 00000020 c0760f58 00000020 [ 76.920015] Call Trace: [ 76.920015] [] ? __kmalloc_track_caller+0x89/0xda [ 76.920015] [] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] ? __alloc_skb+0x4f/0xfb [ 76.920015] [] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] ? try_fill_recv+0x30/0x177 [ 76.920015] [] ? sock_def_readable+0x5e/0x63 [ 76.920015] [] ? virtnet_poll+0x25c/0x309 [ 76.920015] [] ? net_rx_action+0xbd/0x1ea [ 76.920015] [] ? __do_softirq+0x83/0x12e [ 76.920015] [] ? do_softirq+0x48/0x57 [ 76.920015] [] ? irq_exit+0x38/0x6d [ 76.920015] [] ? do_IRQ+0x96/0xae [ 76.920015] [] ? common_interrupt+0x23/0x28 [ 76.920015] [] ? copy_page_range+0x25c/0x51e [ 76.920015] [] ? dup_mm+0x22a/0x30c [ 76.920015] [] ? copy_process+0x906/0x1026 [ 76.920015] [] ? do_fork+0xd6/0x21f [ 76.920015] [] ? copy_to_user+0x2a/0x36 [ 76.920015] [] ? sys_clone+0x25/0x2a [ 76.920015] [] ? syscall_call+0x7/0xb [ 76.920015] Code: c1 e9 02 f3 ab f6 c2 02 74 02 66 ab f6 c2 01 74 01 aa 8b 7d f0 89 fe eb 19 8b 45 e8 89 f9 89 da e8 4d ee ff ff 8b 4d e8 03 79 0c <89> 37 89 f7 03 71 04 8b 55 e8 0f b7 43 0a 0f af 42 04 03 45f0 [ 76.920015] EIP: [] __slab_alloc+0x217/0x42f SS:ESP 0068:f= 72e9b6c [ 76.920015] Kernel panic - not syncing: Fatal exception in interrupt [ 76.920015] ------------[ cut here ]------------ [ 76.920015] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x33= /0x185() [ 76.920015] Modules linked in: [ 76.920015] Pid: 4632, comm: ruby18 Tainted: G D 2.6.28-gent= oo-r4 #2 [ 76.920015] Call Trace: [ 76.920015] [] warn_on_slowpath+0x41/0x5b [ 76.920015] [] ? __delay+0x9/0xb [ 76.920015] [] ? serial8250_console_write+0xd6/0xea [ 76.920015] [] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [] ? _spin_unlock+0x1d/0x20 [ 76.920015] [] ? _raw_spin_lock+0x60/0xf5 [ 76.920015] [] ? up+0x2b/0x2f [ 76.920015] [] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [] ? release_console_sem+0x185/0x198 [ 76.920015] [] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [] ? release_console_sem+0x190/0x198 [ 76.920015] [] smp_call_function_mask+0x33/0x185 [ 76.920015] [] ? stop_this_cpu+0x0/0x50 [ 76.920015] [] ? mutex_unlock+0x8/0xa [ 76.920015] [] ? mutex_unlock+0x8/0xa [ 76.920015] [] ? crash_kexec+0x9f/0xa7 [ 76.920015] [] ? mutex_unlock+0x8/0xa [ 76.920015] [] ? crash_kexec+0x9f/0xa7 [ 76.920015] [] smp_call_function+0x12/0x14 [ 76.920015] [] native_smp_send_stop+0x1b/0x45 [ 76.920015] [] panic+0x4e/0xef [ 76.920015] [] oops_end+0x8f/0xa3 [ 76.920015] [] die+0x57/0x5f [ 76.920015] [] do_page_fault+0x63e/0x6fa [ 76.920015] [] ? __alloc_pages_internal+0xad/0x33a [ 76.920015] [] ? do_page_fault+0x0/0x6fa [ 76.920015] [] error_code+0x72/0x78 [ 76.920015] [] ? __slab_alloc+0x217/0x42f [ 76.920015] [] __kmalloc_track_caller+0x89/0xda [ 76.920015] [] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] __alloc_skb+0x4f/0xfb [ 76.920015] [] __netdev_alloc_skb+0x17/0x34 [ 76.920015] [] try_fill_recv+0x30/0x177 [ 76.920015] [] ? sock_def_readable+0x5e/0x63 [ 76.920015] [] virtnet_poll+0x25c/0x309 [ 76.920015] [] net_rx_action+0xbd/0x1ea [ 76.920015] [] __do_softirq+0x83/0x12e [ 76.920015] [] do_softirq+0x48/0x57 [ 76.920015] [] irq_exit+0x38/0x6d [ 76.920015] [] do_IRQ+0x96/0xae [ 76.920015] [] common_interrupt+0x23/0x28 [ 76.920015] [] ? copy_page_range+0x25c/0x51e [ 76.920015] [] dup_mm+0x22a/0x30c [ 76.920015] [] copy_process+0x906/0x1026 [ 76.920015] [] do_fork+0xd6/0x21f [ 76.920015] [] ? copy_to_user+0x2a/0x36 [ 76.920015] [] sys_clone+0x25/0x2a [ 76.920015] [] syscall_call+0x7/0xb [ 76.920015] ---[ end trace 4656f6f379527461 ]--- --=20 Justin Dossey