Re: Linux x86 guest panics in skb_copy_bits

Justin Dossey
2009-05-06T23:35:25Z

On Tue, May 5, 2009 at 3:29 PM, Justin Dossey <jbd@justindossey.com> wrote:
> On Mon, May 4, 2009 at 3:40 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> Justin,
>>
>> On Sun, May 03, 2009 at 11:40:47AM -0700, Justin Dossey wrote:
> [snip]
>>
>> Seems to be an issue with paravirt mmu. Do you happen to have
>> CONFIG_DEBUG_PAGEALLOC turned on your guests?
>
> I don't, as my VMs are in production use.  To find the source of this
> issue, I can turn it on though.
>
> While I'm at it, are there any other kernel features I should enable?
>

I went ahead and recompiled with CONFIG_DEBUG_PAGEALLOC enabled.
Here's the panic (77 seconds after boot!)

[ 76.911884] BUG: unable to handle kernel paging request at f4d17000
[ 76.915076] IP: [<c02856a9>] __slab_alloc+0x217/0x42f
[ 76.917161] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[ 76.919309] last sysfs file: /sys/kernel/uevent_seqnum
[ 76.920015] Modules linked in:
[ 76.920015]
[ 76.920015] Pid: 4632, comm: ruby18 Not tainted (2.6.28-gentoo-r4 #2)
[ 76.920015] EIP: 0060:[<c02856a9>] EFLAGS: 00210086 CPU: 0
[ 76.920015] EIP is at __slab_alloc+0x217/0x42f
[ 76.920015] EAX: c0761564 EBX: c1abb740 ECX: c0761564 EDX: 00000000
[ 76.920015] ESI: f4d17800 EDI: f4d17000 EBP: f72e9b8c ESP: f72e9b6c
[ 76.920015] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 76.920015] Process ruby18 (pid: 4632, ti=f72e8000 task=f721d8e0
task.ti=f72e8000)
[ 76.920015] Stack:
[ 76.920015] ffffffff 00000020 c0761564 00000000 f4d10000 00000000
00200282 c0761564
[ 76.920015] f72e9bb4 c02867a5 c04b6782 c2a202a8 c04b6782 00000020
00000800 f70583c0
[ 76.920015] 00000600 f70583c0 f72e9bd4 c04b5948 00000000 00000020
c0760f58 00000020
[ 76.920015] Call Trace:
[ 76.920015] [<c02867a5>] ? __kmalloc_track_caller+0x89/0xda
[ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c04b5948>] ? __alloc_skb+0x4f/0xfb
[ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c0452583>] ? try_fill_recv+0x30/0x177
[ 76.920015] [<c04b37a1>] ? sock_def_readable+0x5e/0x63
[ 76.920015] [<c0453237>] ? virtnet_poll+0x25c/0x309
[ 76.920015] [<c04bc32e>] ? net_rx_action+0xbd/0x1ea
[ 76.920015] [<c022b2ed>] ? __do_softirq+0x83/0x12e
[ 76.920015] [<c022b3e0>] ? do_softirq+0x48/0x57
[ 76.920015] [<c022b6fa>] ? irq_exit+0x38/0x6d
[ 76.920015] [<c0205868>] ? do_IRQ+0x96/0xae
[ 76.920015] [<c020471b>] ? common_interrupt+0x23/0x28
[ 76.920015] [<c0274379>] ? copy_page_range+0x25c/0x51e
[ 76.920015] [<c0225731>] ? dup_mm+0x22a/0x30c
[ 76.920015] [<c0226141>] ? copy_process+0x906/0x1026
[ 76.920015] [<c02269a0>] ? do_fork+0xd6/0x21f
[ 76.920015] [<c03ae3e0>] ? copy_to_user+0x2a/0x36
[ 76.920015] [<c0202236>] ? sys_clone+0x25/0x2a
[ 76.920015] [<c0203bd2>] ? syscall_call+0x7/0xb
[ 76.920015] Code: c1 e9 02 f3 ab f6 c2 02 74 02 66 ab f6 c2 01 74
01 aa 8b 7d f0 89 fe eb 19 8b 45 e8 89 f9 89 da e8 4d ee ff ff 8b 4d
e8 03 79 0c <89> 37 89 f7 03 71 04 8b 55 e8 0f b7 43 0a 0f af 42 04 03
45f0
[ 76.920015] EIP: [<c02856a9>] __slab_alloc+0x217/0x42f SS:ESP 0068:f72e9b6c
[ 76.920015] Kernel panic - not syncing: Fatal exception in interrupt
[ 76.920015] ------------[ cut here ]------------
[ 76.920015] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x33/0x185()
[ 76.920015] Modules linked in:
[ 76.920015] Pid: 4632, comm: ruby18 Tainted: G D 2.6.28-gentoo-r4 #2
[ 76.920015] Call Trace:
[ 76.920015] [<c022735a>] warn_on_slowpath+0x41/0x5b
[ 76.920015] [<c03ada5e>] ? __delay+0x9/0xb
[ 76.920015] [<c0405af6>] ? serial8250_console_write+0xd6/0xea
[ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92
[ 76.920015] [<c0586f9c>] ? _spin_unlock+0x1d/0x20
[ 76.920015] [<c03b1457>] ? _raw_spin_lock+0x60/0xf5
[ 76.920015] [<c023af8e>] ? up+0x2b/0x2f
[ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92
[ 76.920015] [<c022797f>] ? release_console_sem+0x185/0x198
[ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92
[ 76.920015] [<c022798a>] ? release_console_sem+0x190/0x198
[ 76.920015] [<c0246630>] smp_call_function_mask+0x33/0x185
[ 76.920015] [<c02100e4>] ? stop_this_cpu+0x0/0x50
[ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa
[ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa
[ 76.920015] [<c024f4cd>] ? crash_kexec+0x9f/0xa7
[ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa
[ 76.920015] [<c024f4cd>] ? crash_kexec+0x9f/0xa7
[ 76.920015] [<c0246794>] smp_call_function+0x12/0x14
[ 76.920015] [<c0210183>] native_smp_send_stop+0x1b/0x45
[ 76.920015] [<c02273d0>] panic+0x4e/0xef
[ 76.920015] [<c0587e2e>] oops_end+0x8f/0xa3
[ 76.920015] [<c0205c87>] die+0x57/0x5f
[ 76.920015] [<c0589477>] do_page_fault+0x63e/0x6fa
[ 76.920015] [<c0268c84>] ? __alloc_pages_internal+0xad/0x33a
[ 76.920015] [<c0588e39>] ? do_page_fault+0x0/0x6fa
[ 76.920015] [<c05874fa>] error_code+0x72/0x78
[ 76.920015] [<c02856a9>] ? __slab_alloc+0x217/0x42f
[ 76.920015] [<c02867a5>] __kmalloc_track_caller+0x89/0xda
[ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c04b5948>] __alloc_skb+0x4f/0xfb
[ 76.920015] [<c04b6782>] __netdev_alloc_skb+0x17/0x34
[ 76.920015] [<c0452583>] try_fill_recv+0x30/0x177
[ 76.920015] [<c04b37a1>] ? sock_def_readable+0x5e/0x63
[ 76.920015] [<c0453237>] virtnet_poll+0x25c/0x309
[ 76.920015] [<c04bc32e>] net_rx_action+0xbd/0x1ea
[ 76.920015] [<c022b2ed>] __do_softirq+0x83/0x12e
[ 76.920015] [<c022b3e0>] do_softirq+0x48/0x57
[ 76.920015] [<c022b6fa>] irq_exit+0x38/0x6d
[ 76.920015] [<c0205868>] do_IRQ+0x96/0xae
[ 76.920015] [<c020471b>] common_interrupt+0x23/0x28
[ 76.920015] [<c0274379>] ? copy_page_range+0x25c/0x51e
[ 76.920015] [<c0225731>] dup_mm+0x22a/0x30c
[ 76.920015] [<c0226141>] copy_process+0x906/0x1026
[ 76.920015] [<c02269a0>] do_fork+0xd6/0x21f
[ 76.920015] [<c03ae3e0>] ? copy_to_user+0x2a/0x36
[ 76.920015] [<c0202236>] sys_clone+0x25/0x2a
[ 76.920015] [<c0203bd2>] syscall_call+0x7/0xb
[ 76.920015] ---[ end trace 4656f6f379527461 ]---