Re: kernel BUG at mm/page_alloc.c:424!

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 08/05/09 06:50, Nathan Stratton wrote:
> On Wed, 5 Aug 2009, Konrad Rzeszutek Wilk wrote:
>
>> On Tue, Aug 04, 2009 at 07:38:10PM -0500, Nathan Stratton wrote:
>>>
>>> [root@cbridge1 etc]# ------------[ cut here ]------------
>>> kernel BUG at mm/page_alloc.c:424!
>>> invalid opcode: 0000 [#1] SMP
>>> last sysfs file: /sys/devices/vif-0/net/eth0/tx_queue_len
>>> CPU 2
>>> Modules linked in: ipv6 joydev pcspkr xen_netfront xen_blkfront [last
>>> unloaded: scsi_wait_scan]
>>> Pid: 1182, comm: texgrid Tainted: G        W 
>>> 2.6.29.4-167.fc11.x86_64 #1
>>> RIP: e030:[<ffffffff810a2eb1>]  [<ffffffff810a2eb1>]
>>> __free_one_page+0x146/0x1e6
>>> RSP: e02b:ffff88007ddc3a38  EFLAGS: 00010002
>>> RAX: ffffe2000194d290 RBX: ffffe2000194d2c8 RCX: ffffe2000194d290
>>> RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff880000008880
>>> RBP: ffff88007ddc3a88 R08: 000000000000001e R09: 0000000000000006
>>> R10: 0000000000000000 R11: 0000000000009f20 R12: 0000000000000000
>>> R13: ffff880000008880 R14: 000000000000029f R15: 0000000000000002
>>> FS:  00007f8b5cbb8910(0000) GS:ffff88007e84c700(0000)
>>> knlGS:0000000000000000
>>> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> CR2: 00007f8b50bb3da8 CR3: 000000007b19d000 CR4: 0000000000002660
>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> Process texgrid (pid: 1182, threadinfo ffff88007ddc2000, task
>>> ffff88007db41700)
>>> Stack:
>>>  ffffe2000194d638 0000000000000004 ffffe20001948868 0000000100000018
>>>  ffff88007e84c590 ffff880000008880 0000000000000002 ffff88007e84c590
>>>  ffffe2000194d2c8 0000000000000000 ffff88007ddc3ab8 ffffffff810a4187
>>> Call Trace:
>>>  [<ffffffff810a4187>] free_pages_bulk.clone.2+0x57/0x72
>>>  [<ffffffff810a4290>] free_hot_cold_page+0xee/0x130
>>>  [<ffffffff810a4338>] free_hot_page+0x10/0x12
>>>  [<ffffffff810a7d49>] put_page+0x91/0x9e
>>>  [<ffffffff8104df91>] ? _local_bh_enable_ip+0xe5/0xeb
>>>  [<ffffffff81308c5a>] skb_release_data+0x77/0xce
>>>  [<ffffffff81308919>] __kfree_skb+0x1e/0x81
>>>  [<ffffffff813089a9>] consume_skb+0x2d/0x2f
>>>  [<ffffffff8130b39b>] skb_free_datagram+0x19/0x3f
>>>  [<ffffffff8135987c>] udp_recvmsg+0x1dc/0x262
>>>  [<ffffffff8102a181>] ? pvclock_clocksource_read+0x47/0x83
>>>  [<ffffffff81303f22>] sock_common_recvmsg+0x39/0x4a
>>>  [<ffffffff8102a181>] ? pvclock_clocksource_read+0x47/0x83
>>>  [<ffffffff81301e71>] __sock_recvmsg+0x71/0x7d
>>>  [<ffffffff81302544>] sock_recvmsg+0xcf/0xe8
>>>  [<ffffffff8102a181>] ? pvclock_clocksource_read+0x47/0x83
>>>  [<ffffffff8105ca4b>] ? autoremove_wake_function+0x0/0x39
>>>  [<ffffffff8100e86f>] ? __xen_spin_lock+0xb7/0xcd
>>>  [<ffffffff8104de90>] ? __local_bh_disable+0xa7/0xaf
>>>  [<ffffffff8104df91>] ? _local_bh_enable_ip+0xe5/0xeb
>>>  [<ffffffff8103fe31>] ? __might_sleep+0x2d/0x110
>>>  [<ffffffff8136092d>] ? inet_ioctl+0xa5/0xa7
>>>  [<ffffffff81301c74>] ? sock_ioctl+0x1f3/0x216
>>>  [<ffffffff81301fb5>] ? sockfd_lookup_light+0x20/0x58
>>>  [<ffffffff813036a1>] sys_recvfrom+0xad/0xf7
>>>  [<ffffffff8102a181>] ? pvclock_clocksource_read+0x47/0x83
>>>  [<ffffffff8100dad7>] ? xen_clocksource_read+0x21/0x23
>>>  [<ffffffff81062f7c>] ? clocksource_read+0xc/0xe
>>>  [<ffffffff8101133a>] system_call_fastpath+0x16/0x1b
>>> Code: 10 48 89 d6 48 c1 ee 35 39 f1 75 29 f7 c2 00 00 04 00 74 21 49 63
>>> cc 48 39 48 10 75 18 80 e6 40 48 0f 44 c8 8b 51 08 85 d2 74 43 <0f>
>>> 0b eb
>>> fe 41 83 fc 09 76 a5 49 63 c4 48 89 43 10 0f ba 2b 12
>>> RIP  [<ffffffff810a2eb1>] __free_one_page+0x146/0x1e6
>>>  RSP <ffff88007ddc3a38>
>>> ---[ end trace 7d5f534b65d69935 ]---
>>>
>>> Message from syslogd@cbridge1 at Aug  4 11:00:25 ...
>>>  kernel:------------[ cut here ]------------
>>
>> How did you accomplish this? Could you be specific on how this occured?
>> What version of Xen are you using (I presume xen-unstable + 2.6-pvops
>> (xen-tip/master))?
>
> We were running our video conference server on a DomU after about 15
> video calls it crashed, it's very repeatable.

Does disabling GRO/LRO and/or checksumming change anything?

    J