Re: 2.6.39-rc2 boot crash

[Eric B Munson <emunson@mgebm.net>]

[-- Attachment #1: Type: text/plain, Size: 8883 bytes --]

On Wed, 06 Apr 2011, David Miller wrote:

> From: Eric B Munson <emunson@mgebm.net>
> Date: Wed, 6 Apr 2011 17:20:41 -0400
> 
> > A bisect points at commit 04f482faf50535229a5a5c8d629cf963899f857c for the
> > first bad one.  Unfortunately, I have not made netconsole work yet and the
> > hang is happening mostly right when X starts so I can't even see the console.
> > I will keep at the netconsole and see if I can get it functioning, also I will
> > try to boot this kernel in a VM and see if that helps.
> 
> Patrick, please help Eric so we can fix this bug.
> 
> Thanks.
> 

I have a useful trace now from netconsole:

[   18.029521] BUG: sleeping function called from invalid context at arch/x86/mm/fault.c:1087
[   18.029527] in_atomic(): 0, irqs_disabled(): 1, pid: 2018, name: cgrulesengd
[   18.029693] BUG: unable to handle kernel paging request at 0000100000000000
[   18.029730] IP: [<ffffffff814c3db8>] __skb_recv_datagram+0x128/0x2b0
[   18.029756] PGD 0 
[   18.029768] Oops: 0002 [#1] SMP 
[   18.029790] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/usb10/10-0:1.0/bInterfaceClass
[   18.029824] CPU 0 
[   18.029833] Modules linked in: kvm_intel kvm parport_pc ppdev snd_hda_codec_hdmi snd_hda_codec_realtek nfs lockd fscache auth_rpcgss nfs_acl sunrpc radeon deflate zlib_deflate ctr twofish_generic twofish_x86_64 twofish_common ttm camellia serpent drm_kms_helper snd_usb_audio blowfish cast5 snd_hda_intel drm des_generic snd_hda_codec snd_hwdep aesni_intel snd_usbmidi_lib cryptd aes_x86_64 aes_generic snd_pcm xcbc snd_seq_midi rmd160 snd_rawmidi sha512_generic sha256_generic uvcvideo snd_seq_midi_event sha1_generic snd_seq snd_timer crypto_null snd_seq_device snd af_key xhci_hcd i7core_edac videodev joydev psmouse edac_core v4l2_compat_ioctl32 w83627ehf soundcore serio_raw hwmon_vid snd_page_alloc max6650 hid_microsoft i2c_algo_bit lp parport asus_atk0110 usbhid hid firewire_ohci firewire_core crc_itu_t
[   18.030424] 
[   18.030432] Pid: 2018, comm: cgrulesengd Not tainted 2.6.39-rc2+ #52 System manufacturer System Product Name/P6X58D PREMIUM
[   18.030477] RIP: 0010:[<ffffffff814c3db8>]  [<ffffffff814c3db8>] __skb_recv_datagram+0x128/0x2b0
[   18.030510] RSP: 0018:ffff880326f03b28  EFLAGS: 00010002
[   18.030528] RAX: 0000000000000286 RBX: ffff8803204c5100 RCX: 0000100000000000
[   18.030552] RDX: ffff88031fe47200 RSI: ffff880326f03bf4 RDI: 0000000000000046
[   18.030576] RBP: ffff880326f03bd8 R08: 0000000000000000 R09: 0000000000000000
[   18.030599] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880327d6e928
[   18.030623] R13: ffff880326f03b78 R14: ffff880326f03b90 R15: ffff880327d6e940
[   18.030646] FS:  00007f3bf9173b20(0000) GS:ffff880331600000(0000) knlGS:0000000000000000
[   18.030673] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   18.030693] CR2: 0000100000000000 CR3: 0000000326dda000 CR4: 00000000000006f0
[   18.030716] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   18.030740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   18.030763] Process cgrulesengd (pid: 2018, threadinfo ffff880326f02000, task ffff8803275aa300)
[   18.030794] Stack:
[   18.030803]  ffff880300000000 ffff8803275aa338 ffff880327d6ebd0 ffff8803275aa300
[   18.030839]  7fffffffffffffff ffff880326f03c74 ffff880326f03bf4 0000000000000001
[   18.030872]  ffff8803275aa300 ffff880327d6e940 00000000000001f7 0000000000000001
[   18.030905] Call Trace:
[   18.030916]  [<ffffffff81009833>] ? native_sched_clock+0x13/0x60
[   18.030936]  [<ffffffff814c3f64>] skb_recv_datagram+0x24/0x30
[   18.030956]  [<ffffffff814f463c>] netlink_recvmsg+0x7c/0x430
[   18.030975]  [<ffffffff814bc185>] ? sock_update_classid+0x65/0x100
[   18.030996]  [<ffffffff814bc19d>] ? sock_update_classid+0x7d/0x100
[   18.031016]  [<ffffffff814bc1c0>] ? sock_update_classid+0xa0/0x100
[   18.031037]  [<ffffffff814b7c1d>] sock_recvmsg+0xfd/0x130
[   18.031055]  [<ffffffff81178af8>] ? set_fd_set+0x48/0x60
[   18.031073]  [<ffffffff8117a25b>] ? core_sys_select+0x26b/0x330
[   18.031093]  [<ffffffff8117a03d>] ? core_sys_select+0x4d/0x330
[   18.031112]  [<ffffffff8108cc05>] ? lock_release_holdtime+0x35/0x160
[   18.031133]  [<ffffffff814b7da1>] sys_recvfrom+0xf1/0x170
[   18.031152]  [<ffffffff815d40ba>] ? sysret_check+0x2e/0x69
[   18.031171]  [<ffffffff812f02de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   18.031193]  [<ffffffff815d4082>] system_call_fastpath+0x16/0x1b
[   18.031212] Code: 41 5d 41 5e 41 5f c9 c3 eb 01 90 ff 8b 38 01 00 00 48 8b 1a 48 8b 4a 08 48 c7 02 00 00 00 00 48 c7 42 08 00 00 00 00 48 89 4b 08 
[   18.031494]  89 19 eb aa eb 01 90 48 8b 83 f0 03 00 00 48 89 85 70 ff ff 
[   18.031601] RIP  [<ffffffff814c3db8>] __skb_recv_datagram+0x128/0x2b0
[   18.031625]  RSP <ffff880326f03b28>
[   18.031637] CR2: 0000100000000000
[   18.039388] ---[ end trace 0e3e016130139f1b ]---
[   18.112703] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   18.112738] IP: [<ffffffff814befed>] skb_queue_tail+0x3d/0x60
[   18.112763] PGD 0 
[   18.112775] Oops: 0002 [#2] SMP 
[   18.112796] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/usb10/10-0:1.0/bInterfaceClass
[   18.112828] CPU 0 
[   18.112837] Modules linked in: kvm_intel kvm parport_pc ppdev snd_hda_codec_hdmi snd_hda_codec_realtek nfs lockd fscache auth_rpcgss nfs_acl sunrpc radeon deflate zlib_deflate ctr twofish_generic twofish_x86_64 twofish_common ttm camellia serpent drm_kms_helper snd_usb_audio blowfish cast5 snd_hda_intel drm des_generic snd_hda_codec snd_hwdep aesni_intel snd_usbmidi_lib cryptd aes_x86_64 aes_generic snd_pcm xcbc snd_seq_midi rmd160 snd_rawmidi sha512_generic sha256_generic uvcvideo snd_seq_midi_event sha1_generic snd_seq snd_timer crypto_null snd_seq_device snd af_key xhci_hcd i7core_edac videodev joydev psmouse edac_core v4l2_compat_ioctl32 w83627ehf soundcore serio_raw hwmon_vid snd_page_alloc max6650 hid_microsoft i2c_algo_bit lp parport asus_atk0110 usbhid hid firewire_ohci firewire_core crc_itu_t
[   18.115476] 
[   18.117533] Pid: 2178, comm: 0dns-down Tainted: G      D     2.6.39-rc2+ #52 System manufacturer System Product Name/P6X58D PREMIUM
[   18.119646] RIP: 0010:[<ffffffff814befed>]  [<ffffffff814befed>] skb_queue_tail+0x3d/0x60
[   18.121757] RSP: 0018:ffff88032666bd08  EFLAGS: 00010096
[   18.123845] RAX: 0000000000000282 RBX: ffff880327d6e928 RCX: 000000000acc7db8
[   18.125948] RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffff880327d6e940
[   18.128046] RBP: ffff88032666bd28 R08: 0000000000000000 R09: 0000000000000001
[   18.130171] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880327d6e940
[   18.132281] R13: ffff880320929b00 R14: ffff880327d6e818 R15: ffff880327d6e800
[   18.134388] FS:  0000000000000000(0000) GS:ffff880331600000(0000) knlGS:0000000000000000
[   18.136498] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   18.138610] CR2: 0000000000000000 CR3: 0000000001a03000 CR4: 00000000000006f0
[   18.140732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   18.142839] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   18.144953] Process 0dns-down (pid: 2178, threadinfo ffff88032666a000, task ffff880326f0a300)
[   18.147057] Stack:
[   18.149156]  ffff88032666bd28 0000000000000000 ffff88032a7fa800 0000000000000000
[   18.151256]  ffff88032666bdb8 ffffffff814f4d12 0000000000000000 ffff880320929b00
[   18.153365]  ffff880327d6e84c ffff880320929bec 0000000026f0a300 0000000000000000
[   18.155464] Call Trace:
[   18.157539]  [<ffffffff814f4d12>] netlink_broadcast_filtered+0x322/0x480
[   18.159575]  [<ffffffff814f4e8d>] netlink_broadcast+0x1d/0x20
[   18.161568]  [<ffffffff813a0223>] cn_netlink_send+0x1a3/0x1c0
[   18.163515]  [<ffffffff813a044a>] proc_exit_connector+0xda/0x100
[   18.165538]  [<ffffffff81055a08>] do_exit+0x1d8/0x870
[   18.167428]  [<ffffffff810570fe>] ? sys_wait4+0xae/0x100
[   18.169287]  [<ffffffff812f0354>] ? lockdep_sys_exit_thunk+0x35/0x67
[   18.171133]  [<ffffffff810560fe>] do_group_exit+0x5e/0xd0
[   18.172965]  [<ffffffff81056187>] sys_exit_group+0x17/0x20
[   18.174782]  [<ffffffff815d4082>] system_call_fastpath+0x16/0x1b
[   18.176600] Code: 6d f8 0f 1f 44 00 00 49 89 f5 48 89 fb 4c 8d 67 18 4c 89 e7 e8 65 c6 10 00 48 8b 53 08 4c 89 e7 49 89 5d 00 49 89 55 08 48 89 c6 <4c> 89 2a 4c 89 6b 08 ff 43 10 e8 54 cf 10 00 48 8b 5d e8 4c 8b 
[   18.178889] RIP  [<ffffffff814befed>] skb_queue_tail+0x3d/0x60
[   18.180925]  RSP <ffff88032666bd08>
[   18.182948] CR2: 0000000000000000
[   18.184969] ---[ end trace 0e3e016130139f1c ]---
[   18.184972] Fixing recursive fault but reboot is needed!

I haven't dug into it at all, but I am happy to help test potential fixes.

Eric

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]