Kernel crash with 2.6.36

Kernel crash with 2.6.36

[Frédéric L. W. Meunier]

Is this supposed to be a bug in 2.6.36 ? Firefox was running, but I 
wasn't browsing with it. The computer just locked up, first killing X. 
SysRq didn't work.

Nov  7 16:21:23 pervalidus kernel: BUG: unable to handle kernel NULL pointer dereference at (null)
Nov  7 16:21:23 pervalidus kernel: IP: [<c118cc76>] skb_dequeue+0x38/0x4a
Nov  7 16:21:23 pervalidus kernel: *pde = 00000000 
Nov  7 16:21:23 pervalidus kernel: Oops: 0002 [#1] PREEMPT SMP 
Nov  7 16:21:23 pervalidus kernel: last sysfs file: /sys/devices/platform/it87.552/in4_input
Nov  7 16:21:23 pervalidus kernel: Modules linked in: pppoe pppox ppp_generic slhc af_packet snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss powernow_k8 freq_table mperf it87 hwmon_vid ipt_LOG xt_tcpudp iptable_filter ip_tables x_tables fuse joydev tuner_simple tuner_types radeon ati_agp usbhid hid ttm drm_kms_helper tuner tvaudio drm tda7432 snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus bttv snd_pcm v4l2_common snd_seq_device videodev snd_timer v4l1_compat videobuf_dma_sg snd_page_alloc videobuf_core snd_util_mem ir_lirc_codec rtc_cmos rtc_core rtc_lib lirc_dev btcx_risc ir_common agpgart ir_core thermal tveeprom i2c_piix4 i2c_algo_bit sr_mod 8250_pnp processor ohci_hcd parport_pc snd_hwdep k8temp i2c_core cdrom 8250 cfbcopyarea ehci_hcd snd usbcore soundcore emu10k1_gp gameport nls_base evdev sg cfbimgblt parport serial_core r8169 cfbfillrect thermal_sys hwmon mii pcspkr floppy psmouse button
Nov  7 16:21:23 pervalidus kernel: 
Nov  7 16:21:23 pervalidus kernel: Pid: 10468, comm: firefox-bin Not tainted 2.6.36 #1 GA-MA69VM-S2/GA-MA69VM-S2
Nov  7 16:21:23 pervalidus kernel: EIP: 0060:[<c118cc76>] EFLAGS: 00210096 CPU: 0
Nov  7 16:21:23 pervalidus kernel: EIP is at skb_dequeue+0x38/0x4a
Nov  7 16:21:23 pervalidus kernel: EAX: 00200246 EBX: f6b1de40 ECX: 00000000 EDX: 00009e9e
Nov  7 16:21:23 pervalidus kernel: ESI: f6715680 EDI: f6b1d180 EBP: ed911dec ESP: ed911de0
Nov  7 16:21:23 pervalidus kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Nov  7 16:21:23 pervalidus kernel: Process firefox-bin (pid: 10468, ti=ed911000 task=f673acb0 task.ti=ed911000)
Nov  7 16:21:23 pervalidus kernel: Stack:
Nov  7 16:21:23 pervalidus kernel: f6715600 00000001 ed911e50 ed911e70 c11da7e0 f673acb0 f673acb0 00000000
Nov  7 16:21:23 pervalidus kernel: <0> f44b7680 f673acb0 f67156d8 f671576c 00000001 f6715674 00000000 00000000
Nov  7 16:21:23 pervalidus kernel: <0> ed911eb0 ed911e90 ffffffa1 f6715794 00000020 f6b1de40 f6987580 f6622f80
Nov  7 16:21:23 pervalidus kernel: Call Trace:
Nov  7 16:21:23 pervalidus kernel: [<c11da7e0>] ? unix_stream_recvmsg+0x103/0x424
Nov  7 16:21:23 pervalidus kernel: [<c11ea637>] ? _raw_spin_unlock_irqrestore+0x16/0x21
Nov  7 16:21:23 pervalidus kernel: [<c1188f69>] ? sock_aio_read+0xe8/0xf2
Nov  7 16:21:23 pervalidus kernel: [<c1079cb0>] ? do_sync_read+0x8a/0xc5
Nov  7 16:21:23 pervalidus kernel: [<c107a205>] ? vfs_read+0x83/0xa8
Nov  7 16:21:23 pervalidus kernel: [<c107a2be>] ? sys_read+0x3b/0x5d
Nov  7 16:21:23 pervalidus kernel: [<c1002690>] ? sysenter_do_call+0x12/0x26
Nov  7 16:21:23 pervalidus kernel: Code: e8 3c d7 05 00 8b 1f 39 fb 75 04 31 db eb 1e 85 db 74 1a ff 4f 08 8b 3b 8b 4b 04 c7 03 00 00 00 00 c7 43 04 00 00 00 00 89 4f 04 <89> 39 89 c2 89 f0 e8 a0 d9 05 00 89 d8 5b 5e 5f c9 c3 55 89 e5 
Nov  7 16:21:23 pervalidus kernel: EIP: [<c118cc76>] skb_dequeue+0x38/0x4a SS:ESP 0068:ed911de0
Nov  7 16:21:23 pervalidus kernel: CR2: 0000000000000000
Nov  7 16:21:23 pervalidus kernel: ---[ end trace 4cd73789c5986bc8 ]---
Nov  7 16:21:23 pervalidus kernel: note: firefox-bin[10468] exited with preempt_count 2

BUG in skb_dequeue (skb->next is NULL) (was: Re: Kernel crash with 2.6.36)

[Simon Schubert]

Frédéric L. W. Meunier <2 <at> pervalidus.net> writes:

> Nov  7 16:21:23 pervalidus kernel: BUG: unable to handle kernel NULL pointer 
dereference at (null)

I can confirm the bug reported by Frederic.  The culprit is at:

        next->prev = prev;

in __skb_unlink().

Something must be putting NULL pointers in the skb list.

Let me know how I can be of further help.


BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81459069>] skb_dequeue+0x59/0x90
PGD 208379067 PUD 20a523067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
CPU 1 
Modules linked in: binfmt_misc kvm_intel kvm ipt_MASQUERADE iptable_nat nf_nat 
xfrm_user nf_conntrack_ipv4 xfrm4_tunnel tunnel4 nf_conntrack ipcomp xfrm_ipcomp 
esp4 ah4 nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle deflate 
ip_tables zlib_deflate ctr x_tables twofish_generic twofish_x86_64 
twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 
sha512_generic sha1_generic crypto_null af_key pppoe pppox xfs exportfs 
snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss 
snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi 
snd_seq_midi_event snd_seq snd_timer ftdi_sio snd_seq_device usbserial hwmon_vid 
coretemp snd lp tpm_tis ppdev tpm parport_pc asus_atk0110 tpm_bios parport 
soundcore snd_page_alloc sha256_generic cryptd aes_x86_64 aes_generic dm_crypt 
raid10 raid1 raid0 multipath linear raid456 async_pq async_xor xor async_memcpy 
async_raid6_recov raid6_pq async_tx fbcon tileblit font bitblit softcursor usb

Pid: 2229, comm: pulseaudio Not tainted 2.6.36 #10 P5Q-VM DO/System Product Name
RIP: 0010:[<ffffffff81459069>]  [<ffffffff81459069>] skb_dequeue+0x59/0x90
RSP: 0018:ffff880221927a78  EFLAGS: 00010097
RAX: 0000000000000282 RBX: ffff88017a6d5e14 RCX: ffff88022236d200
RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffff88017a6d5e14
RBP: ffff880221927a98 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000293 R12: ffff88022236d200
R13: ffff88017a6d5e00 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fedd4d9f740(0000) GS:ffff880001e80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000020837a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process pulseaudio (pid: 2229, threadinfo ffff880221926000, task 
ffff8802210096d0)
Stack:
 ffff88022236d200 ffff88022236d228 ffff88022236d200 ffff880221927f18
<0> ffff880221927be8 ffffffff814ef0fa ffff880221927e4c ffff880221927eb4
<0> 0000000000000040 ffff8802210096d0 ffff8802210096d0 ffff88017a6d5ea4
Call Trace:
 [<ffffffff814ef0fa>] unix_stream_recvmsg+0x1aa/0x790
 [<ffffffff8145124d>] sock_recvmsg+0xfd/0x130
 [<ffffffff81155fd0>] ? pollwake+0x0/0x60
 [<ffffffff81452b54>] __sys_recvmsg+0x144/0x2e0
 [<ffffffff81155fd0>] ? pollwake+0x0/0x60
 [<ffffffff8104d88a>] ? finish_task_switch+0x4a/0xd0
 [<ffffffff8154fa91>] ? schedule+0x411/0xa50
 [<ffffffff81452f99>] sys_recvmsg+0x49/0x80
 [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
Code: e5 74 4f 4d 85 e4 74 26 41 83 6d 10 01 49 8b 0c 24 49 8b 54 24 08 49 c7 04 
24 00 00 00 00 49 c7 44 24 08 00 00 00 00 48 89 51 08 <48> 89 0a 48 89 c6 48 89 
df e8 39 93 0f 00 4c 89 e0 48 8b 5d e8 
RIP  [<ffffffff81459069>] skb_dequeue+0x59/0x90
 RSP <ffff880221927a78>
CR2: 0000000000000000
---[ end trace d4be3de9fdd70935 ]---

cheers
  simon

Re: BUG in skb_dequeue (skb->next is NULL) (was: Re: Kernel crash with 2.6.36)

[Eric Dumazet]

Le mercredi 01 décembre 2010 à 09:02 +0000, Simon Schubert a écrit :
> Frédéric L. W. Meunier <2 <at> pervalidus.net> writes:

CC netdev

> 
> > Nov  7 16:21:23 pervalidus kernel: BUG: unable to handle kernel NULL pointer 
> dereference at (null)
> 
> I can confirm the bug reported by Frederic.  The culprit is at:
> 
>         next->prev = prev;
> 
> in __skb_unlink().
> 
> Something must be putting NULL pointers in the skb list.
> 
> Let me know how I can be of further help.
> 
> 
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<ffffffff81459069>] skb_dequeue+0x59/0x90
> PGD 208379067 PUD 20a523067 PMD 0 
> Oops: 0002 [#1] SMP 
> last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq
> CPU 1 
> Modules linked in: binfmt_misc kvm_intel kvm ipt_MASQUERADE iptable_nat nf_nat 
> xfrm_user nf_conntrack_ipv4 xfrm4_tunnel tunnel4 nf_conntrack ipcomp xfrm_ipcomp 
> esp4 ah4 nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle deflate 
> ip_tables zlib_deflate ctr x_tables twofish_generic twofish_x86_64 
> twofish_common camellia serpent blowfish cast5 des_generic xcbc rmd160 
> sha512_generic sha1_generic crypto_null af_key pppoe pppox xfs exportfs 
> snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss 
> snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi 
> snd_seq_midi_event snd_seq snd_timer ftdi_sio snd_seq_device usbserial hwmon_vid 
> coretemp snd lp tpm_tis ppdev tpm parport_pc asus_atk0110 tpm_bios parport 
> soundcore snd_page_alloc sha256_generic cryptd aes_x86_64 aes_generic dm_crypt 
> raid10 raid1 raid0 multipath linear raid456 async_pq async_xor xor async_memcpy 
> async_raid6_recov raid6_pq async_tx fbcon tileblit font bitblit softcursor usb
> 
> Pid: 2229, comm: pulseaudio Not tainted 2.6.36 #10 P5Q-VM DO/System Product Name
> RIP: 0010:[<ffffffff81459069>]  [<ffffffff81459069>] skb_dequeue+0x59/0x90
> RSP: 0018:ffff880221927a78  EFLAGS: 00010097
> RAX: 0000000000000282 RBX: ffff88017a6d5e14 RCX: ffff88022236d200
> RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffff88017a6d5e14
> RBP: ffff880221927a98 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000293 R12: ffff88022236d200
> R13: ffff88017a6d5e00 R14: 0000000000000000 R15: 0000000000000000
> FS:  00007fedd4d9f740(0000) GS:ffff880001e80000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 000000020837a000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process pulseaudio (pid: 2229, threadinfo ffff880221926000, task 
> ffff8802210096d0)
> Stack:
>  ffff88022236d200 ffff88022236d228 ffff88022236d200 ffff880221927f18
> <0> ffff880221927be8 ffffffff814ef0fa ffff880221927e4c ffff880221927eb4
> <0> 0000000000000040 ffff8802210096d0 ffff8802210096d0 ffff88017a6d5ea4
> Call Trace:
>  [<ffffffff814ef0fa>] unix_stream_recvmsg+0x1aa/0x790
>  [<ffffffff8145124d>] sock_recvmsg+0xfd/0x130
>  [<ffffffff81155fd0>] ? pollwake+0x0/0x60
>  [<ffffffff81452b54>] __sys_recvmsg+0x144/0x2e0
>  [<ffffffff81155fd0>] ? pollwake+0x0/0x60
>  [<ffffffff8104d88a>] ? finish_task_switch+0x4a/0xd0
>  [<ffffffff8154fa91>] ? schedule+0x411/0xa50
>  [<ffffffff81452f99>] sys_recvmsg+0x49/0x80
>  [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
> Code: e5 74 4f 4d 85 e4 74 26 41 83 6d 10 01 49 8b 0c 24 49 8b 54 24 08 49 c7 04 
> 24 00 00 00 00 49 c7 44 24 08 00 00 00 00 48 89 51 08 <48> 89 0a 48 89 c6 48 89 
> df e8 39 93 0f 00 4c 89 e0 48 8b 5d e8 
> RIP  [<ffffffff81459069>] skb_dequeue+0x59/0x90
>  RSP <ffff880221927a78>
> CR2: 0000000000000000
> ---[ end trace d4be3de9fdd70935 ]---
> 
> cheers
>   simon
> 
> 

Is it reproductible ?

On previous kernel (say 2.6.35) you never hit this bug ?

Re: BUG in skb_dequeue (skb->next is NULL)

[Simon Schubert]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On 12/01/2010 11:28 AM, Eric Dumazet wrote:
> Le mercredi 01 décembre 2010 à 09:02 +0000, Simon Schubert a écrit :
>> Frédéric L. W. Meunier <2 <at> pervalidus.net> writes:
> 
> CC netdev
>> RIP: 0010:[<ffffffff81459069>]  [<ffffffff81459069>] skb_dequeue+0x59/0x90
>> Call Trace:
>>  [<ffffffff814ef0fa>] unix_stream_recvmsg+0x1aa/0x790
>>  [<ffffffff8145124d>] sock_recvmsg+0xfd/0x130
>>  [<ffffffff81155fd0>] ? pollwake+0x0/0x60
>>  [<ffffffff81452b54>] __sys_recvmsg+0x144/0x2e0
>>  [<ffffffff81155fd0>] ? pollwake+0x0/0x60
>>  [<ffffffff8104d88a>] ? finish_task_switch+0x4a/0xd0
>>  [<ffffffff8154fa91>] ? schedule+0x411/0xa50
>>  [<ffffffff81452f99>] sys_recvmsg+0x49/0x80
>>  [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
> 
> Is it reproductible ?

I can't trigger it on purpose, but it happened two nights in a row (I
don't have the previous backtrace though)

> On previous kernel (say 2.6.35) you never hit this bug ?

No.

cheers
  simon


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]