[bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

[Ingo Molnar <mingo@elte.hu>]

A regression to v2.6.26:

I started getting this skb-head corruption message today, on a T60 
laptop with e1000:

PM: Removing info for No Bus:vcs11
device: 'vcs11': device_create_release
=============================================================================
BUG skbuff_head_cache: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xf658ae9c-0xf658ae9c. First byte 0x6a instead of 0x6b
INFO: Allocated in __alloc_skb+0x2c/0x110 age=0 cpu=0 pid=5098
INFO: Freed in __kfree_skb+0x31/0x80 age=0 cpu=1 pid=4440
INFO: Slab 0xc16cc140 objects=16 used=1 fp=0xf658ae00 flags=0x400000c3
INFO: Object 0xf658ae00 @offset=3584 fp=0xf658af00

Bytes b4 0xf658adf0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
  Object 0xf658ae00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xf658ae70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
 Redzone 0xf658aea0:  bb bb bb bb                                     ����            
 Padding 0xf658aec8:  5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
 Padding 0xf658aed8:  5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
 Padding 0xf658aee8:  5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
 Padding 0xf658aef8:  5a 5a 5a 5a 5a 5a 5a 5a                         ZZZZZZZZ        
Pid: 5098, comm: gdm-binary Not tainted 2.6.26-tip #3094
 [<c0186f99>] print_trailer+0xa9/0xf0
 [<c018707b>] check_bytes_and_report+0x9b/0xc0
 [<c01874ae>] check_object+0x19e/0x1e0
 [<c0187ef1>] __slab_alloc+0x371/0x4e0
 [<c0188552>] kmem_cache_alloc+0xb2/0xc0
 [<c06732dc>] ? __alloc_skb+0x2c/0x110
 [<c06732dc>] ? __alloc_skb+0x2c/0x110
 [<c06732dc>] __alloc_skb+0x2c/0x110
 [<c0685b6c>] find_skb+0x3c/0x80
 [<c068624b>] netpoll_send_udp+0x2b/0x1f0
 [<c0322e62>] ? notify_update+0x22/0x30
 [<c046d405>] write_msg+0x95/0xe0
 [<c046d370>] ? write_msg+0x0/0xe0
 [<c012df70>] __call_console_drivers+0x60/0x70
 [<c012dff9>] _call_console_drivers+0x79/0x90
 [<c012e3f4>] release_console_sem+0xc4/0x1f0
 [<c012e89e>] vprintk+0x15e/0x3b0
 [<c01ce683>] ? release_sysfs_dirent+0x43/0xa0
 [<c01ce683>] ? release_sysfs_dirent+0x43/0xa0
 [<c01ce683>] ? release_sysfs_dirent+0x43/0xa0
 [<c012eb0b>] printk+0x1b/0x20
 [<c0367597>] device_create_release+0x27/0x40
 [<c0367955>] device_release+0x15/0x70
 [<c02b1d29>] kobject_release+0x39/0x80
 [<c02b1cf0>] ? kobject_release+0x0/0x80
 [<c02b2a0d>] kref_put+0x2d/0x70
 [<c02b1c30>] kobject_put+0x20/0x50
 [<c02b1ce2>] ? kobject_del+0x22/0x30
 [<c03680f3>] ? device_del+0x123/0x140
 [<c0367b7f>] put_device+0xf/0x20
 [<c0368145>] device_unregister+0x35/0x40
 [<c0368179>] device_destroy+0x29/0x30
 [<c031e66c>] vcs_remove_sysfs+0x1c/0x40
 [<c032480e>] con_close+0x5e/0x70
 [<c0317289>] release_dev+0x139/0x600
 [<c0188222>] ? __slab_free+0x1c2/0x240
 [<c019fd59>] ? destroy_inode+0x39/0x40
 [<c019d4d3>] ? __d_free+0x23/0x30
 [<c019d4d3>] ? __d_free+0x23/0x30
 [<c019d4d3>] ? __d_free+0x23/0x30
 [<c0317762>] tty_release+0x12/0x20
 [<c018dba2>] __fput+0xb2/0x1d0
 [<c018dfe9>] fput+0x19/0x20
 [<c018ad59>] filp_close+0x49/0x70
 [<c018c3a6>] sys_close+0x66/0xb0
 [<c0103d01>] sysenter_past_esp+0x6a/0x99
 =======================
FIX skbuff_head_cache: Restoring 0xf658ae9c-0xf658ae9c=0x6b

FIX skbuff_head_cache: Marking all objects used
device: 'vcsa11': device_unregister
PM: Removing info for No Bus:vcsa11
device: 'vcsa11': device_create_release

With this config:

   http://redhat.com/~mingo/misc/config-Thu_Jul_17_20_24_45_CEST_2008.bad
	
The box uses netconsole.

Suspected range of breakage is v2.6.26..a3cf859, or around 3000 commits. 
But a fair portion of those commit were tested on this box before.

Perhaps SLUB debugging got smarter?

	Ingo