From: Dave Jones <davej@redhat.com>
To: Linux Kernel <linux-kernel@vger.kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Subject: 3.6rc6 slab corruption.
Date: Tue, 18 Sep 2012 10:35:04 -0400 [thread overview]
Message-ID: <20120918143504.GA30585@redhat.com> (raw)
I was chasing a networking bug, and had trinity reduced to just making read & setsockopt calls,
and let that run overnight. I woke up to 800mb of traces from a different bug..
The traces look mostly like this..
=============================================================================
BUG kmalloc-64 (Not tainted): Redzone overwritten
-----------------------------------------------------------------------------
INFO: 0xffff88001f4b4970-0xffff88001f4b4977. First byte 0xbb instead of 0xcc
INFO: Allocated in u32_array_read+0xd1/0x110 age=0 cpu=6 pid=32767
__slab_alloc+0x516/0x5a5
__kmalloc+0x213/0x2c0
u32_array_read+0xd1/0x110
vfs_read+0xac/0x180
sys_read+0x4d/0x90
system_call_fastpath+0x1a/0x1f
INFO: Freed in u32_array_read+0x99/0x110 age=0 cpu=0 pid=32749
__slab_free+0x3f/0x3bf
kfree+0x2d5/0x310
u32_array_read+0x99/0x110
vfs_read+0xac/0x180
sys_read+0x4d/0x90
system_call_fastpath+0x1a/0x1f
INFO: Slab 0xffffea00007d2d00 objects=41 used=14 fp=0xffff88001f4b7410 flags=0x10000000004081
INFO: Object 0xffff88001f4b4930 @offset=2352 fp=0xffff88001f4b7410
Bytes b4 ffff88001f4b4920: 1b 20 1c 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a . ......ZZZZZZZZ
Object ffff88001f4b4930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff88001f4b4970: bb bb bb bb bb bb bb bb ........
Padding ffff88001f4b4ab0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Pid: 32756, comm: trinity-child52 Not tainted 3.6.0-rc6+ #44
Call Trace:
[<ffffffff811c10ad>] ? print_section+0x3d/0x40
[<ffffffff811c23fe>] print_trailer+0xfe/0x160
[<ffffffff811c2592>] check_bytes_and_report+0xe2/0x120
[<ffffffff81023b79>] ? native_sched_clock+0x19/0x80
[<ffffffff811c2b5b>] check_object+0x18b/0x250
[<ffffffff8169b7d7>] free_debug_processing+0xc0/0x1fd
[<ffffffff812d2e29>] ? u32_array_read+0x99/0x110
[<ffffffff8169ba5c>] __slab_free+0x3f/0x3bf
[<ffffffff81365a1c>] ? debug_check_no_obj_freed+0x16c/0x210
[<ffffffff810db04f>] ? lock_release_holdtime.part.26+0xf/0x180
[<ffffffff812d2e29>] ? u32_array_read+0x99/0x110
[<ffffffff811c5725>] kfree+0x2d5/0x310
[<ffffffff812d2e29>] u32_array_read+0x99/0x110
[<ffffffff811df88c>] vfs_read+0xac/0x180
[<ffffffff811df9ad>] sys_read+0x4d/0x90
[<ffffffff816aea2d>] system_call_fastpath+0x1a/0x1f
FIX kmalloc-64: Restoring 0xffff88001f4b4970-0xffff88001f4b4977=0xcc
=============================================================================
Which looks like we read some file (probably something in sysfs/procfs) that corrupted some internal state.
Any ideas on what I could do to narrow this down ?
The full traces are at http://www.codemonkey.org.uk/junk/slab-corrupt.txt
They vary a little later, but it looks like it's probably all the same problem to me.
Sometimes it flip-flops between "First byte 0xbb instead of 0xcc" and "First byte 0xcc instead of 0xbb"
The one outlier being this weird message..
Sep 18 02:00:13 bitcrush kernel: [36617.487681] hrtimer: interrupt took 242337 ns
Which is weird, but probably unrelated.
Dave
next reply other threads:[~2012-09-18 14:35 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-18 14:35 Dave Jones [this message]
2012-09-18 18:38 ` 3.6rc6 slab corruption Linus Torvalds
2012-09-18 18:53 ` Dave Jones
2012-09-19 14:00 ` Raghavendra K T
2012-09-19 17:09 ` Linus Torvalds
2012-09-19 21:27 ` David Rientjes
2012-09-19 21:41 ` Dave Jones
2012-09-18 19:23 ` Konrad Rzeszutek Wilk
2012-09-18 20:24 ` David Rientjes
2012-09-18 20:31 ` Konrad Rzeszutek Wilk
2012-09-18 20:36 ` Linus Torvalds
2012-09-19 0:57 ` Raghavendra K T
2012-09-18 20:35 ` Linus Torvalds
2012-09-18 20:37 ` Konrad Rzeszutek Wilk
2012-09-18 20:49 ` Linus Torvalds
2012-09-19 1:16 ` Raghavendra K T
2012-09-19 19:16 ` Konrad Rzeszutek Wilk
2012-09-19 21:29 ` David Rientjes
2012-09-19 21:49 ` David Rientjes
2012-09-19 23:01 ` Linus Torvalds
2012-09-19 23:20 ` David Rientjes
2012-09-20 21:14 ` Konrad Rzeszutek Wilk
2012-09-20 2:29 ` Raghavendra K T
2012-09-20 2:46 ` David Rientjes
2012-09-20 2:55 ` Raghavendra K T
2012-09-20 21:18 ` Konrad Rzeszutek Wilk
2012-09-21 9:16 ` [patch for-3.6] fs, debugfs: fix race in u32_array_read and allocate array at open David Rientjes
2012-09-21 10:22 ` Raghavendra K T
2012-09-24 22:26 ` David Rientjes
2012-09-25 2:54 ` Raghavendra K T
2012-09-20 12:57 ` 3.6rc6 slab corruption Raghavendra K T
2012-09-20 21:18 ` Konrad Rzeszutek Wilk
2012-09-20 12:55 ` Raghavendra K T
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120918143504.GA30585@redhat.com \
--to=davej@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.