From: "Paweł Sikora" <pluto@agmk.net>
To: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, neilb@suse.de
Subject: Re: [2.6.37-rc8] BUG kmalloc-256: Poison overwritten.
Date: Thu, 30 Dec 2010 20:39:33 +0100 [thread overview]
Message-ID: <201012302039.33305.pluto@agmk.net> (raw)
In-Reply-To: <1293723098.25156.2.camel@jaguar>
On Thursday 30 of December 2010 16:31:38 Pekka Enberg wrote:
> On Thu, 2010-12-30 at 16:08 +0100, Pawel Sikora wrote:
> > [ 1863.448308] =============================================================================
> > [ 1863.448313] BUG kmalloc-256: Poison overwritten
> > [ 1863.448315] -----------------------------------------------------------------------------
> > [ 1863.448316]
> > [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b
> > [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766
> > [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266
> > [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1
> > [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338
> > [ 1863.448343]
> > [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ
> > [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk
> > [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk�
> > [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb ��������
> > [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
> > [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1
> > [ 1863.448489] Call Trace:
> > [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160
> > [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130
> > [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270
> > [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390
> > [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390
> > [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0
> > [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410
> > [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0
> > [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390
> > [ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0
> > [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390
> > [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50
> > [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70
> > [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0
> > [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b
> > [ 1863.448557]
> > [ 1863.448559] FIX kmalloc-256: Marking all objects used
>
> This looks like a use-after-free bug somewhere in drivers/md/raid10.c.
>
> Pekka
i think it's quite easy to reproduce this problem. here's a mini howto:
- setup two raid10 matrices.
[root@odra ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid10]
md3 : active raid10 sdd4[1] sdc4[0]
424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
[>....................] resync = 0.4% (1966592/424757248) finish=82.4min speed=85504K/sec
md2 : active raid10 sdb4[1] sda4[0]
424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
[>....................] resync = 0.5% (2446080/424757248) finish=97.1min speed=72432K/sec
- stop matrices.
[root@odra ~]# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
[root@odra ~]# mdadm --stop /dev/md3
mdadm: stopped /dev/md3
- create raid0 on devices previously used by raid10.
[root@odra ~]# mdadm -C /dev/md2 -l 0 -n 4 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@odra ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid10]
md2 : active raid0 sdd4[3] sdc4[2] sdb4[1] sda4[0]
1699028992 blocks super 1.2 512k chunks
- stop it.
[root@odra ~]# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
- create one raid10 matrix once more.
[root@odra ~]# mdadm -C /dev/md2 -l 10 -n 2 --layout f2 /dev/sda4 /dev/sdb4
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
- in this moment i can see a bug report.
Dec 30 20:08:46 odra kernel: [12501.627162] =============================================================================
Dec 30 20:08:46 odra kernel: [12501.627166] BUG kmalloc-256: Poison overwritten
Dec 30 20:08:46 odra kernel: [12501.627168] -----------------------------------------------------------------------------
Dec 30 20:08:46 odra kernel: [12501.627169]
Dec 30 20:08:46 odra kernel: [12501.627172] INFO: 0xffff8803feb5e15c-0xffff8803feb5e15d. First byte 0x6c instead of 0x6b
Dec 30 20:08:46 odra kernel: [12501.627178] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=58297 cpu=2 pid=12007
Dec 30 20:08:46 odra kernel: [12501.627182] INFO: Freed in stop+0x66/0x80 [raid10] age=47657 cpu=2 pid=12047
Dec 30 20:08:46 odra kernel: [12501.627185] INFO: Slab 0xffffea000dfb7c90 objects=24 used=2 fp=0xffff8803feb5e148 flags=0x2000000000040c1
Dec 30 20:08:46 odra kernel: [12501.627188] INFO: Object 0xffff8803feb5e148 @offset=328 fp=0xffff8803feb5e3d8
Dec 30 20:08:46 odra kernel: [12501.627189]
Dec 30 20:08:46 odra kernel: [12501.627191] Bytes b4 0xffff8803feb5e138: df a8 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ߨ��....ZZZZZZZZ
Dec 30 20:08:46 odra kernel: [12501.627199] Object 0xffff8803feb5e148: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627208] Object 0xffff8803feb5e158: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627215] Object 0xffff8803feb5e168: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627223] Object 0xffff8803feb5e178: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627230] Object 0xffff8803feb5e188: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627237] Object 0xffff8803feb5e198: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627245] Object 0xffff8803feb5e1a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627252] Object 0xffff8803feb5e1b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627259] Object 0xffff8803feb5e1c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627267] Object 0xffff8803feb5e1d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627274] Object 0xffff8803feb5e1e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627281] Object 0xffff8803feb5e1f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627289] Object 0xffff8803feb5e208: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627296] Object 0xffff8803feb5e218: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627303] Object 0xffff8803feb5e228: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Dec 30 20:08:46 odra kernel: [12501.627311] Object 0xffff8803feb5e238: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk�
Dec 30 20:08:46 odra kernel: [12501.627318] Redzone 0xffff8803feb5e248: bb bb bb bb bb bb bb bb ��������
Dec 30 20:08:46 odra kernel: [12501.627325] Padding 0xffff8803feb5e288: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Dec 30 20:08:46 odra kernel: [12501.627334] Pid: 12168, comm: mdadm Not tainted 2.6.37-rc8 #1
Dec 30 20:08:46 odra kernel: [12501.627336] Call Trace:
Dec 30 20:08:46 odra kernel: [12501.627343] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160
Dec 30 20:08:46 odra kernel: [12501.627347] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130
Dec 30 20:08:46 odra kernel: [12501.627350] [<ffffffff8111f2da>] check_object+0x22a/0x270
Dec 30 20:08:46 odra kernel: [12501.627354] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10]
Dec 30 20:08:46 odra kernel: [12501.627358] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10]
Dec 30 20:08:46 odra kernel: [12501.627361] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0
Dec 30 20:08:46 odra kernel: [12501.627365] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410
Dec 30 20:08:46 odra kernel: [12501.627369] [<ffffffff810de600>] ? mempool_alloc_slab+0x10/0x20
Dec 30 20:08:46 odra kernel: [12501.627372] [<ffffffff8112166f>] ? kmem_cache_alloc_node_notrace+0xbf/0xe0
Dec 30 20:08:46 odra kernel: [12501.627376] [<ffffffff810de7fe>] ? mempool_create_node+0x7e/0x1a0
Dec 30 20:08:46 odra kernel: [12501.627379] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10]
Dec 30 20:08:46 odra kernel: [12501.627382] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0
Dec 30 20:08:46 odra kernel: [12501.627386] [<ffffffffa03ff1eb>] setup_conf+0x12b/0x360 [raid10]
Dec 30 20:08:46 odra kernel: [12501.627390] [<ffffffffa04026b1>] run+0x21/0x3c0 [raid10]
Dec 30 20:08:46 odra kernel: [12501.627413] [<ffffffffa00ca322>] md_run+0x322/0x920 [md_mod]
Dec 30 20:08:46 odra kernel: [12501.627417] [<ffffffff813fd7a0>] ? __mutex_lock_interruptible_slowpath+0x1e0/0x2b0
Dec 30 20:08:46 odra kernel: [12501.627425] [<ffffffffa00ca939>] do_md_run+0x19/0xa0 [md_mod]
Dec 30 20:08:46 odra kernel: [12501.627432] [<ffffffffa00cbefc>] md_ioctl+0xa1c/0x1350 [md_mod]
Dec 30 20:08:46 odra kernel: [12501.627435] [<ffffffff8111f15f>] ? check_object+0xaf/0x270
Dec 30 20:08:46 odra kernel: [12501.627438] [<ffffffff8111f706>] ? init_object+0x46/0x80
Dec 30 20:08:46 odra kernel: [12501.627442] [<ffffffff812039e0>] blkdev_ioctl+0x230/0x720
Dec 30 20:08:46 odra kernel: [12501.627445] [<ffffffff81120846>] ? __slab_free+0x136/0x150
Dec 30 20:08:46 odra kernel: [12501.627449] [<ffffffff811607dc>] block_ioctl+0x3c/0x40
Dec 30 20:08:46 odra kernel: [12501.627453] [<ffffffff811412f8>] do_vfs_ioctl+0x98/0x580
Dec 30 20:08:46 odra kernel: [12501.627456] [<ffffffff81101af9>] ? remove_vma+0x69/0x90
Dec 30 20:08:46 odra kernel: [12501.627460] [<ffffffff81103244>] ? do_munmap+0x2e4/0x360
Dec 30 20:08:46 odra kernel: [12501.627463] [<ffffffff81141861>] sys_ioctl+0x81/0xa0
Dec 30 20:08:46 odra kernel: [12501.627467] [<ffffffff81002d7b>] system_call_fastpath+0x16/0x1b
next prev parent reply other threads:[~2010-12-30 19:39 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-30 15:08 [2.6.37-rc8] BUG kmalloc-256: Poison overwritten Pawel Sikora
2010-12-30 15:31 ` Pekka Enberg
2010-12-30 15:59 ` Pekka Enberg
2010-12-30 19:39 ` Paweł Sikora [this message]
2010-12-30 23:00 ` Neil Brown
2010-12-31 8:02 ` Paweł Sikora
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201012302039.33305.pluto@agmk.net \
--to=pluto@agmk.net \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=neilb@suse.de \
--cc=penberg@cs.helsinki.fi \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.