From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
To: Jens Axboe <axboe@fb.com>
Cc: Jan Kara <jack@suse.cz>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
James Bottomley <James.Bottomley@HansenPartnership.com>,
Brian J King <bjking1@us.ibm.com>
Subject: Re: [BUG] kernel BUG at /.../block/cfq-iosched.c:3145!
Date: Thu, 10 Apr 2014 10:02:24 +1000 [thread overview]
Message-ID: <1397088144.3671.155.camel@pasglop> (raw)
In-Reply-To: <1397087273.3671.150.camel@pasglop>
On Thu, 2014-04-10 at 09:47 +1000, Benjamin Herrenschmidt wrote:
> > Nope, that's not a known issue. This must be related to the FIFO
> > changes... How reproducible is this?
>
> Dunno yet, haven't had a chance to dig. I'll try to grab that machine
> later today and reproduce &| bisect.
Second boot leads to a slightly different symptom (see below). I'll
try bisecting a bit later today if I get a chance.
=============================================================================
BUG blkdev_requests (Not tainted): Poison overwritten
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: 0xc000003ca627f790-0xc000003ca627f797. First byte 0xc0 instead of 0x6b
INFO: Allocated in .mempool_alloc_slab+0x1c/0x30 age=17 cpu=36 pid=3579
.kmem_cache_alloc+0xc0/0x1ac
.mempool_alloc_slab+0x1c/0x30
.mempool_alloc+0x98/0x18c
.get_request+0x248/0x490
.blk_queue_bio+0x174/0x25c
.generic_make_request+0xa8/0xec
.submit_bio+0x134/0x14c
.mpage_readpages+0x108/0x118
.ext4_readpages+0x48/0x5c
.__do_page_cache_readahead+0x1b0/0x298
.ra_submit+0x28/0x38
.filemap_fault+0x158/0x3e4
.__do_fault+0x48/0xbc
.do_read_fault.isra.65+0x2c/0xe4
.handle_mm_fault+0x460/0xbb8
.do_page_fault+0x498/0x7e0
INFO: Freed in .mempool_free_slab+0x1c/0x30 age=17 cpu=36 pid=0
.kmem_cache_free+0x1d4/0x1fc
.mempool_free_slab+0x1c/0x30
.mempool_free+0xb0/0xbc
.__blk_put_request+0xd0/0x104
.blk_end_bidi_request+0x40/0x64
.scsi_io_completion+0x198/0x5a4
.scsi_finish_command+0xd0/0xdc
.scsi_softirq_done+0x128/0x134
.blk_done_softirq+0xa0/0xb8
.__do_softirq+0x1b8/0x370
.irq_exit+0x74/0xa4
.__do_irq+0x90/0xa4
.call_do_irq+0x14/0x24
.do_IRQ+0x80/0xc0
hardware_interrupt_common+0x138/0x180
0xc000000001e46640
INFO: Slab 0xf00000000d445888 objects=92 used=92 fp=0x (null) flags=0x13fff8000000080
INFO: Object 0xc000003ca627f788 @offset=63368 fp=0xc000003ca6270de8
Bytes b4 c000003ca627f778: 00 00 00 00 ff fe ef be 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
Object c000003ca627f788: 6b 6b 6b 6b 6b 6b 6b 6b c0 00 00 3c a6 27 fa 50 kkkkkkkk...<.'.P
Object c000003ca627f798: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f7f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f808: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f818: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f828: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f838: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f848: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f858: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f868: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f878: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f888: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f898: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object c000003ca627f8f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone c000003ca627f908: bb bb bb bb bb bb bb bb ........
Padding c000003ca627fa48: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
CPU: 60 PID: 3579 Comm: X Tainted: G B 3.14.0-test #1
Call Trace:
[c000003ca331aaf0] [c00000000001405c] .show_stack+0x50/0x14c (unreliable)
[c000003ca331abc0] [c0000000008a3a94] .dump_stack+0x90/0xc0
[c000003ca331ac40] [c00000000014bbec] .print_trailer+0x17c/0x188
[c000003ca331acd0] [c00000000014bd08] .check_bytes_and_report+0xc0/0x114
[c000003ca331ad80] [c00000000014be44] .check_object+0xe8/0x240
[c000003ca331ae20] [c0000000008a2488] .alloc_debug_processing+0x184/0x19c
[c000003ca331aeb0] [c0000000008a2958] .__slab_alloc+0x4b8/0x540
[c000003ca331b020] [c00000000014de68] .kmem_cache_alloc+0xc0/0x1ac
[c000003ca331b0d0] [c000000000107f3c] .mempool_alloc_slab+0x1c/0x30
[c000003ca331b140] [c0000000001080d4] .mempool_alloc+0x98/0x18c
[c000003ca331b220] [c0000000003227c4] .get_request+0x248/0x490
[c000003ca331b340] [c000000000324a68] .blk_queue_bio+0x174/0x25c
[c000003ca331b3e0] [c000000000322280] .generic_make_request+0xa8/0xec
[c000003ca331b470] [c0000000003223f8] .submit_bio+0x134/0x14c
[c000003ca331b530] [c000000000186030] ._submit_bh+0x244/0x278
[c000003ca331b5c0] [c000000000186b14] .ll_rw_block+0xd4/0x100
[c000003ca331b670] [c000000000188d80] .__breadahead+0x2c/0x4c
[c000003ca331b6f0] [c00000000021d18c] .__ext4_get_inode_loc+0x390/0x444
[c000003ca331b7c0] [c00000000021f7cc] .ext4_iget+0x4c/0x97c
[c000003ca331b8c0] [c000000000229558] .ext4_lookup+0xc4/0x15c
[c000003ca331b960] [c00000000016007c] .lookup_real+0x4c/0x74
[c000003ca331b9e0] [c000000000165bdc] .do_last+0x58c/0xc54
[c000003ca331bb10] [c00000000016650c] .path_openat+0x268/0x6a0
[c000003ca331bc30] [c000000000166cc0] .do_filp_open+0x34/0x80
[c000003ca331bd70] [c000000000155a10] .do_sys_open+0x1a4/0x250
[c000003ca331be30] [c00000000000a024] syscall_exit+0x0/0x98
FIX blkdev_requests: Restoring 0xc000003ca627f790-0xc000003ca627f797=0x6b
FIX blkdev_requests: Marking all objects used
------------[ cut here ]------------
kernel BUG at /home/benh/linux-powerpc-test/block/elevator.c:262!
cpu 0x34: Vector: 700 (Program Check) at [c000001febbd32c0]
pc: c00000000031ee40: .elv_rqhash_add+0x10/0x8c
lr: c0000000003202c8: .__elv_add_request+0x264/0x27c
sp: c000001febbd3540
msr: 9000000000029032
current = 0xc000003ca620c320
paca = 0xc00000000ffed000 softe: 0 irq_happened: 0x01
pid = 3565, comm = in:imjournal
kernel BUG at /home/benh/linux-powerpc-test/block/elevator.c:262!
enter ? for help
[link register ] c0000000003202c8 .__elv_add_request+0x264/0x27c
[c000001febbd3540] c0000000003201a0 .__elv_add_request+0x13c/0x27c (unreliable)
[c000001febbd35e0] c000000000324858 .blk_flush_plug_list+0x1f4/0x254
[c000001febbd36a0] c0000000003248d0 .blk_finish_plug+0x18/0x3c
[c000001febbd3720] c000000000111a84 .__do_page_cache_readahead+0x268/0x298
[c000001febbd3870] c000000000111d24 .ra_submit+0x28/0x38
[c000001febbd38e0] c000000000106f54 .filemap_fault+0x158/0x3e4
[c000001febbd39b0] c000000000126cfc .__do_fault+0x48/0xbc
[c000001febbd3a60] c0000000001272b0 .do_read_fault.isra.65+0x2c/0xe4
[c000001febbd3b20] c00000000012a718 .handle_mm_fault+0x460/0xbb8
[c000001febbd3c00] c00000000002ff54 .do_page_fault+0x498/0x7e0
[c000001febbd3e30] c0000000000094e8 handle_page_fault+0x10/0x30
--- Exception: 301 (Data Access) at 00003fff9a40d2d4
SP (3fff9585df30) is in userspace
34:mon>
next prev parent reply other threads:[~2014-04-10 0:02 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-09 4:44 [BUG] kernel BUG at /.../block/cfq-iosched.c:3145! Benjamin Herrenschmidt
2014-04-09 13:35 ` Jens Axboe
2014-04-09 23:47 ` Benjamin Herrenschmidt
2014-04-10 0:02 ` Benjamin Herrenschmidt [this message]
2014-04-10 1:31 ` Jens Axboe
2014-04-10 1:36 ` Benjamin Herrenschmidt
2014-04-10 1:38 ` Jens Axboe
2014-04-10 2:25 ` Jens Axboe
2014-04-10 3:35 ` Benjamin Herrenschmidt
2014-04-10 3:52 ` Jens Axboe
2014-04-10 6:45 ` Benjamin Herrenschmidt
2014-04-10 13:47 ` Jens Axboe
2014-04-10 18:10 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1397088144.3671.155.camel@pasglop \
--to=benh@kernel.crashing.org \
--cc=James.Bottomley@HansenPartnership.com \
--cc=axboe@fb.com \
--cc=bjking1@us.ibm.com \
--cc=fweisbec@gmail.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.