* 4.4-rc2 crash: block related
@ 2015-11-25 6:04 Mika Penttilä
[not found] ` <565554EC.7050603@nextfour.com>
0 siblings, 1 reply; 3+ messages in thread
From: Mika Penttilä @ 2015-11-25 6:04 UTC (permalink / raw)
To: linux-kernel
Hi,
With recent block layer pull i see a 100% repeatable crash on boot while
mounting roots (ext4 partition on eMMC, with cfq io scheduler).
---
5.674294] Unable to handle kernel NULL pointer dereference at virtual
address 00000004
[ 5.682399] pgd = a8ca4000
[ 5.685113] [00000004] *pgd=38a5e831, *pte=00000000, *ppte=00000000
[ 5.691428] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[ 5.696830] Modules linked in: st_drv
[ 5.700533] CPU: 1 PID: 221 Comm: mount Not tainted 4.4.0-rc2 #49
[ 5.706631] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 5.713163] task: a88e2ac0 ti: a88d4000 task.ti: a88d4000
[ 5.718578] PC is at cfq_init_prio_data+0x8/0xec
[ 5.723206] LR is at cfq_insert_request+0x28/0x4f0
[ 5.723211] pc : [<8024bf9c>] lr : [<8024e768>] psr: 600d0093
[ 5.723211] sp : a88d5bc0 ip : 00000000 fp : a8ab5400
[ 5.723219] r10: 00000001 r9 : a617f4c0 r8 : 80b6359c
[ 5.723223] r7 : 80b62100 r6 : a873e200 r5 : a885ac30 r4 : 00000000
[ 5.723226] r3 : a88d5bc0 r2 : a89106c0 r1 : 00000000 r0 : 00000000
[ 5.723232] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM
Segment user
[ 5.723235] Control: 10c5387d Table: 38ca404a DAC: 00000055
[ 5.723239] Process mount (pid: 221, stack limit = 0xa88d4210)
[ 5.723242] Stack: (0xa88d5bc0 to 0xa88d6000)
[ 5.723251] 5bc0: 00000000 a885ac30 a873e200 8024e768 a87c0000
a885ac30 00000005 a88d4000
[ 5.723257] 5be0: 80b6359c a617f4c0 00000001 8023817c ffffffff
a89106c0 a885ac30 00000000
[ 5.723263] 5c00: a89106c0 ffffffff a87c0000 8023654c 00000000
00000000 a8ab5400 a89106c0
[ 5.723269] 5c20: 00000008 00001411 f0000000 80236680 a88d5c44
a87c0168 a617f4c0 a81a45c0
[ 5.723276] 5c40: 00000001 02400000 80b6359c a617f4c0 00000001
80231b04 a00d0013 0000000f
[ 5.723282] 5c60: a617f4c0 a89106c0 00001411 f0000000 80b6359c
a617f4c0 00000001 80110950
[ 5.723288] 5c80: a617f4c0 00000001 00001411 80b6370c 80b6359c
80112490 a8b35c00 00000000
[ 5.723295] 5ca0: 80b63658 801602e4 0205a9d9 00000000 a62b4738
a8ab5400 a8b35c00 a8b36000
[ 5.723301] 5cc0: 00000000 00000000 a8b36000 a8ab5400 a88d5e8c
80162644 a62621e8 800f7004
[ 5.723307] 5ce0: a88d5e8c 806dd610 a62621e8 a617f4c0 a8b35c00
a8b36000 00000001 80165480
[ 5.723313] 5d00: 00000000 00000000 a88d5d58 a88d5d50 a87f2a90
a88d5d54 01897158 800ec9dc
[ 5.723319] 5d20: 00000000 00000002 00000000 a88d5dc8 00000001
a88d5dc0 00000001 a6023000
[ 5.723325] 5d40: a88d5d90 a88d5d88 a8887f10 a88d5d8c 01897158
800ec8d0 a8887f10 00000004
[ 5.723332] 5d60: 00000000 a88d5dc0 a88d5dc0 a6029110 00000001
a80fd000 a88d5d8c a8744800
[ 5.723338] 5d80: 00000000 00000000 00000001 00000980 0000b67c
00000000 00000001 800bf478
[ 5.723343] 5da0: a615e490 00000001 0000006c a8102db0 00000000
00000001 0000000a 00000001
[ 5.723349] 5dc0: 0000ffff 00000000 00000000 00000000 0000002b
a82ec200 80b6e735 00000004
[ 5.723355] 5de0: 00000000 00000000 a8ab5400 00000000 a8b36264
00000000 001013d0 00000000
[ 5.723361] 5e00: 00000001 00000000 a8b36000 00000000 00001000
a8b35e88 00000000 00000000
[ 5.723366] 5e20: 00000000 00000000 a8ab5594 00000000 80be3e54
00000000 00000000 00000000
[ 5.723372] 5e40: 00000000 00004003 00000000 80b70288 01897158
8025e5bc a6298e00 a88d5e6c
[ 5.723378] 5e60: 3b9aca00 00000009 a6298e00 a6298e74 a8b35c00
a6298e00 00000083 00000000
[ 5.723384] 5e80: 00000000 80b70288 01897158 800e6324 a6298e00
800c0050 62636d6d 70306b6c
[ 5.723391] 5ea0: a8000035 800d0013 00000004 80be3e2c a8dca80e
00000000 00000001 8015f030
[ 5.723397] 5ec0: a8dca800 00000000 80b70288 80b70288 80b6aeb0
8015f048 801636d8 a8ab1a48
[ 5.723403] 5ee0: 01897158 800e6f14 00000000 a8dca800 a8ab19c0
a8dca800 00000000 80b70288
[ 5.723409] 5f00: 00000000 800febbc 00000000 00000020 00000000
a8dca800 a8dca840 80101a14
[ 5.723416] 5f20: 00000000 80b60be0 a8001f00 024000c0 000088c5
800df23c 007fffff a8dca800
[ 5.723421] 5f40: a87f2a90 a6138cc0 c0ed0000 a8dca800 0000000f
00000000 0000000f a8dca840
[ 5.723428] 5f60: a8dca800 00000000 018971a0 c0ed0000 a88d4000
00000000 01897158 801027e4
[ 5.723434] 5f80: 00000000 28936a1b 563c86d0 00000000 00000000
76f35688 c0ed0000 00000015
[ 5.723440] 5fa0: 8000f6a4 8000f500 00000000 76f35688 01897188
018971a0 01897158 c0ed0000
[ 5.723447] 5fc0: 00000000 76f35688 c0ed0000 00000015 018971a0
01897188 76f36dac 01897158
[ 5.723453] 5fe0: 76e56dc0 7eedcc30 76f09e70 76e56dd0 600d0010
01897188 00000000 00000000
[ 5.723473] [<8024bf9c>] (cfq_init_prio_data) from [<8024e768>]
(cfq_insert_request+0x28/0x4f0)
[ 5.723484] [<8024e768>] (cfq_insert_request) from [<8023817c>]
(blk_queue_bio+0x254/0x260)
[ 5.723500] [<8023817c>] (blk_queue_bio) from [<8023654c>]
(generic_make_request+0xcc/0x17c)
[ 5.723510] [<8023654c>] (generic_make_request) from [<80236680>]
[ 5.723527] [<80236680>] (submit_bio) from [<80110950>]
(submit_bh_wbc+0x10c/0x144)
[ 5.723537] [<80110950>] (submit_bh_wbc) from [<80112490>]
(__sync_dirty_buffer+0x90/0x114)
[ 5.723550] [<80112490>] (__sync_dirty_buffer) from [<801602e4>]
(ext4_commit_super+0x1c8/0x290)
[ 5.723561] [<801602e4>] (ext4_commit_super) from [<80162644>]
(ext4_setup_super+0xe8/0x1d0)
[ 5.723569] [<80162644>] (ext4_setup_super) from [<80165480>]
(ext4_fill_super+0x1da8/0x32b8)
[ 5.723580] [<80165480>] (ext4_fill_super) from [<800e6324>]
(mount_bdev+0x168/0x190)
[ 5.723588] [<800e6324>] (mount_bdev) from [<8015f048>]
(ext4_mount+0x18/0x20)
[ 5.723596] [<8015f048>] (ext4_mount) from [<800e6f14>]
(mount_fs+0x14/0xa4)
[ 5.723604] [<800e6f14>] (mount_fs) from [<800febbc>]
(vfs_kern_mount+0x4c/0xf4)
[ 5.723614] [<800febbc>] (vfs_kern_mount) from [<80101a14>]
(do_mount+0x1a0/0xc24)
[ 5.723622] [<80101a14>] (do_mount) from [<801027e4>]
(SyS_mount+0x74/0xa0)
[ 5.723635] [<801027e4>] (SyS_mount) from [<8000f500>]
(ret_fast_syscall+0x0/0x34)
[ 5.723642] Code: e3a00000 e8bd8070 e92d4070 e1a0300d (e5902004)
[ 5.723647] ---[ end trace 4087b61c23235e2b ]---
[ 5.723651] Kernel panic - not syncing: Fatal exception
---
Cause seems to be this commit :
commit 1b2ff19e6a957b1ef0f365ad331b608af80e932e
Author: Jan Kara <jack@suse.com>
Date: Thu Nov 12 14:25:52 2015 +0100
blk-flush: Queue through IO scheduler when flush not required
git show 1b2ff19e6a957b1ef0f365ad331b608af80e932e:
diff --git a/block/blk-flush.c b/block/blk-flush.c
index 9c423e5..c81d56e 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -422,7 +422,7 @@ void blk_insert_flush(struct request *rq)
if (q->mq_ops) {
blk_mq_insert_request(rq, false, false, true);
} else
- list_add_tail(&rq->queuelist, &q->queue_head);
+ q->elevator->type->ops.elevator_add_req_fn(q, rq);
return;
}
With this reverted it boots normally.
Thanks,
Mika
^ permalink raw reply related [flat|nested] 3+ messages in thread[parent not found: <565554EC.7050603@nextfour.com>]
* Re: Fwd: 4.4-rc2 crash: block related [not found] ` <565554EC.7050603@nextfour.com> @ 2015-11-25 8:39 ` Jan Kara 2015-11-25 16:30 ` Jens Axboe 0 siblings, 1 reply; 3+ messages in thread From: Jan Kara @ 2015-11-25 8:39 UTC (permalink / raw) To: Mika Penttilä; +Cc: jack, LKML, axboe, Jeff Moyer On Wed 25-11-15 08:27:56, Mika Penttilä wrote: > With recent block layer pull i see a 100% repeatable crash on boot while > mounting roots (ext4 partition on eMMC, with cfq io scheduler). Thanks for report! After some investigation I found out we allocate elevator specific data in __get_request() only for non-flush requests. And this is actually required since the flush machinery uses the space in struct request for something else. Doh. So my patch is just wrong and not easy to fix since at the time __get_request() is called we are not sure whether the flush machinery will be used in the end. Jens, please revert 1b2ff19e6a957b1ef0f365ad331b608af80e932e. Thanks! I'm somewhat surprised that you can reliably hit the race where flushing gets disabled for the device just while the request is in flight. But I guess during boot it makes some sense. Honza > 5.674294] Unable to handle kernel NULL pointer dereference at virtual > address 00000004 > [ 5.682399] pgd = a8ca4000 > [ 5.685113] [00000004] *pgd=38a5e831, *pte=00000000, *ppte=00000000 > [ 5.691428] Internal error: Oops: 17 [#1] PREEMPT SMP ARM > [ 5.696830] Modules linked in: st_drv > [ 5.700533] CPU: 1 PID: 221 Comm: mount Not tainted 4.4.0-rc2 #49 > [ 5.706631] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree) > [ 5.713163] task: a88e2ac0 ti: a88d4000 task.ti: a88d4000 > [ 5.718578] PC is at cfq_init_prio_data+0x8/0xec > [ 5.723206] LR is at cfq_insert_request+0x28/0x4f0 > [ 5.723211] pc : [<8024bf9c>] lr : [<8024e768>] psr: 600d0093 > [ 5.723211] sp : a88d5bc0 ip : 00000000 fp : a8ab5400 > [ 5.723219] r10: 00000001 r9 : a617f4c0 r8 : 80b6359c > [ 5.723223] r7 : 80b62100 r6 : a873e200 r5 : a885ac30 r4 : 00000000 > [ 5.723226] r3 : a88d5bc0 r2 : a89106c0 r1 : 00000000 r0 : 00000000 > [ 5.723232] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM > Segment user > [ 5.723235] Control: 10c5387d Table: 38ca404a DAC: 00000055 > [ 5.723239] Process mount (pid: 221, stack limit = 0xa88d4210) > [ 5.723242] Stack: (0xa88d5bc0 to 0xa88d6000) > [ 5.723251] 5bc0: 00000000 a885ac30 a873e200 8024e768 a87c0000 > a885ac30 00000005 a88d4000 > [ 5.723257] 5be0: 80b6359c a617f4c0 00000001 8023817c ffffffff > a89106c0 a885ac30 00000000 > [ 5.723263] 5c00: a89106c0 ffffffff a87c0000 8023654c 00000000 > 00000000 a8ab5400 a89106c0 > [ 5.723269] 5c20: 00000008 00001411 f0000000 80236680 a88d5c44 > a87c0168 a617f4c0 a81a45c0 > [ 5.723276] 5c40: 00000001 02400000 80b6359c a617f4c0 00000001 > 80231b04 a00d0013 0000000f > [ 5.723282] 5c60: a617f4c0 a89106c0 00001411 f0000000 80b6359c > a617f4c0 00000001 80110950 > [ 5.723288] 5c80: a617f4c0 00000001 00001411 80b6370c 80b6359c > 80112490 a8b35c00 00000000 > [ 5.723295] 5ca0: 80b63658 801602e4 0205a9d9 00000000 a62b4738 > a8ab5400 a8b35c00 a8b36000 > [ 5.723301] 5cc0: 00000000 00000000 a8b36000 a8ab5400 a88d5e8c > 80162644 a62621e8 800f7004 > [ 5.723307] 5ce0: a88d5e8c 806dd610 a62621e8 a617f4c0 a8b35c00 > a8b36000 00000001 80165480 > [ 5.723313] 5d00: 00000000 00000000 a88d5d58 a88d5d50 a87f2a90 > a88d5d54 01897158 800ec9dc > [ 5.723319] 5d20: 00000000 00000002 00000000 a88d5dc8 00000001 > a88d5dc0 00000001 a6023000 > [ 5.723325] 5d40: a88d5d90 a88d5d88 a8887f10 a88d5d8c 01897158 > 800ec8d0 a8887f10 00000004 > [ 5.723332] 5d60: 00000000 a88d5dc0 a88d5dc0 a6029110 00000001 > a80fd000 a88d5d8c a8744800 > [ 5.723338] 5d80: 00000000 00000000 00000001 00000980 0000b67c > 00000000 00000001 800bf478 > [ 5.723343] 5da0: a615e490 00000001 0000006c a8102db0 00000000 > 00000001 0000000a 00000001 > [ 5.723349] 5dc0: 0000ffff 00000000 00000000 00000000 0000002b > a82ec200 80b6e735 00000004 > [ 5.723355] 5de0: 00000000 00000000 a8ab5400 00000000 a8b36264 > 00000000 001013d0 00000000 > [ 5.723361] 5e00: 00000001 00000000 a8b36000 00000000 00001000 > a8b35e88 00000000 00000000 > [ 5.723366] 5e20: 00000000 00000000 a8ab5594 00000000 80be3e54 > 00000000 00000000 00000000 > [ 5.723372] 5e40: 00000000 00004003 00000000 80b70288 01897158 > 8025e5bc a6298e00 a88d5e6c > [ 5.723378] 5e60: 3b9aca00 00000009 a6298e00 a6298e74 a8b35c00 > a6298e00 00000083 00000000 > [ 5.723384] 5e80: 00000000 80b70288 01897158 800e6324 a6298e00 > 800c0050 62636d6d 70306b6c > [ 5.723391] 5ea0: a8000035 800d0013 00000004 80be3e2c a8dca80e > 00000000 00000001 8015f030 > [ 5.723397] 5ec0: a8dca800 00000000 80b70288 80b70288 80b6aeb0 > 8015f048 801636d8 a8ab1a48 > [ 5.723403] 5ee0: 01897158 800e6f14 00000000 a8dca800 a8ab19c0 > a8dca800 00000000 80b70288 > [ 5.723409] 5f00: 00000000 800febbc 00000000 00000020 00000000 > a8dca800 a8dca840 80101a14 > [ 5.723416] 5f20: 00000000 80b60be0 a8001f00 024000c0 000088c5 > 800df23c 007fffff a8dca800 > [ 5.723421] 5f40: a87f2a90 a6138cc0 c0ed0000 a8dca800 0000000f > 00000000 0000000f a8dca840 > [ 5.723428] 5f60: a8dca800 00000000 018971a0 c0ed0000 a88d4000 > 00000000 01897158 801027e4 > [ 5.723434] 5f80: 00000000 28936a1b 563c86d0 00000000 00000000 > 76f35688 c0ed0000 00000015 > [ 5.723440] 5fa0: 8000f6a4 8000f500 00000000 76f35688 01897188 > 018971a0 01897158 c0ed0000 > [ 5.723447] 5fc0: 00000000 76f35688 c0ed0000 00000015 018971a0 > 01897188 76f36dac 01897158 > [ 5.723453] 5fe0: 76e56dc0 7eedcc30 76f09e70 76e56dd0 600d0010 > 01897188 00000000 00000000 > [ 5.723473] [<8024bf9c>] (cfq_init_prio_data) from [<8024e768>] > (cfq_insert_request+0x28/0x4f0) > [ 5.723484] [<8024e768>] (cfq_insert_request) from [<8023817c>] > (blk_queue_bio+0x254/0x260) > [ 5.723500] [<8023817c>] (blk_queue_bio) from [<8023654c>] > (generic_make_request+0xcc/0x17c) > [ 5.723510] [<8023654c>] (generic_make_request) from [<80236680>] > [ 5.723527] [<80236680>] (submit_bio) from [<80110950>] > (submit_bh_wbc+0x10c/0x144) > [ 5.723537] [<80110950>] (submit_bh_wbc) from [<80112490>] > (__sync_dirty_buffer+0x90/0x114) > [ 5.723550] [<80112490>] (__sync_dirty_buffer) from [<801602e4>] > (ext4_commit_super+0x1c8/0x290) > [ 5.723561] [<801602e4>] (ext4_commit_super) from [<80162644>] > (ext4_setup_super+0xe8/0x1d0) > [ 5.723569] [<80162644>] (ext4_setup_super) from [<80165480>] > (ext4_fill_super+0x1da8/0x32b8) > [ 5.723580] [<80165480>] (ext4_fill_super) from [<800e6324>] > (mount_bdev+0x168/0x190) > [ 5.723588] [<800e6324>] (mount_bdev) from [<8015f048>] > (ext4_mount+0x18/0x20) > [ 5.723596] [<8015f048>] (ext4_mount) from [<800e6f14>] > (mount_fs+0x14/0xa4) > [ 5.723604] [<800e6f14>] (mount_fs) from [<800febbc>] > (vfs_kern_mount+0x4c/0xf4) > [ 5.723614] [<800febbc>] (vfs_kern_mount) from [<80101a14>] > (do_mount+0x1a0/0xc24) > [ 5.723622] [<80101a14>] (do_mount) from [<801027e4>] > (SyS_mount+0x74/0xa0) > [ 5.723635] [<801027e4>] (SyS_mount) from [<8000f500>] > (ret_fast_syscall+0x0/0x34) > [ 5.723642] Code: e3a00000 e8bd8070 e92d4070 e1a0300d (e5902004) > [ 5.723647] ---[ end trace 4087b61c23235e2b ]--- > [ 5.723651] Kernel panic - not syncing: Fatal exception > --- > > > Cause seems to be this commit : > > commit 1b2ff19e6a957b1ef0f365ad331b608af80e932e > Author: Jan Kara <jack@suse.com> > Date: Thu Nov 12 14:25:52 2015 +0100 > > blk-flush: Queue through IO scheduler when flush not required > > > > git show 1b2ff19e6a957b1ef0f365ad331b608af80e932e: > > diff --git a/block/blk-flush.c b/block/blk-flush.c > index 9c423e5..c81d56e 100644 > --- a/block/blk-flush.c > +++ b/block/blk-flush.c > @@ -422,7 +422,7 @@ void blk_insert_flush(struct request *rq) > if (q->mq_ops) { > blk_mq_insert_request(rq, false, false, true); > } else > - list_add_tail(&rq->queuelist, &q->queue_head); > + q->elevator->type->ops.elevator_add_req_fn(q, rq); > return; > } > > > > With this reverted it boots normally. > > Thanks, > Mika > > > -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Fwd: 4.4-rc2 crash: block related 2015-11-25 8:39 ` Fwd: " Jan Kara @ 2015-11-25 16:30 ` Jens Axboe 0 siblings, 0 replies; 3+ messages in thread From: Jens Axboe @ 2015-11-25 16:30 UTC (permalink / raw) To: Jan Kara, Mika Penttilä; +Cc: jack, LKML, Jeff Moyer On 11/25/2015 01:39 AM, Jan Kara wrote: > On Wed 25-11-15 08:27:56, Mika Penttilä wrote: >> With recent block layer pull i see a 100% repeatable crash on boot while >> mounting roots (ext4 partition on eMMC, with cfq io scheduler). > > Thanks for report! After some investigation I found out we allocate > elevator specific data in __get_request() only for non-flush requests. And > this is actually required since the flush machinery uses the space in > struct request for something else. Doh. So my patch is just wrong and not > easy to fix since at the time __get_request() is called we are not sure > whether the flush machinery will be used in the end. Jens, please revert > 1b2ff19e6a957b1ef0f365ad331b608af80e932e. Thanks! > > I'm somewhat surprised that you can reliably hit the race where flushing > gets disabled for the device just while the request is in flight. But I > guess during boot it makes some sense. Yeah, that's impressive. I'll revert it and get it sent up before -rc3 is cut. -- Jens Axboe ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-11-25 16:30 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-25 6:04 4.4-rc2 crash: block related Mika Penttilä
[not found] ` <565554EC.7050603@nextfour.com>
2015-11-25 8:39 ` Fwd: " Jan Kara
2015-11-25 16:30 ` Jens Axboe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox