From: Mike Snitzer <snitzer@redhat.com>
To: Jens Axboe <axboe@kernel.dk>
Cc: Keith Busch <keith.busch@intel.com>,
Bart Van Assche <bart.vanassche@sandisk.com>,
device-mapper development <dm-devel@redhat.com>,
Jun'ichi Nomura <j-nomura@ce.jp.nec.com>,
Christoph Hellwig <hch@infradead.org>
Subject: Re: blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls]
Date: Wed, 28 Jan 2015 12:44:43 -0500 [thread overview]
Message-ID: <20150128174442.GA11951@redhat.com> (raw)
In-Reply-To: <54C91172.10002@kernel.dk>
On Wed, Jan 28 2015 at 11:42am -0500,
Jens Axboe <axboe@kernel.dk> wrote:
> On 01/27/2015 11:42 AM, Mike Snitzer wrote:
> >Hey Jens,
> >
> >I _think_ we've resolved the issues Bart raised for request-based DM's
> >support for blk-mq devices (anything remaining seems specific to iSER's
> >blk-mq support which is in development). Though Keith did have that one
> >additional patch for that block scatter gather attribute that we still
> >need to review closer.
> >
> >Anyway, I think what we have is a solid start and see no reason to hold
> >these changes back further. So I've rebased the 'dm-for-3.20' branch of
> >linux-dm.git ontop of 3.19-rc6 and reordered the required block changes
> >to be at the front of the series, see:
> >https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-for-3.20
> >
> >(these changes have been in Linux next for a month, via linux-dm.git
> >'for-next')
> >
> >With your OK, I'd be happy to carry the required block changes and
> >ultimately request Linus pull them for 3.20 (I can backfill your Acks if
> >you approve). BUT I also have no problem with you picking up the block
> >changes to submit via your block tree (I'd just have to rebase ontop of
> >your 3.20 branch once you pull them in).
>
> I'd prefer to take these prep patches through the block tree.
Great, should I send the patches or can you cherry-pick?
> Only one I don't really like is this one:
>
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20&id=23556c2461407495099d1eb20b0de43432dc727d
>
> I prefer keeping the alloc path as lean as possible, normal allocs
> always initialize ->bio since they need to associate a bio with it.
Would be very surprised if this initialization were measurable but..
I could push this initialization into the DM-mpath driver (just after
blk_get_request, like Keith opted for) but that seemed really gross.
> Do you have the oops trace from this one? Just curious if we can get
> rid of it, depending on how deep in the caller this is.
I did't but it was easy enough to recreate:
[ 3.112949] BUG: unable to handle kernel NULL pointer dereference at (null) |
[ 3.113416] IP: [<ffffffff812f6734>] blk_rq_prep_clone+0x44/0x160 |
[ 3.113416] PGD 0 |
[ 3.113416] Oops: 0002 [#1] SMP |
[ 3.113416] Modules linked in: dm_service_time crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper crypt|
d serio_raw pcspkr virtio_balloon 8139too i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath dm_mod ext4 mbcache jbd2 sd_mod ata_generic cirrus pata_ac|
pi syscopyarea sysfillrect sysimgblt drm_kms_helper ttm drm virtio_scsi virtio_blk 8139cp virtio_pci mii i2c_core virtio_ring ata_piix virtio libata floppy |
[ 3.113416] CPU: 0 PID: 483 Comm: kdmwork-252:3 Tainted: G W 3.18.0+ #29 |
[ 3.113416] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 |
[ 3.113416] task: ffff880035c1ad20 ti: ffff8800d6900000 task.ti: ffff8800d6900000 |
[ 3.113416] RIP: 0010:[<ffffffff812f6734>] [<ffffffff812f6734>] blk_rq_prep_clone+0x44/0x160 |
[ 3.113416] RSP: 0000:ffff8800d6903d48 EFLAGS: 00010286 |
[ 3.113416] RAX: 0000000000000000 RBX: ffffffffa0208500 RCX: 0000000000000001 |
[ 3.113416] RDX: ffff8800d7a3b0a0 RSI: ffff880035d0ab00 RDI: ffff880119f8f510 |
[ 3.113416] RBP: ffff8800d6903d98 R08: 00000000000185a0 R09: 00000000000000d0 |
[ 3.113416] R10: ffff8800d7547680 R11: ffff880035c1b8c8 R12: ffff8800d83d7900
[ 3.113416] R13: ffff880035d0ab00 R14: ffff880119f8f510 R15: ffff8800d7547680
[ 3.113416] FS: 0000000000000000(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[ 3.113416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3.113416] CR2: 0000000000000000 CR3: 00000000daeec000 CR4: 00000000000407f0
[ 3.113416] Stack:
[ 3.113416] ffff8800d6903db8 ffff8800d71502e0 ffff8800d7a3b0a0 000000d0d71502e0
[ 3.113416] ffff8800d6e89800 ffff8800d71502e0 ffff8800d6e89800 0000000000000001
[ 3.113416] ffff8800d7a3b0a0 ffffc90000998040 ffff8800d6903df8 ffffffffa0209c69
[ 3.113416] Call Trace:
[ 3.113416] [<ffffffffa0209c69>] map_tio_request+0x219/0x2b0 [dm_mod]
[ 3.113416] [<ffffffff8109a4ee>] kthread_worker_fn+0x7e/0x1b0
[ 3.113416] [<ffffffff8109a470>] ? __init_kthread_worker+0x60/0x60
[ 3.113416] [<ffffffff8109a3f7>] kthread+0x107/0x120
[ 3.113416] [<ffffffff8109a2f0>] ? kthread_create_on_node+0x240/0x240
[ 3.113416] [<ffffffff816952bc>] ret_from_fork+0x7c/0xb0
[ 3.113416] [<ffffffff8109a2f0>] ? kthread_create_on_node+0x240/0x240
[ 3.113416] Code: 89 c3 48 83 ec 28 4c 8b 6e 68 48 85 d2 4c 0f 44 25 22 b7 92 01 48 89 75 b8 89 4d cc 4c 89 4d c0 4d 85 ed 75 16 eb 60 49 8b 47 70 <4c> 89 30 4d 89
77 70 4d 8b 6d 00 4d 85 ed 74 4c 8b 75 cc 4c 89
[ 3.113416] RIP [<ffffffff812f6734>] blk_rq_prep_clone+0x44/0x160
[ 3.113416] RSP <ffff8800d6903d48>
[ 3.113416] CR2: 0000000000000000
[ 3.113416] ---[ end trace 9b3bb6dd6cc4435d ]---
crash> dis -l blk_rq_prep_clone+0x44
/home/snitm/git/linux/block/blk-core.c: 2945
0xffffffff812f6734 <blk_rq_prep_clone+0x44>: mov %r14,(%rax)
crash> l /home/snitm/git/linux/block/blk-core.c: 2945
2940
2941 if (bio_ctr && bio_ctr(bio, bio_src, data))
2942 goto free_and_out;
2943
2944 if (rq->bio) {
2945 rq->biotail->bi_next = bio;
2946 rq->biotail = bio;
2947 } else
2948 rq->bio = rq->biotail = bio;
2949 }
Given it would seem the NULL pointer occurs when attempting to
dereference rq->biotail a revised check of "if (rq->bio && rq->biotail)"
should suffice but I unfortunately then get:
[ 2.801634] general protection fault: 0000 [#1] SMP
[ 2.802504] Modules linked in: dm_service_time crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper crypt
d pcspkr serio_raw 8139too virtio_balloon i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath dm_mod ext4 mbcache jbd2 ata_generic sd_mod pata_acpi cirr
us syscopyarea sysfillrect sysimgblt drm_kms_helper ttm virtio_scsi virtio_blk drm virtio_pci virtio_ring ata_piix 8139cp libata mii i2c_core virtio floppy
[ 2.802504] CPU: 0 PID: 474 Comm: kdmwork-252:1 Tainted: G W 3.18.0+ #30
[ 2.802504] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 2.802504] task: ffff8801194b1690 ti: ffff880119abc000 task.ti: ffff880119abc000
[ 2.802504] RIP: 0010:[<ffffffff812f6739>] [<ffffffff812f6739>] blk_rq_prep_clone+0x49/0x160
[ 2.802504] RSP: 0018:ffff880119abfd48 EFLAGS: 00010206
[ 2.802504] RAX: 6de900000000e800 RBX: ffffffffa0218500 RCX: 0000000000000001
[ 2.802504] RDX: ffff8800daca30a0 RSI: ffff880119dcaf00 RDI: ffff880119dca310
[ 2.802504] RBP: ffff880119abfd98 R08: 00000000000185a0 R09: 00000000000000d0
[ 2.802504] R10: ffff880035937680 R11: ffff8801194b2238 R12: ffff880035876900
[ 2.802504] R13: ffff880119dcaf00 R14: ffff880119dca310 R15: ffff880035937680
[ 2.802504] FS: 0000000000000000(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
[ 2.802504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.802504] CR2: 00007f45598c2350 CR3: 000000003614a000 CR4: 00000000000407f0
[ 2.802504] Stack:
[ 2.802504] ffff880119abfdb8 ffff8800dae602e0 ffff8800daca30a0 000000d0dae602e0
[ 2.802504] ffff8800dad6a000 ffff8800dae602e0 ffff8800dad6a000 0000000000000001
[ 2.802504] ffff8800daca30a0 ffffc9000097d040 ffff880119abfdf8 ffffffffa0219c69
[ 2.802504] Call Trace:
[ 2.802504] [<ffffffffa0219c69>] map_tio_request+0x219/0x2b0 [dm_mod]
[ 2.802504] [<ffffffff8109a4ee>] kthread_worker_fn+0x7e/0x1b0
[ 2.802504] [<ffffffff8109a470>] ? __init_kthread_worker+0x60/0x60
[ 2.802504] [<ffffffff8109a3f7>] kthread+0x107/0x120
[ 2.802504] [<ffffffff8109a2f0>] ? kthread_create_on_node+0x240/0x240
[ 2.802504] [<ffffffff816952bc>] ret_from_fork+0x7c/0xb0
[ 2.802504] [<ffffffff8109a2f0>] ? kthread_create_on_node+0x240/0x240
[ 2.802504] Code: 28 4c 8b 6e 68 48 85 d2 4c 0f 44 25 22 b7 92 01 48 89 75 b8 89 4d cc 4c 89 4d c0 4d 85 ed 75 1b eb 64 49 8b 47 70 48 85 c0 74 4a <4c> 89 30 4d 89
77 70 4d 8b 6d 00 4d 85 ed 74 4b 8b 75 cc 4c 89
[ 2.802504] RIP [<ffffffff812f6739>] blk_rq_prep_clone+0x49/0x160
[ 2.802504] RSP <ffff880119abfd48>
[ 2.802386] general protection fault: 0000 [#2] [ 2.893050] ---[ end trace 20d230269dc05eca ]---
Not sure what to make of this (other than rq->biotail is pointing at
crap too, which is actually likely if rq->bio is):
crash> dis -l blk_rq_prep_clone+0x49
/home/snitm/git/linux/block/blk-core.c: 2945
0xffffffff812f6739 <blk_rq_prep_clone+0x49>: mov %r14,(%rax)
crash> l /home/snitm/git/linux/block/blk-core.c: 2945
2940
2941 if (bio_ctr && bio_ctr(bio, bio_src, data))
2942 goto free_and_out;
2943
2944 if (rq->bio && rq->biotail) {
2945 rq->biotail->bi_next = bio;
2946 rq->biotail = bio;
2947 } else
2948 rq->bio = rq->biotail = bio;
2949 }
next prev parent reply other threads:[~2015-01-28 17:44 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-17 3:59 [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 1/8] block: require blk_rq_prep_clone() be given an initialized clone request Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 2/8] block: initialize bio member of blk-mq request to NULL Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 3/8] block: add blk-mq support to blk_insert_cloned_request() Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 4/8] block: mark blk-mq devices as stackable Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 5/8] dm: remove exports for request-based interfaces without external callers Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 6/8] dm: split request structure out from dm_rq_target_io structure Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 7/8] dm: submit stacked requests in irq enabled context Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 8/8] dm: allocate requests from target when stacking on blk-mq devices Mike Snitzer
2014-12-17 22:35 ` Mike Snitzer
2014-12-17 21:42 ` [PATCH v3 0/8] dm: add request-based blk-mq support Keith Busch
2014-12-17 21:43 ` Jens Axboe
2014-12-17 23:06 ` Mike Snitzer
2014-12-18 1:41 ` Keith Busch
2014-12-18 4:58 ` Mike Snitzer
2014-12-19 14:32 ` Bart Van Assche
2014-12-19 15:38 ` Mike Snitzer
2014-12-19 17:14 ` Mike Snitzer
2014-12-22 15:28 ` Bart Van Assche
2014-12-22 18:49 ` Mike Snitzer
2014-12-23 16:24 ` Bart Van Assche
2014-12-23 17:13 ` Mike Snitzer
2014-12-23 21:42 ` Mike Snitzer
2014-12-24 13:02 ` Bart Van Assche
2014-12-24 18:21 ` Mike Snitzer
2014-12-24 18:55 ` Mike Snitzer
2014-12-24 19:26 ` Mike Snitzer
2015-01-02 17:53 ` Bart Van Assche
2015-01-05 21:35 ` Mike Snitzer
2015-01-06 8:59 ` Christoph Hellwig
2015-01-06 9:31 ` Bart Van Assche
2015-01-06 16:05 ` blk-mq request allocation stalls [was: Re: [PATCH v3 0/8] dm: add request-based blk-mq support] Mike Snitzer
2015-01-06 16:15 ` Jens Axboe
2015-01-07 10:33 ` Bart Van Assche
2015-01-07 15:32 ` Jens Axboe
2015-01-07 16:15 ` Mike Snitzer
2015-01-07 16:18 ` Jens Axboe
2015-01-07 16:22 ` Mike Snitzer
2015-01-07 16:24 ` Jens Axboe
2015-01-07 17:18 ` Mike Snitzer
2015-01-07 17:35 ` Jens Axboe
2015-01-07 20:09 ` Mike Snitzer
2015-01-07 20:40 ` Keith Busch
2015-01-09 19:49 ` Mike Snitzer
2015-01-09 21:07 ` Jens Axboe
2015-01-09 21:11 ` Jens Axboe
2015-01-09 21:40 ` Mike Snitzer
2015-01-09 21:56 ` Jens Axboe
2015-01-09 22:25 ` Mike Snitzer
2015-01-10 0:27 ` Jens Axboe
2015-01-10 1:48 ` Mike Snitzer
2015-01-10 1:59 ` Jens Axboe
2015-01-10 3:10 ` Mike Snitzer
2015-01-12 14:46 ` blk-mq request allocation stalls Bart Van Assche
2015-01-12 15:42 ` Jens Axboe
2015-01-12 16:12 ` Bart Van Assche
2015-01-12 16:34 ` Jens Axboe
2015-01-12 16:58 ` Mike Snitzer
2015-01-12 16:59 ` Jens Axboe
2015-01-12 17:04 ` Bart Van Assche
2015-01-12 17:09 ` Jens Axboe
2015-01-12 17:53 ` Keith Busch
2015-01-12 18:12 ` Jens Axboe
2015-01-12 18:22 ` Keith Busch
2015-01-12 18:35 ` Keith Busch
2015-01-12 19:11 ` Mike Snitzer
2015-01-12 20:21 ` Mike Snitzer
2015-01-13 12:29 ` Bart Van Assche
2015-01-13 14:17 ` Mike Snitzer
2015-01-13 14:28 ` dm + blk-mq soft lockup complaint Bart Van Assche
2015-01-13 16:20 ` Mike Snitzer
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 18:59 ` Mike Snitzer
2015-01-15 8:11 ` Bart Van Assche
2015-01-15 15:43 ` Mike Snitzer
2015-01-15 15:55 ` Bart Van Assche
2015-01-13 14:59 ` blk-mq request allocation stalls Jens Axboe
2015-01-13 15:11 ` Keith Busch
2015-01-13 15:27 ` Keith Busch
2015-01-13 15:41 ` Mike Snitzer
2015-01-13 15:14 ` Mike Snitzer
2015-01-27 18:42 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls] Mike Snitzer
2015-01-28 16:42 ` Jens Axboe
2015-01-28 17:44 ` Mike Snitzer [this message]
2015-01-28 17:49 ` Jens Axboe
2015-01-28 18:10 ` Mike Snitzer
2015-01-29 22:43 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls]X Keith Busch
2015-01-29 23:09 ` Mike Snitzer
2015-01-29 23:44 ` Keith Busch
2015-01-30 0:32 ` Mike Snitzer
2015-01-12 19:05 ` blk-mq request allocation stalls Jens Axboe
2015-01-12 19:07 ` Mike Snitzer
2015-01-12 18:19 ` Mike Snitzer
2014-12-17 22:51 ` [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150128174442.GA11951@redhat.com \
--to=snitzer@redhat.com \
--cc=axboe@kernel.dk \
--cc=bart.vanassche@sandisk.com \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=j-nomura@ce.jp.nec.com \
--cc=keith.busch@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.