From: Mike Snitzer <snitzer@redhat.com>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Jens Axboe <axboe@kernel.dk>,
hch@infradead.org, Keith Busch <keith.busch@intel.com>,
device-mapper development <dm-devel@redhat.com>,
j-nomura@ce.jp.nec.com
Subject: Re: [PATCH v3 0/8] dm: add request-based blk-mq support
Date: Fri, 19 Dec 2014 10:38:02 -0500 [thread overview]
Message-ID: <20141219153801.GA8697@redhat.com> (raw)
In-Reply-To: <54943714.2040704@acm.org>
On Fri, Dec 19 2014 at 9:32am -0500,
Bart Van Assche <bvanassche@acm.org> wrote:
> On 12/18/14 00:06, Mike Snitzer wrote:
> > So if you know someone with relevant blk-mq hardware who might benefit
> > from blk-mq multipathing please point them at this code and have them
> > report back!
>
> Hello Mike,
>
> Great to see that you are working on blk-mq multipathing. Unfortunately
> a test with the SRP initiator and your dm-for-3.20-blk-mq tree merged
> with Linus' latest tree was not successful. This is what was reported
> when I tried to start multipathd (without call trace, followed by a
> hard lockup):
>
> =========================================================
> [ INFO: possible irq lock inversion dependency detected ]
> 3.18.0-debug+ #1 Tainted: G W
> ---------------------------------------------------------
> kdmwork-253:0/5347 just changed the state of lock:
> (&(&m->lock)->rlock){+.....}, at: [<ffffffffa080eb80>] __multipath_map.isra.15+0x40/0x1f0 [dm_multipath]
> but this lock was taken by another, HARDIRQ-safe lock in the past:
> (&(&q->__queue_lock)->rlock){-.-...}
>
> and interrupts could create inverse lock ordering between them.
>
> other info that might help us debug this:
> Possible interrupt unsafe locking scenario:
This "dm: submit stacked requests in irq enabled context" commit
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-for-3.20-blk-mq&id=1844ba7e2e013fa38c45d646248c517eb363e26c
changed the locking needed in the multipath target. I altered
__multipath_map but didn't audit elsewhere. I'll work through it.
I rebuilt my kernel with lockdep enabled and can easily see this too:
[ 181.819735] =========================================================
[ 181.820046] [ INFO: possible irq lock inversion dependency detected ]
[ 181.820046] 3.18.0+ #12 Tainted: G W
[ 181.820046] ---------------------------------------------------------
[ 181.820046] swapper/1/0 just changed the state of lock:
[ 181.820046] (&(&q->__queue_lock)->rlock){..-...}, at: [<ffffffff812fb2c4>] blk_end_bidi_request+0x34/0x60
[ 181.820046] but this lock took another, SOFTIRQ-unsafe lock in the past:
[ 181.820046] (&(&m->lock)->rlock){+.+...}
and interrupts could create inverse lock ordering between them.
[ 181.820046]
[ 181.820046] other info that might help us debug this:
[ 181.820046] Possible interrupt unsafe locking scenario:
[ 181.820046]
[ 181.820046] CPU0 CPU1
[ 181.820046] ---- ----
[ 181.820046] lock(&(&m->lock)->rlock);
[ 181.820046] local_irq_disable();
[ 181.820046] lock(&(&q->__queue_lock)->rlock);
[ 181.820046] lock(&(&m->lock)->rlock);
[ 181.820046] <Interrupt>
[ 181.820046] lock(&(&q->__queue_lock)->rlock);
[ 181.820046]
[ 181.820046] *** DEADLOCK ***
[ 181.820046]
[ 181.820046] no locks held by swapper/1/0.
[ 181.820046]
[ 181.820046] the shortest dependencies between 2nd lock and 1st lock:
[ 181.820046] -> (&(&m->lock)->rlock){+.+...} ops: 4 {
[ 181.820046] HARDIRQ-ON-W at:
[ 181.820046] [<ffffffff810c8f46>] __lock_acquire+0x5d6/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694618>] _raw_spin_lock+0x38/0x50
[ 181.820046] [<ffffffffa00adbd0>] __multipath_map.isra.15+0x40/0x1f0 [dm_multipath]
[ 181.820046] [<ffffffffa00add9a>] multipath_clone_and_map+0x1a/0x20 [dm_multipath]
[ 181.820046] [<ffffffffa0206c25>] map_tio_request+0x1d5/0x2b0 [dm_mod]
[ 181.820046] [<ffffffff8109a4ee>] kthread_worker_fn+0x7e/0x1b0
[ 181.820046] [<ffffffff8109a3f7>] kthread+0x107/0x120
[ 181.820046] [<ffffffff8169527c>] ret_from_fork+0x7c/0xb0
[ 181.820046] SOFTIRQ-ON-W at:
[ 181.820046] [<ffffffff810c8ca0>] __lock_acquire+0x330/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694618>] _raw_spin_lock+0x38/0x50
[ 181.820046] [<ffffffffa00adbd0>] __multipath_map.isra.15+0x40/0x1f0 [dm_multipath]
[ 181.820046] [<ffffffffa00add9a>] multipath_clone_and_map+0x1a/0x20 [dm_multipath]
[ 181.820046] [<ffffffffa0206c25>] map_tio_request+0x1d5/0x2b0 [dm_mod]
[ 181.820046] [<ffffffff8109a4ee>] kthread_worker_fn+0x7e/0x1b0
[ 181.820046] [<ffffffff8109a3f7>] kthread+0x107/0x120
[ 181.820046] [<ffffffff8169527c>] ret_from_fork+0x7c/0xb0
[ 181.820046] INITIAL USE at:
[ 181.820046] [<ffffffff810c8d2f>] __lock_acquire+0x3bf/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694f50>] _raw_spin_lock_irqsave+0x50/0x70
[ 181.820046] [<ffffffffa00ac63c>] multipath_resume+0x1c/0x50 [dm_multipath]
[ 181.820046] [<ffffffffa020bed9>] dm_table_resume_targets+0x99/0xe0 [dm_mod]
[ 181.820046] [<ffffffffa0209289>] dm_resume+0xd9/0x120 [dm_mod]
[ 181.820046] [<ffffffffa020e7bb>] dev_suspend+0x12b/0x250 [dm_mod]
[ 181.820046] [<ffffffffa020f108>] ctl_ioctl+0x278/0x520 [dm_mod]
[ 181.820046] [<ffffffffa020f3c3>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
[ 181.820046] [<ffffffff81219488>] do_vfs_ioctl+0x318/0x560
[ 181.820046] [<ffffffff81219751>] SyS_ioctl+0x81/0xa0
[ 181.820046] [<ffffffff81695329>] system_call_fastpath+0x12/0x17
[ 181.820046] }
[ 181.820046] ... key at: [<ffffffffa00b04a0>] __key.33455+0x0/0xffffffffffffeb60 [dm_multipath]
[ 181.820046] ... acquired at:
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694618>] _raw_spin_lock+0x38/0x50
[ 181.820046] [<ffffffffa020893d>] dm_blk_open+0x1d/0x90 [dm_mod]
[ 181.820046] [<ffffffff812416fe>] __blkdev_get+0xde/0x4e0
[ 181.820046] [<ffffffff81241cf8>] blkdev_get+0x1f8/0x3b0
[ 181.820046] [<ffffffff81241f6f>] blkdev_open+0x5f/0x90
[ 181.820046] [<ffffffff812014cf>] do_dentry_open+0x1ff/0x350
[ 181.820046] [<ffffffff81201789>] vfs_open+0x49/0x50
[ 181.820046] [<ffffffff81211cd2>] do_last+0x682/0x13b0
[ 181.820046] [<ffffffff81214535>] path_openat+0xc5/0x640
[ 181.820046] [<ffffffff81216b69>] do_filp_open+0x49/0xc0
[ 181.820046] [<ffffffff812033a7>] do_sys_open+0x137/0x240
[ 181.820046] [<ffffffff812034ce>] SyS_open+0x1e/0x20
[ 181.820046] [<ffffffff81695329>] system_call_fastpath+0x12/0x17
[ 181.820046]
[ 181.820046] -> (&(&q->__queue_lock)->rlock){..-...} ops: 71 {
[ 181.820046] IN-SOFTIRQ-W at:
[ 181.820046] [<ffffffff810c8c25>] __lock_acquire+0x2b5/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694f50>] _raw_spin_lock_irqsave+0x50/0x70
[ 181.820046] [<ffffffff812fb2c4>] blk_end_bidi_request+0x34/0x60
[ 181.820046] [<ffffffff812fb3cf>] blk_end_request_all+0x1f/0x30
[ 181.820046] [<ffffffffa0206359>] dm_softirq_done+0xe9/0x1e0 [dm_mod]
[ 181.820046] [<ffffffff813023b0>] blk_done_softirq+0xa0/0xd0
[ 181.820046] [<ffffffff8107e091>] __do_softirq+0x141/0x370
[ 181.820046] [<ffffffff8107e655>] irq_exit+0x125/0x130
[ 181.820046] [<ffffffff8104bd05>] smp_call_function_single_interrupt+0x35/0x40
[ 181.820046] [<ffffffff81696822>] call_function_single_interrupt+0x72/0x80
[ 181.820046] [<ffffffff810c0084>] cpu_startup_entry+0x194/0x420
[ 181.820046] [<ffffffff8104c6dd>] start_secondary+0x19d/0x210
[ 181.820046] INITIAL USE at:
[ 181.820046] [<ffffffff810c8d2f>] __lock_acquire+0x3bf/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694764>] _raw_spin_lock_irq+0x44/0x60
[ 181.820046] [<ffffffff812f998d>] blk_queue_bypass_start+0x1d/0xb0
[ 181.820046] [<ffffffff813168c6>] blkcg_activate_policy+0x96/0x340
[ 181.820046] [<ffffffff8131a04e>] blk_throtl_init+0xee/0x130
[ 181.820046] [<ffffffff81316c01>] blkcg_init_queue+0x31/0x40
[ 181.820046] [<ffffffff812f6551>] blk_alloc_queue_node+0x251/0x2c0
[ 181.820046] [<ffffffff812fa854>] blk_init_queue_node+0x24/0x70
[ 181.820046] [<ffffffff812fa8b3>] blk_init_queue+0x13/0x20
[ 181.820046] [<ffffffffa0015364>] virtqueue_get_buf+0x14/0x130 [virtio_ring]
[ 181.820046] [<ffffffff81002144>] do_one_initcall+0xd4/0x210
[ 181.820046] [<ffffffff8110b4e2>] load_module+0x17d2/0x1c10
[ 181.820046] [<ffffffff8110baf6>] SyS_finit_module+0xa6/0xd0
[ 181.820046] [<ffffffff81695329>] system_call_fastpath+0x12/0x17
[ 181.820046] }
[ 181.820046] ... key at: [<ffffffff82c21ec0>] __key.42329+0x0/0x8
[ 181.820046] ... acquired at:
[ 181.820046] [<ffffffff810c7909>] check_usage_forwards+0x199/0x1b0
[ 181.820046] [<ffffffff810c81e1>] mark_lock+0x1a1/0x2a0
[ 181.820046] [<ffffffff810c8c25>] __lock_acquire+0x2b5/0x1d40
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff81694f50>] _raw_spin_lock_irqsave+0x50/0x70
[ 181.820046] [<ffffffff812fb2c4>] blk_end_bidi_request+0x34/0x60
[ 181.820046] [<ffffffff812fb3cf>] blk_end_request_all+0x1f/0x30
[ 181.820046] [<ffffffffa0206359>] dm_softirq_done+0xe9/0x1e0 [dm_mod]
[ 181.820046] [<ffffffff813023b0>] blk_done_softirq+0xa0/0xd0
[ 181.820046] [<ffffffff8107e091>] __do_softirq+0x141/0x370
[ 181.820046] [<ffffffff8107e655>] irq_exit+0x125/0x130
[ 181.820046] [<ffffffff8104bd05>] smp_call_function_single_interrupt+0x35/0x40
[ 181.820046] [<ffffffff81696822>] call_function_single_interrupt+0x72/0x80
[ 181.820046] [<ffffffff810c0084>] cpu_startup_entry+0x194/0x420
[ 181.820046] [<ffffffff8104c6dd>] start_secondary+0x19d/0x210
[ 181.820046]
[ 181.820046]
[ 181.820046] stack backtrace:
[ 181.820046] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 3.18.0+ #12
[ 181.820046] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 181.820046] 0000000000000000 79d40dcd9747394f ffff88011fc83b68 ffffffff8168b840
[ 181.820046] 0000000000000000 ffffffff828b2340 ffff88011fc83bb8 ffffffff81685d9f
[ 181.820046] ffff88011fc83be0 ffffffff818d58f4 ffff88011fc83bd4 0000000000000000
[ 181.820046] Call Trace:
[ 181.820046] <IRQ> [<ffffffff8168b840>] dump_stack+0x4c/0x65
[ 181.820046] [<ffffffff81685d9f>] print_irq_inversion_bug.part.37+0x1ae/0x1bd
[ 181.820046] [<ffffffff810c7909>] check_usage_forwards+0x199/0x1b0
[ 181.820046] [<ffffffff810c7770>] ? check_usage_backwards+0x1a0/0x1a0
[ 181.820046] [<ffffffff810c81e1>] mark_lock+0x1a1/0x2a0
[ 181.820046] [<ffffffff810c8c25>] __lock_acquire+0x2b5/0x1d40
[ 181.820046] [<ffffffff81689193>] ? __slab_free+0x11c/0x2b0
[ 181.820046] [<ffffffff810cae37>] lock_acquire+0xb7/0x140
[ 181.820046] [<ffffffff812fb2c4>] ? blk_end_bidi_request+0x34/0x60
[ 181.820046] [<ffffffff81694f50>] _raw_spin_lock_irqsave+0x50/0x70
[ 181.820046] [<ffffffff812fb2c4>] ? blk_end_bidi_request+0x34/0x60
[ 181.820046] [<ffffffff812fb2c4>] blk_end_bidi_request+0x34/0x60
[ 181.820046] [<ffffffff812fb3cf>] blk_end_request_all+0x1f/0x30
[ 181.820046] [<ffffffffa0206359>] dm_softirq_done+0xe9/0x1e0 [dm_mod]
[ 181.820046] [<ffffffff813023b0>] blk_done_softirq+0xa0/0xd0
[ 181.820046] [<ffffffff8107e091>] __do_softirq+0x141/0x370
[ 181.820046] [<ffffffff8107e655>] irq_exit+0x125/0x130
[ 181.820046] [<ffffffff8104bd05>] smp_call_function_single_interrupt+0x35/0x40
[ 181.820046] [<ffffffff81696822>] call_function_single_interrupt+0x72/0x80
[ 181.820046] <EOI> [<ffffffff810fecc9>] ? tick_nohz_idle_exit+0xc9/0x150
[ 181.820046] [<ffffffff810fecc5>] ? tick_nohz_idle_exit+0xc5/0x150
[ 181.820046] [<ffffffff810c0084>] cpu_startup_entry+0x194/0x420
[ 181.820046] [<ffffffff8104c6dd>] start_secondary+0x19d/0x210
next prev parent reply other threads:[~2014-12-19 15:38 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-17 3:59 [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 1/8] block: require blk_rq_prep_clone() be given an initialized clone request Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 2/8] block: initialize bio member of blk-mq request to NULL Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 3/8] block: add blk-mq support to blk_insert_cloned_request() Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 4/8] block: mark blk-mq devices as stackable Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 5/8] dm: remove exports for request-based interfaces without external callers Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 6/8] dm: split request structure out from dm_rq_target_io structure Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 7/8] dm: submit stacked requests in irq enabled context Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 8/8] dm: allocate requests from target when stacking on blk-mq devices Mike Snitzer
2014-12-17 22:35 ` Mike Snitzer
2014-12-17 21:42 ` [PATCH v3 0/8] dm: add request-based blk-mq support Keith Busch
2014-12-17 21:43 ` Jens Axboe
2014-12-17 23:06 ` Mike Snitzer
2014-12-18 1:41 ` Keith Busch
2014-12-18 4:58 ` Mike Snitzer
2014-12-19 14:32 ` Bart Van Assche
2014-12-19 15:38 ` Mike Snitzer [this message]
2014-12-19 17:14 ` Mike Snitzer
2014-12-22 15:28 ` Bart Van Assche
2014-12-22 18:49 ` Mike Snitzer
2014-12-23 16:24 ` Bart Van Assche
2014-12-23 17:13 ` Mike Snitzer
2014-12-23 21:42 ` Mike Snitzer
2014-12-24 13:02 ` Bart Van Assche
2014-12-24 18:21 ` Mike Snitzer
2014-12-24 18:55 ` Mike Snitzer
2014-12-24 19:26 ` Mike Snitzer
2015-01-02 17:53 ` Bart Van Assche
2015-01-05 21:35 ` Mike Snitzer
2015-01-06 8:59 ` Christoph Hellwig
2015-01-06 9:31 ` Bart Van Assche
2015-01-06 16:05 ` blk-mq request allocation stalls [was: Re: [PATCH v3 0/8] dm: add request-based blk-mq support] Mike Snitzer
2015-01-06 16:15 ` Jens Axboe
2015-01-07 10:33 ` Bart Van Assche
2015-01-07 15:32 ` Jens Axboe
2015-01-07 16:15 ` Mike Snitzer
2015-01-07 16:18 ` Jens Axboe
2015-01-07 16:22 ` Mike Snitzer
2015-01-07 16:24 ` Jens Axboe
2015-01-07 17:18 ` Mike Snitzer
2015-01-07 17:35 ` Jens Axboe
2015-01-07 20:09 ` Mike Snitzer
2015-01-07 20:40 ` Keith Busch
2015-01-09 19:49 ` Mike Snitzer
2015-01-09 21:07 ` Jens Axboe
2015-01-09 21:11 ` Jens Axboe
2015-01-09 21:40 ` Mike Snitzer
2015-01-09 21:56 ` Jens Axboe
2015-01-09 22:25 ` Mike Snitzer
2015-01-10 0:27 ` Jens Axboe
2015-01-10 1:48 ` Mike Snitzer
2015-01-10 1:59 ` Jens Axboe
2015-01-10 3:10 ` Mike Snitzer
2015-01-12 14:46 ` blk-mq request allocation stalls Bart Van Assche
2015-01-12 15:42 ` Jens Axboe
2015-01-12 16:12 ` Bart Van Assche
2015-01-12 16:34 ` Jens Axboe
2015-01-12 16:58 ` Mike Snitzer
2015-01-12 16:59 ` Jens Axboe
2015-01-12 17:04 ` Bart Van Assche
2015-01-12 17:09 ` Jens Axboe
2015-01-12 17:53 ` Keith Busch
2015-01-12 18:12 ` Jens Axboe
2015-01-12 18:22 ` Keith Busch
2015-01-12 18:35 ` Keith Busch
2015-01-12 19:11 ` Mike Snitzer
2015-01-12 20:21 ` Mike Snitzer
2015-01-13 12:29 ` Bart Van Assche
2015-01-13 14:17 ` Mike Snitzer
2015-01-13 14:28 ` dm + blk-mq soft lockup complaint Bart Van Assche
2015-01-13 16:20 ` Mike Snitzer
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 18:59 ` Mike Snitzer
2015-01-15 8:11 ` Bart Van Assche
2015-01-15 15:43 ` Mike Snitzer
2015-01-15 15:55 ` Bart Van Assche
2015-01-13 14:59 ` blk-mq request allocation stalls Jens Axboe
2015-01-13 15:11 ` Keith Busch
2015-01-13 15:27 ` Keith Busch
2015-01-13 15:41 ` Mike Snitzer
2015-01-13 15:14 ` Mike Snitzer
2015-01-27 18:42 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls] Mike Snitzer
2015-01-28 16:42 ` Jens Axboe
2015-01-28 17:44 ` Mike Snitzer
2015-01-28 17:49 ` Jens Axboe
2015-01-28 18:10 ` Mike Snitzer
2015-01-29 22:43 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls]X Keith Busch
2015-01-29 23:09 ` Mike Snitzer
2015-01-29 23:44 ` Keith Busch
2015-01-30 0:32 ` Mike Snitzer
2015-01-12 19:05 ` blk-mq request allocation stalls Jens Axboe
2015-01-12 19:07 ` Mike Snitzer
2015-01-12 18:19 ` Mike Snitzer
2014-12-17 22:51 ` [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141219153801.GA8697@redhat.com \
--to=snitzer@redhat.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=j-nomura@ce.jp.nec.com \
--cc=keith.busch@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox