From: Bart Van Assche <bart.vanassche@sandisk.com>
To: Mike Snitzer <snitzer@redhat.com>
Cc: Keith Busch <keith.busch@intel.com>, Jens Axboe <axboe@kernel.dk>,
device-mapper development <dm-devel@redhat.com>,
Jun'ichi Nomura <j-nomura@ce.jp.nec.com>,
Christoph Hellwig <hch@infradead.org>
Subject: Re: dm + blk-mq soft lockup complaint
Date: Tue, 13 Jan 2015 15:28:03 +0100 [thread overview]
Message-ID: <54B52B73.7040807@sandisk.com> (raw)
In-Reply-To: <20150113141746.GA30411@redhat.com>
On 01/13/15 15:18, Mike Snitzer wrote:
> On Tue, Jan 13 2015 at 7:29am -0500,
> Bart Van Assche <bart.vanassche@sandisk.com> wrote:
>> However, I hit another issue while running I/O on top of a multipath
>> device (on a kernel with lockdep and SLUB memory poisoning enabled):
>>
>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kdmwork-253:0:3116]
>> CPU: 7 PID: 3116 Comm: kdmwork-253:0 Tainted: G W 3.19.0-rc4-debug+ #1
>> Call Trace:
>> [<ffffffff8118e4be>] kmem_cache_alloc+0x28e/0x2c0
>> [<ffffffff81346aca>] alloc_iova_mem+0x1a/0x20
>> [<ffffffff81342c8e>] alloc_iova+0x2e/0x250
>> [<ffffffff81344b65>] intel_alloc_iova+0x95/0xd0
>> [<ffffffff81348a15>] intel_map_sg+0xc5/0x260
>> [<ffffffffa07e0661>] srp_queuecommand+0xa11/0xc30 [ib_srp]
>> [<ffffffffa001698e>] scsi_dispatch_cmd+0xde/0x5a0 [scsi_mod]
>> [<ffffffffa0017480>] scsi_queue_rq+0x630/0x700 [scsi_mod]
>> [<ffffffff8125683d>] __blk_mq_run_hw_queue+0x1dd/0x370
>> [<ffffffff81256aae>] blk_mq_alloc_request+0xde/0x150
>> [<ffffffff8124bade>] blk_get_request+0x2e/0xe0
>> [<ffffffffa07ebd0f>] __multipath_map.isra.15+0x1cf/0x210 [dm_multipath]
>> [<ffffffffa07ebd6a>] multipath_clone_and_map+0x1a/0x20 [dm_multipath]
>> [<ffffffffa044abb5>] map_tio_request+0x1d5/0x3a0 [dm_mod]
>> [<ffffffff81075d16>] kthread_worker_fn+0x86/0x1b0
>> [<ffffffff81075c0f>] kthread+0xef/0x110
>> [<ffffffff814db42c>] ret_from_fork+0x7c/0xb0
>
> Unfortunate. Is this still with a 16MB backing device or is it real
> hardware? Can you share the workload so that myself and/or Keith could
> try to reproduce?
Hello Mike,
This is still with a 16MB RAM disk as backing device. The fio job I
used to trigger this was as follows:
dev=/dev/sdc
fio --bs=4K --ioengine=libaio --rw=randread --buffered=0 --numjobs=12 \
--iodepth=128 --iodepth_batch=64 --iodepth_batch_complete=64 \
--thread --norandommap --loops=$((2**31)) --runtime=60 \
--group_reporting --gtod_reduce=1 --name=$dev --filename=$dev \
--invalidate=1
Bart.
next prev parent reply other threads:[~2015-01-13 14:28 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-17 3:59 [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 1/8] block: require blk_rq_prep_clone() be given an initialized clone request Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 2/8] block: initialize bio member of blk-mq request to NULL Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 3/8] block: add blk-mq support to blk_insert_cloned_request() Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 4/8] block: mark blk-mq devices as stackable Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 5/8] dm: remove exports for request-based interfaces without external callers Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 6/8] dm: split request structure out from dm_rq_target_io structure Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 7/8] dm: submit stacked requests in irq enabled context Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 8/8] dm: allocate requests from target when stacking on blk-mq devices Mike Snitzer
2014-12-17 22:35 ` Mike Snitzer
2014-12-17 21:42 ` [PATCH v3 0/8] dm: add request-based blk-mq support Keith Busch
2014-12-17 21:43 ` Jens Axboe
2014-12-17 23:06 ` Mike Snitzer
2014-12-18 1:41 ` Keith Busch
2014-12-18 4:58 ` Mike Snitzer
2014-12-19 14:32 ` Bart Van Assche
2014-12-19 15:38 ` Mike Snitzer
2014-12-19 17:14 ` Mike Snitzer
2014-12-22 15:28 ` Bart Van Assche
2014-12-22 18:49 ` Mike Snitzer
2014-12-23 16:24 ` Bart Van Assche
2014-12-23 17:13 ` Mike Snitzer
2014-12-23 21:42 ` Mike Snitzer
2014-12-24 13:02 ` Bart Van Assche
2014-12-24 18:21 ` Mike Snitzer
2014-12-24 18:55 ` Mike Snitzer
2014-12-24 19:26 ` Mike Snitzer
2015-01-02 17:53 ` Bart Van Assche
2015-01-05 21:35 ` Mike Snitzer
2015-01-06 8:59 ` Christoph Hellwig
2015-01-06 9:31 ` Bart Van Assche
2015-01-06 16:05 ` blk-mq request allocation stalls [was: Re: [PATCH v3 0/8] dm: add request-based blk-mq support] Mike Snitzer
2015-01-06 16:15 ` Jens Axboe
2015-01-07 10:33 ` Bart Van Assche
2015-01-07 15:32 ` Jens Axboe
2015-01-07 16:15 ` Mike Snitzer
2015-01-07 16:18 ` Jens Axboe
2015-01-07 16:22 ` Mike Snitzer
2015-01-07 16:24 ` Jens Axboe
2015-01-07 17:18 ` Mike Snitzer
2015-01-07 17:35 ` Jens Axboe
2015-01-07 20:09 ` Mike Snitzer
2015-01-07 20:40 ` Keith Busch
2015-01-09 19:49 ` Mike Snitzer
2015-01-09 21:07 ` Jens Axboe
2015-01-09 21:11 ` Jens Axboe
2015-01-09 21:40 ` Mike Snitzer
2015-01-09 21:56 ` Jens Axboe
2015-01-09 22:25 ` Mike Snitzer
2015-01-10 0:27 ` Jens Axboe
2015-01-10 1:48 ` Mike Snitzer
2015-01-10 1:59 ` Jens Axboe
2015-01-10 3:10 ` Mike Snitzer
2015-01-12 14:46 ` blk-mq request allocation stalls Bart Van Assche
2015-01-12 15:42 ` Jens Axboe
2015-01-12 16:12 ` Bart Van Assche
2015-01-12 16:34 ` Jens Axboe
2015-01-12 16:58 ` Mike Snitzer
2015-01-12 16:59 ` Jens Axboe
2015-01-12 17:04 ` Bart Van Assche
2015-01-12 17:09 ` Jens Axboe
2015-01-12 17:53 ` Keith Busch
2015-01-12 18:12 ` Jens Axboe
2015-01-12 18:22 ` Keith Busch
2015-01-12 18:35 ` Keith Busch
2015-01-12 19:11 ` Mike Snitzer
2015-01-12 20:21 ` Mike Snitzer
2015-01-13 12:29 ` Bart Van Assche
2015-01-13 14:17 ` Mike Snitzer
2015-01-13 14:28 ` Bart Van Assche [this message]
2015-01-13 16:20 ` dm + blk-mq soft lockup complaint Mike Snitzer
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 18:59 ` Mike Snitzer
2015-01-15 8:11 ` Bart Van Assche
2015-01-15 15:43 ` Mike Snitzer
2015-01-15 15:55 ` Bart Van Assche
2015-01-13 14:59 ` blk-mq request allocation stalls Jens Axboe
2015-01-13 15:11 ` Keith Busch
2015-01-13 15:27 ` Keith Busch
2015-01-13 15:41 ` Mike Snitzer
2015-01-13 15:14 ` Mike Snitzer
2015-01-27 18:42 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls] Mike Snitzer
2015-01-28 16:42 ` Jens Axboe
2015-01-28 17:44 ` Mike Snitzer
2015-01-28 17:49 ` Jens Axboe
2015-01-28 18:10 ` Mike Snitzer
2015-01-29 22:43 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls]X Keith Busch
2015-01-29 23:09 ` Mike Snitzer
2015-01-29 23:44 ` Keith Busch
2015-01-30 0:32 ` Mike Snitzer
2015-01-12 19:05 ` blk-mq request allocation stalls Jens Axboe
2015-01-12 19:07 ` Mike Snitzer
2015-01-12 18:19 ` Mike Snitzer
2014-12-17 22:51 ` [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54B52B73.7040807@sandisk.com \
--to=bart.vanassche@sandisk.com \
--cc=axboe@kernel.dk \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=j-nomura@ce.jp.nec.com \
--cc=keith.busch@intel.com \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.