From: Mike Snitzer <snitzer@redhat.com>
To: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Keith Busch <keith.busch@intel.com>, Jens Axboe <axboe@kernel.dk>,
Christoph Hellwig <hch@infradead.org>,
device-mapper development <dm-devel@redhat.com>,
Jun'ichi Nomura <j-nomura@ce.jp.nec.com>,
linux-scsi@vger.kernel.org
Subject: Re: dm + blk-mq soft lockup complaint
Date: Tue, 13 Jan 2015 11:20:52 -0500 [thread overview]
Message-ID: <20150113162052.GA30837@redhat.com> (raw)
In-Reply-To: <54B52B73.7040807@sandisk.com>
On Tue, Jan 13 2015 at 9:28am -0500,
Bart Van Assche <bart.vanassche@sandisk.com> wrote:
> On 01/13/15 15:18, Mike Snitzer wrote:
> > On Tue, Jan 13 2015 at 7:29am -0500,
> > Bart Van Assche <bart.vanassche@sandisk.com> wrote:
> >> However, I hit another issue while running I/O on top of a multipath
> >> device (on a kernel with lockdep and SLUB memory poisoning enabled):
> >>
> >> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kdmwork-253:0:3116]
> >> CPU: 7 PID: 3116 Comm: kdmwork-253:0 Tainted: G W 3.19.0-rc4-debug+ #1
> >> Call Trace:
> >> [<ffffffff8118e4be>] kmem_cache_alloc+0x28e/0x2c0
> >> [<ffffffff81346aca>] alloc_iova_mem+0x1a/0x20
> >> [<ffffffff81342c8e>] alloc_iova+0x2e/0x250
> >> [<ffffffff81344b65>] intel_alloc_iova+0x95/0xd0
> >> [<ffffffff81348a15>] intel_map_sg+0xc5/0x260
> >> [<ffffffffa07e0661>] srp_queuecommand+0xa11/0xc30 [ib_srp]
> >> [<ffffffffa001698e>] scsi_dispatch_cmd+0xde/0x5a0 [scsi_mod]
> >> [<ffffffffa0017480>] scsi_queue_rq+0x630/0x700 [scsi_mod]
> >> [<ffffffff8125683d>] __blk_mq_run_hw_queue+0x1dd/0x370
> >> [<ffffffff81256aae>] blk_mq_alloc_request+0xde/0x150
> >> [<ffffffff8124bade>] blk_get_request+0x2e/0xe0
> >> [<ffffffffa07ebd0f>] __multipath_map.isra.15+0x1cf/0x210 [dm_multipath]
> >> [<ffffffffa07ebd6a>] multipath_clone_and_map+0x1a/0x20 [dm_multipath]
> >> [<ffffffffa044abb5>] map_tio_request+0x1d5/0x3a0 [dm_mod]
> >> [<ffffffff81075d16>] kthread_worker_fn+0x86/0x1b0
> >> [<ffffffff81075c0f>] kthread+0xef/0x110
> >> [<ffffffff814db42c>] ret_from_fork+0x7c/0xb0
> >
> > Unfortunate. Is this still with a 16MB backing device or is it real
> > hardware? Can you share the workload so that myself and/or Keith could
> > try to reproduce?
>
> Hello Mike,
>
> This is still with a 16MB RAM disk as backing device. The fio job I
> used to trigger this was as follows:
>
> dev=/dev/sdc
> fio --bs=4K --ioengine=libaio --rw=randread --buffered=0 --numjobs=12 \
> --iodepth=128 --iodepth_batch=64 --iodepth_batch_complete=64 \
> --thread --norandommap --loops=$((2**31)) --runtime=60 \
> --group_reporting --gtod_reduce=1 --name=$dev --filename=$dev \
> --invalidate=1
OK, I assume you specified the mpath device for the test that failed.
This test works fine on my 100MB scsi_debug device with 4 paths exported
over virtio-blk to a guest that assembles the mpath device.
Could be a hang that is unique to scsi-mq.
Any chance you'd be willing to provide a HOWTO for setting up your
SRP/iscsi configuration?
Are you carrying any related changes that are not upstream? (I can hunt
down the email in this thread where you describe your kernel tree...)
I'll try to reproduce but this info could be useful to others that are
more scsi-mq inclined who might need to chase this too.
next prev parent reply other threads:[~2015-01-13 16:20 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-17 3:59 [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 1/8] block: require blk_rq_prep_clone() be given an initialized clone request Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 2/8] block: initialize bio member of blk-mq request to NULL Mike Snitzer
2014-12-17 3:59 ` [PATCH v3 3/8] block: add blk-mq support to blk_insert_cloned_request() Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 4/8] block: mark blk-mq devices as stackable Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 5/8] dm: remove exports for request-based interfaces without external callers Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 6/8] dm: split request structure out from dm_rq_target_io structure Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 7/8] dm: submit stacked requests in irq enabled context Mike Snitzer
2014-12-17 4:00 ` [PATCH v3 8/8] dm: allocate requests from target when stacking on blk-mq devices Mike Snitzer
2014-12-17 22:35 ` Mike Snitzer
2014-12-17 21:42 ` [PATCH v3 0/8] dm: add request-based blk-mq support Keith Busch
2014-12-17 21:43 ` Jens Axboe
2014-12-17 23:06 ` Mike Snitzer
2014-12-18 1:41 ` Keith Busch
2014-12-18 4:58 ` Mike Snitzer
2014-12-19 14:32 ` Bart Van Assche
2014-12-19 15:38 ` Mike Snitzer
2014-12-19 17:14 ` Mike Snitzer
2014-12-22 15:28 ` Bart Van Assche
2014-12-22 18:49 ` Mike Snitzer
2014-12-23 16:24 ` Bart Van Assche
2014-12-23 17:13 ` Mike Snitzer
2014-12-23 21:42 ` Mike Snitzer
2014-12-24 13:02 ` Bart Van Assche
2014-12-24 18:21 ` Mike Snitzer
2014-12-24 18:55 ` Mike Snitzer
2014-12-24 19:26 ` Mike Snitzer
2015-01-02 17:53 ` Bart Van Assche
2015-01-05 21:35 ` Mike Snitzer
2015-01-06 8:59 ` Christoph Hellwig
2015-01-06 9:31 ` Bart Van Assche
2015-01-06 16:05 ` blk-mq request allocation stalls [was: Re: [PATCH v3 0/8] dm: add request-based blk-mq support] Mike Snitzer
2015-01-06 16:15 ` Jens Axboe
2015-01-07 10:33 ` Bart Van Assche
2015-01-07 15:32 ` Jens Axboe
2015-01-07 16:15 ` Mike Snitzer
2015-01-07 16:18 ` Jens Axboe
2015-01-07 16:22 ` Mike Snitzer
2015-01-07 16:24 ` Jens Axboe
2015-01-07 17:18 ` Mike Snitzer
2015-01-07 17:35 ` Jens Axboe
2015-01-07 20:09 ` Mike Snitzer
2015-01-07 20:40 ` Keith Busch
2015-01-09 19:49 ` Mike Snitzer
2015-01-09 21:07 ` Jens Axboe
2015-01-09 21:11 ` Jens Axboe
2015-01-09 21:40 ` Mike Snitzer
2015-01-09 21:56 ` Jens Axboe
2015-01-09 22:25 ` Mike Snitzer
2015-01-10 0:27 ` Jens Axboe
2015-01-10 1:48 ` Mike Snitzer
2015-01-10 1:59 ` Jens Axboe
2015-01-10 3:10 ` Mike Snitzer
2015-01-12 14:46 ` blk-mq request allocation stalls Bart Van Assche
2015-01-12 15:42 ` Jens Axboe
2015-01-12 16:12 ` Bart Van Assche
2015-01-12 16:34 ` Jens Axboe
2015-01-12 16:58 ` Mike Snitzer
2015-01-12 16:59 ` Jens Axboe
2015-01-12 17:04 ` Bart Van Assche
2015-01-12 17:09 ` Jens Axboe
2015-01-12 17:53 ` Keith Busch
2015-01-12 18:12 ` Jens Axboe
2015-01-12 18:22 ` Keith Busch
2015-01-12 18:35 ` Keith Busch
2015-01-12 19:11 ` Mike Snitzer
2015-01-12 20:21 ` Mike Snitzer
2015-01-13 12:29 ` Bart Van Assche
2015-01-13 14:17 ` Mike Snitzer
2015-01-13 14:28 ` dm + blk-mq soft lockup complaint Bart Van Assche
2015-01-13 16:20 ` Mike Snitzer [this message]
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 9:16 ` Bart Van Assche
2015-01-14 18:59 ` Mike Snitzer
2015-01-15 8:11 ` Bart Van Assche
2015-01-15 15:43 ` Mike Snitzer
2015-01-15 15:55 ` Bart Van Assche
2015-01-13 14:59 ` blk-mq request allocation stalls Jens Axboe
2015-01-13 15:11 ` Keith Busch
2015-01-13 15:27 ` Keith Busch
2015-01-13 15:41 ` Mike Snitzer
2015-01-13 15:14 ` Mike Snitzer
2015-01-27 18:42 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls] Mike Snitzer
2015-01-28 16:42 ` Jens Axboe
2015-01-28 17:44 ` Mike Snitzer
2015-01-28 17:49 ` Jens Axboe
2015-01-28 18:10 ` Mike Snitzer
2015-01-29 22:43 ` blk-mq DM changes for 3.20 [was: Re: blk-mq request allocation stalls]X Keith Busch
2015-01-29 23:09 ` Mike Snitzer
2015-01-29 23:44 ` Keith Busch
2015-01-30 0:32 ` Mike Snitzer
2015-01-12 19:05 ` blk-mq request allocation stalls Jens Axboe
2015-01-12 19:07 ` Mike Snitzer
2015-01-12 18:19 ` Mike Snitzer
2014-12-17 22:51 ` [PATCH v3 0/8] dm: add request-based blk-mq support Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150113162052.GA30837@redhat.com \
--to=snitzer@redhat.com \
--cc=axboe@kernel.dk \
--cc=bart.vanassche@sandisk.com \
--cc=dm-devel@redhat.com \
--cc=hch@infradead.org \
--cc=j-nomura@ce.jp.nec.com \
--cc=keith.busch@intel.com \
--cc=linux-scsi@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.