From: keith.busch@intel.com (Keith Busch)
Subject: [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues
Date: Tue, 20 Sep 2016 11:00:37 -0400 [thread overview]
Message-ID: <20160920150037.GA11190@keith> (raw)
In-Reply-To: <20160919103805.GA22169@agordeev.lab.eng.brq.redhat.com>
On Mon, Sep 19, 2016@12:38:05PM +0200, Alexander Gordeev wrote:
> On Fri, Sep 16, 2016@05:04:48PM -0400, Keith Busch wrote:
>
> > Having a 1:1 already seemed like the ideal solution since you can't
> > simultaneously utilize more than that from the host, so there's no more
> > h/w parallelisms from we can exploit. On the controller side, fetching
> > commands is serialized memory reads, so I don't think spreading IO
> > among more h/w queues helps the target over posting more commands to a
> > single queue.
>
> I take a notion of un-ordered commands completion you described below.
> But I fail to realize why a CPU would not simultaneously utilize more
> than one queue by posting to multiple. Is it due to nvme specifics or
> you assume the host would not issue that many commands?
What I mean is that if you have N CPUs, you can't possibly simultaneously
write more than N submission queue entries. The benefit of having 1:1
for the queue <-> CPU mapping is that each CPU can post a command to
its queue without lock contention at the same time as another thread.
Having more to choose from doesn't let the host post commands any faster
than we can today.
When we're out of tags, the request currently just waits for one to
become available, increasing submission latency. You can fix that by
increasing the available tags with deeper or more h/w queues, but that
just increases completion latency since the device can't process them
any faster. It's six of one, half dozen of the other.
The depth per queue defaults to 1k. If your process really is able to use
all those resources, the hardware is completely saturated and you're not
going to benefit from introducing more tags [1]. It could conceivably
be worse by reducing cache-hits, or hit inappropriate timeout handling
with the increased completion latency.
[1] http://lists.infradead.org/pipermail/linux-nvme/2014-July/001064.html
prev parent reply other threads:[~2016-09-20 15:00 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-09-16 8:51 [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues Alexander Gordeev
2016-09-16 8:51 ` [PATCH 01/21] blk-mq: Fix memory leaks on a queue cleanup Alexander Gordeev
2016-09-16 8:51 ` [PATCH 02/21] blk-mq: Fix a potential NULL pointer assignment to hctx tags Alexander Gordeev
2016-09-16 8:51 ` [PATCH 03/21] block: Get rid of unused request_queue::nr_queues member Alexander Gordeev
2016-09-16 8:51 ` [PATCH 04/21] blk-mq: Do not limit number of queues to 'nr_cpu_ids' in allocations Alexander Gordeev
2016-09-16 8:51 ` [PATCH 05/21] blk-mq: Update hardware queue map after q->nr_hw_queues is set Alexander Gordeev
2016-09-16 8:51 ` [PATCH 06/21] block: Remove redundant blk_mq_ops::map_queue() interface Alexander Gordeev
2016-09-16 8:51 ` [PATCH 07/21] blk-mq: Remove a redundant assignment Alexander Gordeev
2016-09-16 8:51 ` [PATCH 08/21] blk-mq: Cleanup hardware context data node selection Alexander Gordeev
2016-09-16 8:51 ` [PATCH 09/21] blk-mq: Cleanup a loop exit condition Alexander Gordeev
2016-09-16 8:51 ` [PATCH 10/21] blk-mq: Get rid of unnecessary blk_mq_free_hw_queues() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 11/21] blk-mq: Move duplicating code to blk_mq_exit_hctx() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 12/21] blk-mq: Uninit hardware context in order reverse to init Alexander Gordeev
2016-09-16 8:51 ` [PATCH 13/21] blk-mq: Move hardware context init code into blk_mq_init_hctx() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 14/21] blk-mq: Rework blk_mq_init_hctx() function Alexander Gordeev
2016-09-16 8:51 ` [PATCH 15/21] blk-mq: Pair blk_mq_hctx_kobj_init() with blk_mq_hctx_kobj_put() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 16/21] blk-mq: Set flush_start_tag to BLK_MQ_MAX_DEPTH Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 17/21] blk-mq: Introduce a 1:N hardware contexts Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 18/21] blk-mq: Enable tag numbers exceed hardware queue depth Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 19/21] blk-mq: Enable combined hardware queues Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 20/21] blk-mq: Allow " Alexander Gordeev
2016-09-16 8:51 ` [PATCH 21/21] null_blk: Do not limit # of hardware queues to # of CPUs Alexander Gordeev
2016-09-16 9:27 ` [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues Christoph Hellwig
2016-09-16 10:10 ` Alexander Gordeev
2016-09-16 21:04 ` Keith Busch
2016-09-19 10:38 ` Alexander Gordeev
2016-09-19 13:33 ` Bart Van Assche
2016-09-20 15:00 ` Keith Busch [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160920150037.GA11190@keith \
--to=keith.busch@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).