From: Alexander Gordeev <agordeev@redhat.com>
To: Keith Busch <keith.busch@intel.com>
Cc: linux-kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org
Subject: Re: [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues
Date: Mon, 19 Sep 2016 12:38:05 +0200 [thread overview]
Message-ID: <20160919103805.GA22169@agordeev.lab.eng.brq.redhat.com> (raw)
In-Reply-To: <20160916210448.GA1178@localhost.localdomain>
On Fri, Sep 16, 2016 at 05:04:48PM -0400, Keith Busch wrote:
CC-ing linux-block@vger.kernel.org
> I'm not sure I see how this helps. That probably means I'm not considering
> the right scenario. Could you elaborate on when having multiple hardware
> queues to choose from a given CPU will provide a benefit?
No, I do not keep in mind any particular scenario besides common
sense. Just an assumption deeper queues are better (in this RFC
a virtual combined queue consisting of multipe h/w queues).
Apparently, there could be positive effects only in systems where
# of queues / # of CPUs > 1 or # of queues / # of cores > 1. But
I do not happen to have ones. If I had numbers this would not be
the RFC and I probably would not have posted in the first place ;)
Would it be possible to give it a try on your hardware?
> If we're out of avaliable h/w tags, having more queues shouldn't
> improve performance. The tag depth on each nvme hw context is already
> deep enough that it should mean even one full queue has saturated the
> device capabilities.
Am I getting you right - a single full nvme hardware queue makes
other queues stalled?
> Having a 1:1 already seemed like the ideal solution since you can't
> simultaneously utilize more than that from the host, so there's no more
> h/w parallelisms from we can exploit. On the controller side, fetching
> commands is serialized memory reads, so I don't think spreading IO
> among more h/w queues helps the target over posting more commands to a
> single queue.
I take a notion of un-ordered commands completion you described below.
But I fail to realize why a CPU would not simultaneously utilize more
than one queue by posting to multiple. Is it due to nvme specifics or
you assume the host would not issue that many commands?
Besides, blk-mq-tag re-uses the latest freed tag and IO should not
actually get spred. Instead, if only currently used hardware queue is
full, the next available queue is chosen. But this is a speculation
without real benchmarks, of course.
> If a CPU has more than one to choose from, a command sent to a less
> used queue would be serviced ahead of previously issued commands on a
> more heavily used one from the same CPU thread due to how NVMe command
> arbitraration works, so it sounds like this would create odd latency
> outliers.
Yep, that sounds scary indeed. Still, any hints on benchmarking
are welcomed.
Many thanks!
> Thanks,
> Keith
next prev parent reply other threads:[~2016-09-19 10:32 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-09-16 8:51 [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues Alexander Gordeev
2016-09-16 8:51 ` [PATCH 01/21] blk-mq: Fix memory leaks on a queue cleanup Alexander Gordeev
2016-09-16 8:51 ` [PATCH 02/21] blk-mq: Fix a potential NULL pointer assignment to hctx tags Alexander Gordeev
2016-09-16 8:51 ` [PATCH 03/21] block: Get rid of unused request_queue::nr_queues member Alexander Gordeev
2016-09-16 8:51 ` [PATCH 04/21] blk-mq: Do not limit number of queues to 'nr_cpu_ids' in allocations Alexander Gordeev
2016-09-16 8:51 ` [PATCH 05/21] blk-mq: Update hardware queue map after q->nr_hw_queues is set Alexander Gordeev
2016-09-16 8:51 ` [PATCH 06/21] block: Remove redundant blk_mq_ops::map_queue() interface Alexander Gordeev
2016-09-16 8:51 ` [PATCH 07/21] blk-mq: Remove a redundant assignment Alexander Gordeev
2016-09-16 8:51 ` [PATCH 08/21] blk-mq: Cleanup hardware context data node selection Alexander Gordeev
2016-09-16 8:51 ` [PATCH 09/21] blk-mq: Cleanup a loop exit condition Alexander Gordeev
2016-09-16 8:51 ` [PATCH 10/21] blk-mq: Get rid of unnecessary blk_mq_free_hw_queues() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 11/21] blk-mq: Move duplicating code to blk_mq_exit_hctx() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 12/21] blk-mq: Uninit hardware context in order reverse to init Alexander Gordeev
2016-09-16 8:51 ` [PATCH 13/21] blk-mq: Move hardware context init code into blk_mq_init_hctx() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 14/21] blk-mq: Rework blk_mq_init_hctx() function Alexander Gordeev
2016-09-16 8:51 ` [PATCH 15/21] blk-mq: Pair blk_mq_hctx_kobj_init() with blk_mq_hctx_kobj_put() Alexander Gordeev
2016-09-16 8:51 ` [PATCH 16/21] blk-mq: Set flush_start_tag to BLK_MQ_MAX_DEPTH Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 17/21] blk-mq: Introduce a 1:N hardware contexts Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 18/21] blk-mq: Enable tag numbers exceed hardware queue depth Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 19/21] blk-mq: Enable combined hardware queues Alexander Gordeev
2016-09-16 8:51 ` [PATCH RFC 20/21] blk-mq: Allow " Alexander Gordeev
2016-09-16 8:51 ` [PATCH 21/21] null_blk: Do not limit # of hardware queues to # of CPUs Alexander Gordeev
2016-09-16 9:27 ` [PATCH RFC 00/21] blk-mq: Introduce combined hardware queues Christoph Hellwig
2016-09-16 10:10 ` Alexander Gordeev
2016-09-16 21:04 ` Keith Busch
2016-09-19 10:38 ` Alexander Gordeev [this message]
2016-09-19 13:33 ` Bart Van Assche
2016-09-20 15:00 ` Keith Busch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160919103805.GA22169@agordeev.lab.eng.brq.redhat.com \
--to=agordeev@redhat.com \
--cc=axboe@kernel.dk \
--cc=keith.busch@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).