From: Sinan Kaya <okaya@codeaurora.org>
To: Keith Busch <keith.busch@intel.com>
Cc: linux-nvme@lists.infradead.org, timur@codeaurora.org,
linux-arm-msm@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, Jens Axboe <axboe@fb.com>,
Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH] nvme: Acknowledge completion queue on each iteration
Date: Mon, 17 Jul 2017 18:46:11 -0400 [thread overview]
Message-ID: <6d10032c-35ec-978c-6b8f-1ab9c07adf7f@codeaurora.org> (raw)
In-Reply-To: <20170717224551.GA1496@localhost.localdomain>
Hi Keith,
On 7/17/2017 6:45 PM, Keith Busch wrote:
> On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote:
>> Code is moving the completion queue doorbell after processing all completed
>> events and sending callbacks to the block layer on each iteration.
>>
>> This is causing a performance drop when a lot of jobs are queued towards
>> the HW. Move the completion queue doorbell on each loop instead and allow new
>> jobs to be queued by the HW.
>
> That doesn't make sense. Aggregating doorbell writes should be much more
> efficient for high depth workloads.
>
Problem is that code is throttling the HW as HW cannot queue more completions until
SW get a chance to clear it.
As an example:
for each in N
(
blk_layer()
)
ring door bell
HW cannot queue new job until N x blk_layer operations are processed and queue
element ownership is passed to the HW after the loop. HW is just sitting idle
there if no queue entries are available.
Sinan
--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
WARNING: multiple messages have this Message-ID (diff)
From: okaya@codeaurora.org (Sinan Kaya)
Subject: [PATCH] nvme: Acknowledge completion queue on each iteration
Date: Mon, 17 Jul 2017 18:46:11 -0400 [thread overview]
Message-ID: <6d10032c-35ec-978c-6b8f-1ab9c07adf7f@codeaurora.org> (raw)
In-Reply-To: <20170717224551.GA1496@localhost.localdomain>
Hi Keith,
On 7/17/2017 6:45 PM, Keith Busch wrote:
> On Mon, Jul 17, 2017@06:36:23PM -0400, Sinan Kaya wrote:
>> Code is moving the completion queue doorbell after processing all completed
>> events and sending callbacks to the block layer on each iteration.
>>
>> This is causing a performance drop when a lot of jobs are queued towards
>> the HW. Move the completion queue doorbell on each loop instead and allow new
>> jobs to be queued by the HW.
>
> That doesn't make sense. Aggregating doorbell writes should be much more
> efficient for high depth workloads.
>
Problem is that code is throttling the HW as HW cannot queue more completions until
SW get a chance to clear it.
As an example:
for each in N
(
blk_layer()
)
ring door bell
HW cannot queue new job until N x blk_layer operations are processed and queue
element ownership is passed to the HW after the loop. HW is just sitting idle
there if no queue entries are available.
Sinan
--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
WARNING: multiple messages have this Message-ID (diff)
From: okaya@codeaurora.org (Sinan Kaya)
To: linux-arm-kernel@lists.infradead.org
Subject: [PATCH] nvme: Acknowledge completion queue on each iteration
Date: Mon, 17 Jul 2017 18:46:11 -0400 [thread overview]
Message-ID: <6d10032c-35ec-978c-6b8f-1ab9c07adf7f@codeaurora.org> (raw)
In-Reply-To: <20170717224551.GA1496@localhost.localdomain>
Hi Keith,
On 7/17/2017 6:45 PM, Keith Busch wrote:
> On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote:
>> Code is moving the completion queue doorbell after processing all completed
>> events and sending callbacks to the block layer on each iteration.
>>
>> This is causing a performance drop when a lot of jobs are queued towards
>> the HW. Move the completion queue doorbell on each loop instead and allow new
>> jobs to be queued by the HW.
>
> That doesn't make sense. Aggregating doorbell writes should be much more
> efficient for high depth workloads.
>
Problem is that code is throttling the HW as HW cannot queue more completions until
SW get a chance to clear it.
As an example:
for each in N
(
blk_layer()
)
ring door bell
HW cannot queue new job until N x blk_layer operations are processed and queue
element ownership is passed to the HW after the loop. HW is just sitting idle
there if no queue entries are available.
Sinan
--
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
next prev parent reply other threads:[~2017-07-17 22:46 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-17 22:36 [PATCH] nvme: Acknowledge completion queue on each iteration Sinan Kaya
2017-07-17 22:36 ` Sinan Kaya
2017-07-17 22:36 ` Sinan Kaya
2017-07-17 22:45 ` Keith Busch
2017-07-17 22:45 ` Keith Busch
2017-07-17 22:45 ` Keith Busch
2017-07-17 22:46 ` Sinan Kaya [this message]
2017-07-17 22:46 ` Sinan Kaya
2017-07-17 22:46 ` Sinan Kaya
2017-07-17 22:56 ` Keith Busch
2017-07-17 22:56 ` Keith Busch
2017-07-17 22:56 ` Keith Busch
2017-07-17 23:07 ` okaya
2017-07-17 23:07 ` okaya at codeaurora.org
2017-07-17 23:07 ` okaya
2017-07-18 14:36 ` Keith Busch
2017-07-18 14:36 ` Keith Busch
2017-07-18 14:36 ` Keith Busch
2017-07-18 18:52 ` Sinan Kaya
2017-07-18 18:52 ` Sinan Kaya
2017-07-18 18:52 ` Sinan Kaya
2017-07-18 21:26 ` Keith Busch
2017-07-18 21:26 ` Keith Busch
2017-07-18 21:26 ` Keith Busch
2017-07-19 9:20 ` Sagi Grimberg
2017-07-19 9:20 ` Sagi Grimberg
2017-07-19 9:20 ` Sagi Grimberg
2017-07-19 10:37 ` Sinan Kaya
2017-07-19 10:37 ` Sinan Kaya
2017-07-19 10:37 ` Sinan Kaya
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6d10032c-35ec-978c-6b8f-1ab9c07adf7f@codeaurora.org \
--to=okaya@codeaurora.org \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=keith.busch@intel.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-arm-msm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
--cc=timur@codeaurora.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.