[PATCH 0/2] nvme-pic: improve max I/O queue handling

[PATCH 0/2] nvme-pic: improve max I/O queue handling

[Niklas Schnelle]

Hi,

while searching for a bug around zPCI + NVMe IRQ handling on a distro
kernel, I got confused around handling of the maximum number
of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following
improvements, that said I'm quite new to this code.
I tested both patches on s390x (with a debug config) and x86_64 so
with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried
always returning 1 from nvme_max_io_queues() and confirmed that on my
Evo 970 Pro this resulted in about half the performance in a fio test
but did not otherwise break things. I couldn't find a reason why
allocating only the I/O queues we actually use would be problematic in
the code either but I might have missed something of course.

Best regards,
Niklas Schnelle

Niklas Schnelle (2):
  nvme-pci: drop min() from nr_io_queues assignment
  nvme-pci: don't allocate unused I/O queues

 drivers/nvme/host/pci.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

-- 
2.17.1

[PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment

[Niklas Schnelle]

in nvme_setup_io_queues() the number of I/O queues is set to either 1 in
case of a quirky Apple device or to the min of nvme_max_io_queues() or
dev->nr_allocated_queues - 1.
This is unnecessarily complicated as dev->nr_allocated_queues is only
assigned once and is nvme_max_io_queues() + 1.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/nvme/host/pci.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 0578ff253c47..b56250b83bdf 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2114,8 +2114,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
 		nr_io_queues = 1;
 	else
-		nr_io_queues = min(nvme_max_io_queues(dev),
-				   dev->nr_allocated_queues - 1);
+		nr_io_queues = dev->nr_allocated_queues - 1;
 
 	result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
 	if (result < 0)
-- 
2.17.1

Re: [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment

[Christoph Hellwig]

Thanks,

applied both patches to nvme-5.11.

[PATCH 2/2] nvme-pci: don't allocate unused I/O queues

[Niklas Schnelle]

currently the NVME_QUIRK_SHARED_TAGS quirk for Apple devices is handled
during the assignment of nr_io_queues in nvme_setup_io_queues().
This however means that for these devices nvme_max_io_queues() will
actually not return the supported maximum which is confusing and
unexpected and also means that in nvme_probe() we are allocating
for I/O queues that will never be used.
Fix this by moving the quirk handling into nvme_max_io_queues().

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
 drivers/nvme/host/pci.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index b56250b83bdf..0f8cea5b30eb 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2089,6 +2089,12 @@ static void nvme_disable_io_queues(struct nvme_dev *dev)
 
 static unsigned int nvme_max_io_queues(struct nvme_dev *dev)
 {
+	/*
+	 * If tags are shared with admin queue (Apple bug), then
+	 * make sure we only use one IO queue.
+	 */
+	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
+		return 1;
 	return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues;
 }
 
@@ -2107,15 +2113,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
 	dev->nr_write_queues = write_queues;
 	dev->nr_poll_queues = poll_queues;
 
-	/*
-	 * If tags are shared with admin queue (Apple bug), then
-	 * make sure we only use one IO queue.
-	 */
-	if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
-		nr_io_queues = 1;
-	else
-		nr_io_queues = dev->nr_allocated_queues - 1;
-
+	nr_io_queues = dev->nr_allocated_queues - 1;
 	result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
 	if (result < 0)
 		return result;
-- 
2.17.1

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Keith Busch]

On Thu, Nov 12, 2020 at 09:23:00AM +0100, Niklas Schnelle wrote:
> while searching for a bug around zPCI + NVMe IRQ handling on a distro
> kernel, I got confused around handling of the maximum number
> of I/O queues in the NVMe driver.
> I think I groked it in the end but would like to propose the following
> improvements, that said I'm quite new to this code.
> I tested both patches on s390x (with a debug config) and x86_64 so
> with both data center and consumer NVMes.
> For the second patch, since I don't own a device with the quirk, I tried
> always returning 1 from nvme_max_io_queues() and confirmed that on my
> Evo 970 Pro this resulted in about half the performance in a fio test
> but did not otherwise break things. I couldn't find a reason why
> allocating only the I/O queues we actually use would be problematic in
> the code either but I might have missed something of course.

I don't think you missed anything, and the series looks like a
reasonable cleanup. I suspect the code was left over from a time when we
didn't allocate the possible queues up-front.

Reviewed-by: Keith Busch <kbusch@kernel.org>

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Niklas Schnelle]

On 11/12/20 3:53 PM, Keith Busch wrote:
> On Thu, Nov 12, 2020 at 09:23:00AM +0100, Niklas Schnelle wrote:
>> while searching for a bug around zPCI + NVMe IRQ handling on a distro
>> kernel, I got confused around handling of the maximum number
>> of I/O queues in the NVMe driver.
>> I think I groked it in the end but would like to propose the following
>> improvements, that said I'm quite new to this code.
>> I tested both patches on s390x (with a debug config) and x86_64 so
>> with both data center and consumer NVMes.
>> For the second patch, since I don't own a device with the quirk, I tried
>> always returning 1 from nvme_max_io_queues() and confirmed that on my
>> Evo 970 Pro this resulted in about half the performance in a fio test
>> but did not otherwise break things. I couldn't find a reason why
>> allocating only the I/O queues we actually use would be problematic in
>> the code either but I might have missed something of course.
> 
> I don't think you missed anything, and the series looks like a
> reasonable cleanup. I suspect the code was left over from a time when we
> didn't allocate the possible queues up-front.
> 
> Reviewed-by: Keith Busch <kbusch@kernel.org>
> 

You got to get something wrong, I hope in this case it's just the subject
of the cover letter :D
Thanks for the review, I appreciate it. Might be getting ahead of
myself but I'm curious who would take this change through their
tree if accepted?

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Keith Busch]

On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
> You got to get something wrong, I hope in this case it's just the subject
> of the cover letter :D

I suppose the change logs could be worded a little better :)

> Thanks for the review, I appreciate it. Might be getting ahead of
> myself but I'm curious who would take this change through their
> tree if accepted?

The linux-nvme tree is located here:

  http://git.infradead.org/nvme.git

Christoph is currently handling patch commits there.

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Niklas Schnelle]

On 11/12/20 6:36 PM, Keith Busch wrote:
> On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
>> You got to get something wrong, I hope in this case it's just the subject
>> of the cover letter :D
> 
> I suppose the change logs could be worded a little better :)

Do you think I should send a v2 with an improved message?
I just realized I'm pretty alone (and wrong) in starting
the commit message body lower case too.

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Christoph Hellwig]

On Fri, Nov 13, 2020 at 02:15:59PM +0100, Niklas Schnelle wrote:
> 
> 
> On 11/12/20 6:36 PM, Keith Busch wrote:
> > On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
> >> You got to get something wrong, I hope in this case it's just the subject
> >> of the cover letter :D
> > 
> > I suppose the change logs could be worded a little better :)
> 
> Do you think I should send a v2 with an improved message?
> I just realized I'm pretty alone (and wrong) in starting
> the commit message body lower case too.

I'll pull in soon, no need to resend.

Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling

[Niklas Schnelle]

On 11/13/20 5:25 PM, Christoph Hellwig wrote:
> On Fri, Nov 13, 2020 at 02:15:59PM +0100, Niklas Schnelle wrote:
>>
>>
>> On 11/12/20 6:36 PM, Keith Busch wrote:
>>> On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
>>>> You got to get something wrong, I hope in this case it's just the subject
>>>> of the cover letter :D
>>>
>>> I suppose the change logs could be worded a little better :)
>>
>> Do you think I should send a v2 with an improved message?
>> I just realized I'm pretty alone (and wrong) in starting
>> the commit message body lower case too.
> 
> I'll pull in soon, no need to resend.
> 

Thanks, appreciate your effort!