[PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Christoph Hellwig]

This turns down to cause up to 15% performance regressions on standard
large transfer workloads.  Revert it and hope the IOVA allocator can
eventually be fixed to remove the need for the original workaround.

This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/nvme/host/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 5440cf18b55b..82e308fe4f4c 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3747,7 +3747,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
 	 */
 	dev->ctrl.max_hw_sectors = min_t(u32,
 			NVME_MAX_BYTES >> SECTOR_SHIFT,
-			dma_opt_mapping_size(&pdev->dev) >> 9);
+			dma_max_mapping_size(&pdev->dev) >> 9);
 	dev->ctrl.max_segments = NVME_MAX_SEGS;
 	dev->ctrl.max_integrity_segments = 1;
 	return dev;
-- 
2.53.0

Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[John Garry]

On 10/6/26 13:02, Christoph Hellwig wrote:
> This turns down to cause up to 15% performance regressions on standard
> large transfer workloads.  Revert it and hope the IOVA allocator can
> eventually be fixed to remove the need for the original workaround.
> 
> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
> 

What was the block size and conditions for the original issue? The 
commit just mentions that mdts=10 and we get a 4096KB max HW sectors.

I can't find the original thread.

> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>   drivers/nvme/host/pci.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 5440cf18b55b..82e308fe4f4c 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -3747,7 +3747,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci_dev *pdev,
>   	 */
>   	dev->ctrl.max_hw_sectors = min_t(u32,
>   			NVME_MAX_BYTES >> SECTOR_SHIFT,
> -			dma_opt_mapping_size(&pdev->dev) >> 9);
> +			dma_max_mapping_size(&pdev->dev) >> 9);
>   	dev->ctrl.max_segments = NVME_MAX_SEGS;
>   	dev->ctrl.max_integrity_segments = 1;
>   	return dev;

Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Christoph Hellwig]

On Thu, Oct 08, 2026 at 08:36:54AM +0100, John Garry wrote:
> On 10/6/26 13:02, Christoph Hellwig wrote:
>> This turns down to cause up to 15% performance regressions on standard
>> large transfer workloads.  Revert it and hope the IOVA allocator can
>> eventually be fixed to remove the need for the original workaround.
>>
>> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
>>
>
> What was the block size and conditions for the original issue? The commit 
> just mentions that mdts=10 and we get a 4096KB max HW sectors.

Good question.  The original submitter and reporter are on Cc.

RE: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Adrian Huang12]

> -----Original Message-----
> From: Christoph Hellwig <hch@lst.de>
> Sent: Thursday, October 8, 2026 8:37 PM
> To: John Garry <john.garry@linux.dev>
> Cc: Christoph Hellwig <hch@lst.de>; kbusch@kernel.org; Adrian Huang12
> <ahuang12@lenovo.com>; sagi@grimberg.me; linux-nvme@lists.infradead.org;
> Jiwei JW10 Sun <sunjw10@lenovo.com>; iommu@lists.linux.dev
> Subject: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors
> based on DMA optimized limitation"
> 
> On Thu, Oct 08, 2026 at 08:36:54AM +0100, John Garry wrote:
> > On 10/6/26 13:02, Christoph Hellwig wrote:
> >> This turns down to cause up to 15% performance regressions on
> >> standard large transfer workloads.  Revert it and hope the IOVA
> >> allocator can eventually be fixed to remove the need for the original
> workaround.
> >>
> >> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
> >>
> >
> > What was the block size and conditions for the original issue? The
> > commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
> 
> Good question.  The original submitter and reporter are on Cc.

1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.

[Detail]
1. A NVME with mdts=10 (maximum data transfer size = 4096 KB)
  1.1 fio with bs=128k or below => Passed
  1.2 fio with bs > 128k => Failed
2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
  2.1 The issue does not reproduce regardless of the 'bs' configuration
3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
5. The issue only appears with high-core-count servers
  5.1 Reproduced servers: 448-core server and 384-core server
  5.2 Non-reproduced server: 128-core server

BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.

-- Adrian

Re: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Keith Busch]

On Fri, Oct 09, 2026 at 09:11:56AM +0000, Adrian Huang12 wrote:
> 
> BTW, with this reverted patch, I'm setting up the test environment to
> check whether the issue does not appear with a newer kernel that
> includes the updated IOMMU subsystem. I'll share the results by the
> end of next week.

It does sound like an IOMMU implementation issue if the revert regresses
your testing. Let's say the application wants to write 1MB of data:
we're DMA mapping the same total amount either way, so I don't think it
should have been more efficient for the driver to send eight 128k
mapping requests vs a single 1MB.

Re: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[John Garry]

On 10/9/26 10:11, Adrian Huang12 wrote:
>> -----Original Message-----
>> From: Christoph Hellwig <hch@lst.de>
>> Sent: Thursday, October 8, 2026 8:37 PM
>> To: John Garry <john.garry@linux.dev>
>> Cc: Christoph Hellwig <hch@lst.de>; kbusch@kernel.org; Adrian Huang12
>> <ahuang12@lenovo.com>; sagi@grimberg.me; linux-nvme@lists.infradead.org;
>> Jiwei JW10 Sun <sunjw10@lenovo.com>; iommu@lists.linux.dev
>> Subject: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors
>> based on DMA optimized limitation"
>>
>> On Thu, Oct 08, 2026 at 08:36:54AM +0100, John Garry wrote:
>>> On 10/6/26 13:02, Christoph Hellwig wrote:
>>>> This turns down to cause up to 15% performance regressions on
>>>> standard large transfer workloads.  Revert it and hope the IOVA
>>>> allocator can eventually be fixed to remove the need for the original
>> workaround.
>>>>
>>>> This reverts commit 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd.
>>>>
>>>
>>> What was the block size and conditions for the original issue? The
>>> commit just mentions that mdts=10 and we get a 4096KB max HW sectors.
>>
>> Good question.  The original submitter and reporter are on Cc.
> 
> 1. The block size depends on the configuration of the fio parameter 'bs'. See below for details.
> 2. Conditions for the original issue: The issue is triggered if the IO size > 128KB.
> 
> [Detail]
> 1. A NVME with mdts=10 (maximum data transfer size = 4096 KB)
>    1.1 fio with bs=128k or below => Passed
>    1.2 fio with bs > 128k => Failed


That block size limit is an important detail missing from 
3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd commit message.

And now we seem to have a conflict, as Christoph wants block size > IOVA 
rcache limit but you find it may cause this soft lockup.

> 2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
>    2.1 The issue does not reproduce regardless of the 'bs' configuration
> 3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
> 4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
> 5. The issue only appears with high-core-count servers
>    5.1 Reproduced servers: 448-core server and 384-core server
>    5.2 Non-reproduced server: 128-core server

How long does it take for the lockup to trigger? Pretty quickly, I guess.

Does changing the number of CPUs affect throughput (before the lockup)? 
Are you using just a single disk?

> 
> BTW, with this reverted patch, I'm setting up the test environment to check whether the issue does not appear with a newer kernel that includes the updated IOMMU subsystem. I'll share the results by the end of next week.
> 

thanks

Re: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[Christoph Hellwig]

On Mon, Oct 12, 2026 at 09:42:04AM +0100, John Garry wrote:
> And now we seem to have a conflict, as Christoph wants block size > IOVA 
> rcache limit but you find it may cause this soft lockup.
>
>> 2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
>>    2.1 The issue does not reproduce regardless of the 'bs' configuration
>> 3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
>> 4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
>> 5. The issue only appears with high-core-count servers
>>    5.1 Reproduced servers: 448-core server and 384-core server
>>    5.2 Non-reproduced server: 128-core server
>
> How long does it take for the lockup to trigger? Pretty quickly, I guess.
>
> Does changing the number of CPUs affect throughput (before the lockup)? Are 
> you using just a single disk?

If the iova cache locks up under a perfectly normal workloads it
is broken.  We really need to fix the code there.

Re: [External] Re: [PATCH] Revert "nvme-pci: clamp max_hw_sectors based on DMA optimized limitation"

[John Garry]

On 10/12/26 13:17, Christoph Hellwig wrote:
> On Mon, Oct 12, 2026 at 09:42:04AM +0100, John Garry wrote:
>> And now we seem to have a conflict, as Christoph wants block size > IOVA
>> rcache limit but you find it may cause this soft lockup.
>>
>>> 2. Another NVME with mdts=5 (maximum data transfer size = 128KB)
>>>     2.1 The issue does not reproduce regardless of the 'bs' configuration
>>> 3. The condition '128 KB' is identified because the IOMMU scalable IOVA mechanism supports a maximum transfer size of 128 KB.
>>> 4. IOMMU pass-through mode does not trigger the issue with the 'mdts=10' nvme disk.
>>> 5. The issue only appears with high-core-count servers
>>>     5.1 Reproduced servers: 448-core server and 384-core server
>>>     5.2 Non-reproduced server: 128-core server
>>
>> How long does it take for the lockup to trigger? Pretty quickly, I guess.
>>
>> Does changing the number of CPUs affect throughput (before the lockup)? Are
>> you using just a single disk?
> 
> If the iova cache locks up under a perfectly normal workloads it
> is broken. 


> We really need to fix the code there.

Sure, but it would help to first fully understand the conditions to fix it.