[PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[Rishabh Bhatnagar]

If swiotlb is force enabled dma_max_mapping_size ends up calling
swiotlb_max_mapping_size which takes into account the min align
mask for the device.
Set the min align mask for nvme driver before calling
dma_max_mapping_size while calculating max hw sectors.

Signed-off-by: Rishabh Bhatnagar <risbhat@amazon.com>
---
 drivers/nvme/host/pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 98864b853eef..30e71e41a0a2 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2834,6 +2834,8 @@ static void nvme_reset_work(struct work_struct *work)
 		nvme_start_admin_queue(&dev->ctrl);
 	}
 
+	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
+
 	/*
 	 * Limit the max command size to prevent iod->sg allocations going
 	 * over a single page.
@@ -2846,7 +2848,6 @@ static void nvme_reset_work(struct work_struct *work)
 	 * Don't limit the IOMMU merged segment size.
 	 */
 	dma_set_max_seg_size(dev->dev, 0xffffffff);
-	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
 
 	mutex_unlock(&dev->shutdown_lock);
 
-- 
2.37.1

Re: [PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[Sagi Grimberg]

> If swiotlb is force enabled dma_max_mapping_size ends up calling
> swiotlb_max_mapping_size which takes into account the min align
> mask for the device.
> Set the min align mask for nvme driver before calling
> dma_max_mapping_size while calculating max hw sectors.

Does this fix a specific bug? if so it needs a fixes tag so it can
go to stable backports as well.

> 
> Signed-off-by: Rishabh Bhatnagar <risbhat@amazon.com>
> ---
>   drivers/nvme/host/pci.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index 98864b853eef..30e71e41a0a2 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2834,6 +2834,8 @@ static void nvme_reset_work(struct work_struct *work)
>   		nvme_start_admin_queue(&dev->ctrl);
>   	}
>   
> +	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
> +
>   	/*
>   	 * Limit the max command size to prevent iod->sg allocations going
>   	 * over a single page.
> @@ -2846,7 +2848,6 @@ static void nvme_reset_work(struct work_struct *work)
>   	 * Don't limit the IOMMU merged segment size.
>   	 */
>   	dma_set_max_seg_size(dev->dev, 0xffffffff);
> -	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
>   
>   	mutex_unlock(&dev->shutdown_lock);
>   

Re: [PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[Christoph Hellwig]

On Wed, Sep 21, 2022 at 11:10:58AM +0300, Sagi Grimberg wrote:
>
>> If swiotlb is force enabled dma_max_mapping_size ends up calling
>> swiotlb_max_mapping_size which takes into account the min align
>> mask for the device.
>> Set the min align mask for nvme driver before calling
>> dma_max_mapping_size while calculating max hw sectors.
>
> Does this fix a specific bug? if so it needs a fixes tag so it can
> go to stable backports as well.

Hmm.  Basically this is another fixlet for nvme on swiotlb, which
already drove adding the max_mapping size and the align_mask, and
it seems we still hadn't covered all corner cases properly.  So
I think it basically has been broken since day 1, but nvme on
swiotlb only started becoming "interesting" with the trusted
hypervisor schemes.

Re: [PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[Bhatnagar, Rishabh]

On 9/21/22, 10:58 PM, "Christoph Hellwig" <hch@lst.de> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    On Wed, Sep 21, 2022 at 11:10:58AM +0300, Sagi Grimberg wrote:
    >
    >> If swiotlb is force enabled dma_max_mapping_size ends up calling
    >> swiotlb_max_mapping_size which takes into account the min align
    >> mask for the device.
    >> Set the min align mask for nvme driver before calling
    >> dma_max_mapping_size while calculating max hw sectors.
    >
    > Does this fix a specific bug? if so it needs a fixes tag so it can
    > go to stable backports as well.

    Hmm.  Basically this is another fixlet for nvme on swiotlb, which
    already drove adding the max_mapping size and the align_mask, and
    it seems we still hadn't covered all corner cases properly.  So
    I think it basically has been broken since day 1, but nvme on
    swiotlb only started becoming "interesting" with the trusted
    hypervisor schemes.

Yes this fixes a bug that we saw in nvme driver(after enabling swiotlb) that resembles what was
fixed in 82806744fd7dde603b64c151eeddaa4ee62193fd. But not setting the
min_align_mask before deriving max hw sectors was causing the bug to still exist.
Also we would probably need to backport the above fix along with (82806744fd7dde603b64c151eeddaa4ee62193fd)
to 5.10 and 5.15 kernels. Let me know what you think is the best way to achieve that.

[PATCH] nvme-pci: Set min align mask before calculating max_hw_sectors

[Rishabh Bhatnagar]

If swiotlb is force enabled dma_max_mapping_size ends up calling
swiotlb_max_mapping_size which takes into account the min align
mask for the device.
Set the min align mask for nvme driver before calling
dma_max_mapping_size while calculating max hw sectors.

Fixes: 7637de311bd2 ("nvme-pci: limit max_hw_sectors based on the DMA max
mapping size")
Signed-off-by: Rishabh Bhatnagar <risbhat@amazon.com>
---
 drivers/nvme/host/pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 98864b853eef..30e71e41a0a2 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2834,6 +2834,8 @@ static void nvme_reset_work(struct work_struct *work)
 		nvme_start_admin_queue(&dev->ctrl);
 	}
 
+	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
+
 	/*
 	 * Limit the max command size to prevent iod->sg allocations going
 	 * over a single page.
@@ -2846,7 +2848,6 @@ static void nvme_reset_work(struct work_struct *work)
 	 * Don't limit the IOMMU merged segment size.
 	 */
 	dma_set_max_seg_size(dev->dev, 0xffffffff);
-	dma_set_min_align_mask(dev->dev, NVME_CTRL_PAGE_SIZE - 1);
 
 	mutex_unlock(&dev->shutdown_lock);
 
-- 
2.37.1