* [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization
@ 2020-02-20 10:08 Max Gurtovoy
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Max Gurtovoy @ 2020-02-20 10:08 UTC (permalink / raw)
To: jgg, leon, linux-rdma; +Cc: israelr, logang, Max Gurtovoy
In case the SGL was mapped for P2P DMA operation, we must unmap it using
pci_p2pdma_unmap_sg.
Fixes: 7f73eac3a713 ("PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg()")
Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
---
drivers/infiniband/core/rw.c | 32 +++++++++++++++++++++-----------
1 file changed, 21 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 4fad732f9b3c..69513b484507 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -273,6 +273,24 @@ static int rdma_rw_init_single_wr(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
return 1;
}
+static void rdma_rw_unmap_sg(struct ib_device *dev, struct scatterlist *sg,
+ u32 sg_cnt, enum dma_data_direction dir)
+{
+ if (is_pci_p2pdma_page(sg_page(sg)))
+ pci_p2pdma_unmap_sg(dev->dma_device, sg, sg_cnt, dir);
+ else
+ ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
+}
+
+static int rdma_rw_map_sg(struct ib_device *dev, struct scatterlist *sg,
+ u32 sg_cnt, enum dma_data_direction dir)
+{
+ if (is_pci_p2pdma_page(sg_page(sg)))
+ return pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir);
+ else
+ return ib_dma_map_sg(dev, sg, sg_cnt, dir);
+}
+
/**
* rdma_rw_ctx_init - initialize a RDMA READ/WRITE context
* @ctx: context to initialize
@@ -295,11 +313,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
struct ib_device *dev = qp->pd->device;
int ret;
- if (is_pci_p2pdma_page(sg_page(sg)))
- ret = pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir);
- else
- ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
-
+ ret = rdma_rw_map_sg(dev, sg, sg_cnt, dir);
if (!ret)
return -ENOMEM;
sg_cnt = ret;
@@ -338,7 +352,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
return ret;
out_unmap_sg:
- ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
+ rdma_rw_unmap_sg(dev, sg, sg_cnt, dir);
return ret;
}
EXPORT_SYMBOL(rdma_rw_ctx_init);
@@ -588,11 +602,7 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,
break;
}
- if (is_pci_p2pdma_page(sg_page(sg)))
- pci_p2pdma_unmap_sg(qp->pd->device->dma_device, sg,
- sg_cnt, dir);
- else
- ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
+ rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
}
EXPORT_SYMBOL(rdma_rw_ctx_destroy);
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations
2020-02-20 10:08 [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Max Gurtovoy
@ 2020-02-20 10:08 ` Max Gurtovoy
2020-02-20 16:51 ` Logan Gunthorpe
2020-03-10 16:02 ` Jason Gunthorpe
2020-02-20 10:31 ` [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Leon Romanovsky
` (2 subsequent siblings)
3 siblings, 2 replies; 8+ messages in thread
From: Max Gurtovoy @ 2020-02-20 10:08 UTC (permalink / raw)
To: jgg, leon, linux-rdma; +Cc: israelr, logang, Max Gurtovoy
Since RDMA rw API support operations with P2P memory sg list, make sure
to map/unmap the scatter list for signature operation correctly.
Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
---
drivers/infiniband/core/rw.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index 69513b484507..6eba8453f206 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -392,13 +392,13 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
return -EINVAL;
}
- ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
+ ret = rdma_rw_map_sg(dev, sg, sg_cnt, dir);
if (!ret)
return -ENOMEM;
sg_cnt = ret;
if (prot_sg_cnt) {
- ret = ib_dma_map_sg(dev, prot_sg, prot_sg_cnt, dir);
+ ret = rdma_rw_map_sg(dev, prot_sg, prot_sg_cnt, dir);
if (!ret) {
ret = -ENOMEM;
goto out_unmap_sg;
@@ -467,9 +467,9 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
kfree(ctx->reg);
out_unmap_prot_sg:
if (prot_sg_cnt)
- ib_dma_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
+ rdma_rw_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
out_unmap_sg:
- ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
+ rdma_rw_unmap_sg(dev, sg, sg_cnt, dir);
return ret;
}
EXPORT_SYMBOL(rdma_rw_ctx_signature_init);
@@ -629,9 +629,9 @@ void rdma_rw_ctx_destroy_signature(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
ib_mr_pool_put(qp, &qp->sig_mrs, ctx->reg->mr);
kfree(ctx->reg);
- ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
if (prot_sg_cnt)
- ib_dma_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
+ rdma_rw_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
+ rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
}
EXPORT_SYMBOL(rdma_rw_ctx_destroy_signature);
--
2.20.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
@ 2020-02-20 16:51 ` Logan Gunthorpe
2020-02-20 16:57 ` Max Gurtovoy
2020-03-10 16:02 ` Jason Gunthorpe
1 sibling, 1 reply; 8+ messages in thread
From: Logan Gunthorpe @ 2020-02-20 16:51 UTC (permalink / raw)
To: Max Gurtovoy, jgg, leon, linux-rdma; +Cc: israelr
On 2020-02-20 3:08 a.m., Max Gurtovoy wrote:
> Since RDMA rw API support operations with P2P memory sg list, make sure
> to map/unmap the scatter list for signature operation correctly.
Does anyone actually use P2P pages with rdma_rw_ctx_signature_init()?
Logan
> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
> ---
> drivers/infiniband/core/rw.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
> index 69513b484507..6eba8453f206 100644
> --- a/drivers/infiniband/core/rw.c
> +++ b/drivers/infiniband/core/rw.c
> @@ -392,13 +392,13 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> return -EINVAL;
> }
>
> - ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
> + ret = rdma_rw_map_sg(dev, sg, sg_cnt, dir);
> if (!ret)
> return -ENOMEM;
> sg_cnt = ret;
>
> if (prot_sg_cnt) {
> - ret = ib_dma_map_sg(dev, prot_sg, prot_sg_cnt, dir);
> + ret = rdma_rw_map_sg(dev, prot_sg, prot_sg_cnt, dir);
> if (!ret) {
> ret = -ENOMEM;
> goto out_unmap_sg;
> @@ -467,9 +467,9 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> kfree(ctx->reg);
> out_unmap_prot_sg:
> if (prot_sg_cnt)
> - ib_dma_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
> + rdma_rw_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
> out_unmap_sg:
> - ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
> + rdma_rw_unmap_sg(dev, sg, sg_cnt, dir);
> return ret;
> }
> EXPORT_SYMBOL(rdma_rw_ctx_signature_init);
> @@ -629,9 +629,9 @@ void rdma_rw_ctx_destroy_signature(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> ib_mr_pool_put(qp, &qp->sig_mrs, ctx->reg->mr);
> kfree(ctx->reg);
>
> - ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
> if (prot_sg_cnt)
> - ib_dma_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
> + rdma_rw_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
> + rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
> }
> EXPORT_SYMBOL(rdma_rw_ctx_destroy_signature);
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations
2020-02-20 16:51 ` Logan Gunthorpe
@ 2020-02-20 16:57 ` Max Gurtovoy
0 siblings, 0 replies; 8+ messages in thread
From: Max Gurtovoy @ 2020-02-20 16:57 UTC (permalink / raw)
To: Logan Gunthorpe, jgg, leon, linux-rdma; +Cc: israelr
On 2/20/2020 6:51 PM, Logan Gunthorpe wrote:
>
> On 2020-02-20 3:08 a.m., Max Gurtovoy wrote:
>> Since RDMA rw API support operations with P2P memory sg list, make sure
>> to map/unmap the scatter list for signature operation correctly.
> Does anyone actually use P2P pages with rdma_rw_ctx_signature_init()?
We're adding support for NVMf/target in these days.
Nevertheless, the RW API defined as an API that might use P2P pages.
>
> Logan
>
>> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
>> ---
>> drivers/infiniband/core/rw.c | 12 ++++++------
>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
>> index 69513b484507..6eba8453f206 100644
>> --- a/drivers/infiniband/core/rw.c
>> +++ b/drivers/infiniband/core/rw.c
>> @@ -392,13 +392,13 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
>> return -EINVAL;
>> }
>>
>> - ret = ib_dma_map_sg(dev, sg, sg_cnt, dir);
>> + ret = rdma_rw_map_sg(dev, sg, sg_cnt, dir);
>> if (!ret)
>> return -ENOMEM;
>> sg_cnt = ret;
>>
>> if (prot_sg_cnt) {
>> - ret = ib_dma_map_sg(dev, prot_sg, prot_sg_cnt, dir);
>> + ret = rdma_rw_map_sg(dev, prot_sg, prot_sg_cnt, dir);
>> if (!ret) {
>> ret = -ENOMEM;
>> goto out_unmap_sg;
>> @@ -467,9 +467,9 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
>> kfree(ctx->reg);
>> out_unmap_prot_sg:
>> if (prot_sg_cnt)
>> - ib_dma_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
>> + rdma_rw_unmap_sg(dev, prot_sg, prot_sg_cnt, dir);
>> out_unmap_sg:
>> - ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
>> + rdma_rw_unmap_sg(dev, sg, sg_cnt, dir);
>> return ret;
>> }
>> EXPORT_SYMBOL(rdma_rw_ctx_signature_init);
>> @@ -629,9 +629,9 @@ void rdma_rw_ctx_destroy_signature(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
>> ib_mr_pool_put(qp, &qp->sig_mrs, ctx->reg->mr);
>> kfree(ctx->reg);
>>
>> - ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
>> if (prot_sg_cnt)
>> - ib_dma_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
>> + rdma_rw_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir);
>> + rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir);
>> }
>> EXPORT_SYMBOL(rdma_rw_ctx_destroy_signature);
>>
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
2020-02-20 16:51 ` Logan Gunthorpe
@ 2020-03-10 16:02 ` Jason Gunthorpe
1 sibling, 0 replies; 8+ messages in thread
From: Jason Gunthorpe @ 2020-03-10 16:02 UTC (permalink / raw)
To: Max Gurtovoy; +Cc: leon, linux-rdma, israelr, logang
On Thu, Feb 20, 2020 at 12:08:19PM +0200, Max Gurtovoy wrote:
> Since RDMA rw API support operations with P2P memory sg list, make sure
> to map/unmap the scatter list for signature operation correctly.
>
> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
> ---
> drivers/infiniband/core/rw.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
Applied to for-next
Thanks,
Jason
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization
2020-02-20 10:08 [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Max Gurtovoy
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
@ 2020-02-20 10:31 ` Leon Romanovsky
2020-02-20 16:50 ` Logan Gunthorpe
2020-02-20 17:21 ` Jason Gunthorpe
3 siblings, 0 replies; 8+ messages in thread
From: Leon Romanovsky @ 2020-02-20 10:31 UTC (permalink / raw)
To: Max Gurtovoy; +Cc: jgg, linux-rdma, israelr, logang
On Thu, Feb 20, 2020 at 12:08:18PM +0200, Max Gurtovoy wrote:
> In case the SGL was mapped for P2P DMA operation, we must unmap it using
> pci_p2pdma_unmap_sg.
>
> Fixes: 7f73eac3a713 ("PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg()")
> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
> ---
> drivers/infiniband/core/rw.c | 32 +++++++++++++++++++++-----------
> 1 file changed, 21 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
> index 4fad732f9b3c..69513b484507 100644
> --- a/drivers/infiniband/core/rw.c
> +++ b/drivers/infiniband/core/rw.c
> @@ -273,6 +273,24 @@ static int rdma_rw_init_single_wr(struct rdma_rw_ctx *ctx, struct ib_qp *qp,
> return 1;
> }
>
> +static void rdma_rw_unmap_sg(struct ib_device *dev, struct scatterlist *sg,
> + u32 sg_cnt, enum dma_data_direction dir)
> +{
> + if (is_pci_p2pdma_page(sg_page(sg)))
> + pci_p2pdma_unmap_sg(dev->dma_device, sg, sg_cnt, dir);
> + else
> + ib_dma_unmap_sg(dev, sg, sg_cnt, dir);
> +}
> +
> +static int rdma_rw_map_sg(struct ib_device *dev, struct scatterlist *sg,
> + u32 sg_cnt, enum dma_data_direction dir)
> +{
> + if (is_pci_p2pdma_page(sg_page(sg)))
> + return pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir);
> + else
This "else" is not needed.
> + return ib_dma_map_sg(dev, sg, sg_cnt, dir);
> +}> +
> /**
Thanks,
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization
2020-02-20 10:08 [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Max Gurtovoy
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
2020-02-20 10:31 ` [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Leon Romanovsky
@ 2020-02-20 16:50 ` Logan Gunthorpe
2020-02-20 17:21 ` Jason Gunthorpe
3 siblings, 0 replies; 8+ messages in thread
From: Logan Gunthorpe @ 2020-02-20 16:50 UTC (permalink / raw)
To: Max Gurtovoy, jgg, leon, linux-rdma; +Cc: israelr
On 2020-02-20 3:08 a.m., Max Gurtovoy wrote:
> In case the SGL was mapped for P2P DMA operation, we must unmap it using
> pci_p2pdma_unmap_sg.
>
> Fixes: 7f73eac3a713 ("PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg()")
> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
Makes sense to me.
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Thanks!
Logan
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization
2020-02-20 10:08 [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Max Gurtovoy
` (2 preceding siblings ...)
2020-02-20 16:50 ` Logan Gunthorpe
@ 2020-02-20 17:21 ` Jason Gunthorpe
3 siblings, 0 replies; 8+ messages in thread
From: Jason Gunthorpe @ 2020-02-20 17:21 UTC (permalink / raw)
To: Max Gurtovoy; +Cc: leon, linux-rdma, israelr, logang
On Thu, Feb 20, 2020 at 12:08:18PM +0200, Max Gurtovoy wrote:
> In case the SGL was mapped for P2P DMA operation, we must unmap it using
> pci_p2pdma_unmap_sg.
>
> Fixes: 7f73eac3a713 ("PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg()")
> Signed-off-by: Max Gurtovoy <maxg@mellanox.com>
> Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
> Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
> ---
> drivers/infiniband/core/rw.c | 32 +++++++++++++++++++++-----------
> 1 file changed, 21 insertions(+), 11 deletions(-)
Applied to for-rc. The other one can go to -next in a bit as there is
no current bug
Jason
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-03-10 16:02 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-02-20 10:08 [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Max Gurtovoy
2020-02-20 10:08 ` [PATCH v2 2/2] RDMA/rw: map P2P memory correctly for signature operations Max Gurtovoy
2020-02-20 16:51 ` Logan Gunthorpe
2020-02-20 16:57 ` Max Gurtovoy
2020-03-10 16:02 ` Jason Gunthorpe
2020-02-20 10:31 ` [PATCH v2 1/2] RDMA/rw: fix error flow during RDMA context initialization Leon Romanovsky
2020-02-20 16:50 ` Logan Gunthorpe
2020-02-20 17:21 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).