* [PATCH] mlx4: Limit num of fast reg WRs
@ 2010-10-07 14:24 Eli Cohen
2010-10-11 21:37 ` Roland Dreier
2010-10-11 22:13 ` Or Gerlitz
0 siblings, 2 replies; 6+ messages in thread
From: Eli Cohen @ 2010-10-07 14:24 UTC (permalink / raw)
To: Roland Dreier; +Cc: RDMA list
Fix the limit of max fast regisreation WRs that can be posted to CX to match
hardware capabilities.
Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
drivers/infiniband/hw/mlx4/main.c | 2 +-
drivers/infiniband/hw/mlx4/mr.c | 2 +-
include/linux/mlx4/device.h | 4 ++++
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 4e94e36..829225d 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -135,7 +135,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
props->max_srq = dev->dev->caps.num_srqs - dev->dev->caps.reserved_srqs;
props->max_srq_wr = dev->dev->caps.max_srq_wqes - 1;
props->max_srq_sge = dev->dev->caps.max_srq_sge;
- props->max_fast_reg_page_list_len = PAGE_SIZE / sizeof (u64);
+ props->max_fast_reg_page_list_len = MAX_FAST_REG_PAGES;
props->local_ca_ack_delay = dev->dev->caps.local_ca_ack_delay;
props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ?
IB_ATOMIC_HCA : IB_ATOMIC_NONE;
diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
index 1d27b9a..83e3cc7 100644
--- a/drivers/infiniband/hw/mlx4/mr.c
+++ b/drivers/infiniband/hw/mlx4/mr.c
@@ -226,7 +226,7 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
struct mlx4_ib_fast_reg_page_list *mfrpl;
int size = page_list_len * sizeof (u64);
- if (size > PAGE_SIZE)
+ if (page_list_len > MAX_FAST_REG_PAGES)
return ERR_PTR(-EINVAL);
mfrpl = kmalloc(sizeof *mfrpl, GFP_KERNEL);
diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 7a7f9c1..69fe3f7 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -171,6 +171,10 @@ enum {
MLX4_NUM_FEXCH = 64 * 1024,
};
+enum {
+ MAX_FAST_REG_PAGES = 511,
+};
+
static inline u64 mlx4_fw_ver(u64 major, u64 minor, u64 subminor)
{
return (major << 32) | (minor << 16) | subminor;
--
1.7.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs
2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen
@ 2010-10-11 21:37 ` Roland Dreier
2010-10-11 22:13 ` Or Gerlitz
1 sibling, 0 replies; 6+ messages in thread
From: Roland Dreier @ 2010-10-11 21:37 UTC (permalink / raw)
To: Eli Cohen; +Cc: RDMA list
thanks, applied (with s/MAX_FAST_REG_PAGES/MLX4_MAX_FAST_REG_PAGES/ to
avoid namespace pollution).
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs
2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen
2010-10-11 21:37 ` Roland Dreier
@ 2010-10-11 22:13 ` Or Gerlitz
[not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
1 sibling, 1 reply; 6+ messages in thread
From: Or Gerlitz @ 2010-10-11 22:13 UTC (permalink / raw)
To: Eli Cohen; +Cc: Roland Dreier, RDMA list
Eli Cohen <eli-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Fix the limit of max fast regisreation WRs that can be posted to CX to match
> hardware capabilities.
Guys, can you clarify if the hardware limitation is 511 entries or its
(PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can
change if the page size gets bigger or smaller?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs
[not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2010-10-12 8:19 ` Eli Cohen
2010-10-12 20:37 ` Roland Dreier
0 siblings, 1 reply; 6+ messages in thread
From: Eli Cohen @ 2010-10-12 8:19 UTC (permalink / raw)
To: Or Gerlitz; +Cc: Roland Dreier, RDMA list
On Tue, Oct 12, 2010 at 12:13:26AM +0200, Or Gerlitz wrote:
> Guys, can you clarify if the hardware limitation is 511 entries or its
> (PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can
> change if the page size gets bigger or smaller?
>
The limit is 511 entries.
After I posted this patch, I was told that there is yet another
constraint on the page list: The buffer containing the list must not
cross a page boundary. So I was thinking what is the best way to deal
with this. One way is to always allocate a whole page and map it using
dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete
patch, just the idea).
diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
index 83e3cc7..e9b2c8a 100644
--- a/drivers/infiniband/hw/mlx4/mr.c
+++ b/drivers/infiniband/hw/mlx4/mr.c
@@ -237,18 +237,23 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
if (!mfrpl->ibfrpl.page_list)
goto err_free;
- mfrpl->mapped_page_list = dma_alloc_coherent(&dev->dev->pdev->dev,
- size, &mfrpl->map,
- GFP_KERNEL);
+ mfrpl->mapped_page_list = (__be64 *)__get_free_page(GFP_KERNEL);
if (!mfrpl->mapped_page_list)
goto err_free;
- WARN_ON(mfrpl->map & 0x3f);
+ mfrpl->map = dma_map_single(ibdev->dma_device, mfrpl->mapped_page_list,
+ PAGE_SIZE, DMA_TO_DEVICE);
+ if (dma_mapping_error(ibdev->dma_device, mfrpl->map))
+ goto err_page;
+
+
return &mfrpl->ibfrpl;
+err_page:
+ free_page((unsigned long) mfrpl->mapped_page_list);
+
err_free:
- kfree(mfrpl->ibfrpl.page_list);
kfree(mfrpl);
return ERR_PTR(-ENOMEM);
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs
2010-10-12 8:19 ` Eli Cohen
@ 2010-10-12 20:37 ` Roland Dreier
[not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Roland Dreier @ 2010-10-12 20:37 UTC (permalink / raw)
To: Eli Cohen; +Cc: Or Gerlitz, RDMA list
> After I posted this patch, I was told that there is yet another
> constraint on the page list: The buffer containing the list must not
> cross a page boundary. So I was thinking what is the best way to deal
> with this. One way is to always allocate a whole page and map it using
> dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete
> patch, just the idea).
Is there any chance of the dma_alloc_coherent() in the current code
allocating memory that crosses a page boundary?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs
[not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
@ 2010-10-12 20:44 ` Eli Cohen
0 siblings, 0 replies; 6+ messages in thread
From: Eli Cohen @ 2010-10-12 20:44 UTC (permalink / raw)
To: Roland Dreier; +Cc: Or Gerlitz, RDMA list
On Tue, Oct 12, 2010 at 01:37:37PM -0700, Roland Dreier wrote:
>
> Is there any chance of the dma_alloc_coherent() in the current code
> allocating memory that crosses a page boundary?
>
You mean that the allocation is aligned at least to its size? I could
not find any commitment to this anywhere.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-10-12 20:44 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen
2010-10-11 21:37 ` Roland Dreier
2010-10-11 22:13 ` Or Gerlitz
[not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2010-10-12 8:19 ` Eli Cohen
2010-10-12 20:37 ` Roland Dreier
[not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2010-10-12 20:44 ` Eli Cohen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox