* [PATCH] mlx4: Limit num of fast reg WRs @ 2010-10-07 14:24 Eli Cohen 2010-10-11 21:37 ` Roland Dreier 2010-10-11 22:13 ` Or Gerlitz 0 siblings, 2 replies; 6+ messages in thread From: Eli Cohen @ 2010-10-07 14:24 UTC (permalink / raw) To: Roland Dreier; +Cc: RDMA list Fix the limit of max fast regisreation WRs that can be posted to CX to match hardware capabilities. Signed-off-by: Eli Cohen <eli-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org> --- drivers/infiniband/hw/mlx4/main.c | 2 +- drivers/infiniband/hw/mlx4/mr.c | 2 +- include/linux/mlx4/device.h | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 4e94e36..829225d 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -135,7 +135,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, props->max_srq = dev->dev->caps.num_srqs - dev->dev->caps.reserved_srqs; props->max_srq_wr = dev->dev->caps.max_srq_wqes - 1; props->max_srq_sge = dev->dev->caps.max_srq_sge; - props->max_fast_reg_page_list_len = PAGE_SIZE / sizeof (u64); + props->max_fast_reg_page_list_len = MAX_FAST_REG_PAGES; props->local_ca_ack_delay = dev->dev->caps.local_ca_ack_delay; props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ? IB_ATOMIC_HCA : IB_ATOMIC_NONE; diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index 1d27b9a..83e3cc7 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -226,7 +226,7 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device struct mlx4_ib_fast_reg_page_list *mfrpl; int size = page_list_len * sizeof (u64); - if (size > PAGE_SIZE) + if (page_list_len > MAX_FAST_REG_PAGES) return ERR_PTR(-EINVAL); mfrpl = kmalloc(sizeof *mfrpl, GFP_KERNEL); diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 7a7f9c1..69fe3f7 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -171,6 +171,10 @@ enum { MLX4_NUM_FEXCH = 64 * 1024, }; +enum { + MAX_FAST_REG_PAGES = 511, +}; + static inline u64 mlx4_fw_ver(u64 major, u64 minor, u64 subminor) { return (major << 32) | (minor << 16) | subminor; -- 1.7.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs 2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen @ 2010-10-11 21:37 ` Roland Dreier 2010-10-11 22:13 ` Or Gerlitz 1 sibling, 0 replies; 6+ messages in thread From: Roland Dreier @ 2010-10-11 21:37 UTC (permalink / raw) To: Eli Cohen; +Cc: RDMA list thanks, applied (with s/MAX_FAST_REG_PAGES/MLX4_MAX_FAST_REG_PAGES/ to avoid namespace pollution). -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs 2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen 2010-10-11 21:37 ` Roland Dreier @ 2010-10-11 22:13 ` Or Gerlitz [not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 1 reply; 6+ messages in thread From: Or Gerlitz @ 2010-10-11 22:13 UTC (permalink / raw) To: Eli Cohen; +Cc: Roland Dreier, RDMA list Eli Cohen <eli-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote: > Fix the limit of max fast regisreation WRs that can be posted to CX to match > hardware capabilities. Guys, can you clarify if the hardware limitation is 511 entries or its (PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can change if the page size gets bigger or smaller? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH] mlx4: Limit num of fast reg WRs [not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2010-10-12 8:19 ` Eli Cohen 2010-10-12 20:37 ` Roland Dreier 0 siblings, 1 reply; 6+ messages in thread From: Eli Cohen @ 2010-10-12 8:19 UTC (permalink / raw) To: Or Gerlitz; +Cc: Roland Dreier, RDMA list On Tue, Oct 12, 2010 at 12:13:26AM +0200, Or Gerlitz wrote: > Guys, can you clarify if the hardware limitation is 511 entries or its > (PAGE_SIZE / sizeof(pointer)) - 1 which is 4096 / 8 - 1 = 511 but can > change if the page size gets bigger or smaller? > The limit is 511 entries. After I posted this patch, I was told that there is yet another constraint on the page list: The buffer containing the list must not cross a page boundary. So I was thinking what is the best way to deal with this. One way is to always allocate a whole page and map it using dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete patch, just the idea). diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index 83e3cc7..e9b2c8a 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -237,18 +237,23 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device if (!mfrpl->ibfrpl.page_list) goto err_free; - mfrpl->mapped_page_list = dma_alloc_coherent(&dev->dev->pdev->dev, - size, &mfrpl->map, - GFP_KERNEL); + mfrpl->mapped_page_list = (__be64 *)__get_free_page(GFP_KERNEL); if (!mfrpl->mapped_page_list) goto err_free; - WARN_ON(mfrpl->map & 0x3f); + mfrpl->map = dma_map_single(ibdev->dma_device, mfrpl->mapped_page_list, + PAGE_SIZE, DMA_TO_DEVICE); + if (dma_mapping_error(ibdev->dma_device, mfrpl->map)) + goto err_page; + + return &mfrpl->ibfrpl; +err_page: + free_page((unsigned long) mfrpl->mapped_page_list); + err_free: - kfree(mfrpl->ibfrpl.page_list); kfree(mfrpl); return ERR_PTR(-ENOMEM); } -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] mlx4: Limit num of fast reg WRs 2010-10-12 8:19 ` Eli Cohen @ 2010-10-12 20:37 ` Roland Dreier [not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Roland Dreier @ 2010-10-12 20:37 UTC (permalink / raw) To: Eli Cohen; +Cc: Or Gerlitz, RDMA list > After I posted this patch, I was told that there is yet another > constraint on the page list: The buffer containing the list must not > cross a page boundary. So I was thinking what is the best way to deal > with this. One way is to always allocate a whole page and map it using > dma_map_page(page, DMA_TO_DEVICE), something like this (not a complete > patch, just the idea). Is there any chance of the dma_alloc_coherent() in the current code allocating memory that crosses a page boundary? - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH] mlx4: Limit num of fast reg WRs [not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> @ 2010-10-12 20:44 ` Eli Cohen 0 siblings, 0 replies; 6+ messages in thread From: Eli Cohen @ 2010-10-12 20:44 UTC (permalink / raw) To: Roland Dreier; +Cc: Or Gerlitz, RDMA list On Tue, Oct 12, 2010 at 01:37:37PM -0700, Roland Dreier wrote: > > Is there any chance of the dma_alloc_coherent() in the current code > allocating memory that crosses a page boundary? > You mean that the allocation is aligned at least to its size? I could not find any commitment to this anywhere. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-10-12 20:44 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-10-07 14:24 [PATCH] mlx4: Limit num of fast reg WRs Eli Cohen
2010-10-11 21:37 ` Roland Dreier
2010-10-11 22:13 ` Or Gerlitz
[not found] ` <AANLkTinKiBHOT2sSA+_s5CGhqC8nEoDTp2_pGmo6MrQp-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2010-10-12 8:19 ` Eli Cohen
2010-10-12 20:37 ` Roland Dreier
[not found] ` <adad3rf171q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
2010-10-12 20:44 ` Eli Cohen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox