[PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Sagi Grimberg]

Their completely logical, so don't pose an artificial limitation.

Signed-off-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
---
 drivers/infiniband/sw/rxe/rxe_param.h | 1 -
 drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h
index 13ed2cc6eaa2..1b596fbbe251 100644
--- a/drivers/infiniband/sw/rxe/rxe_param.h
+++ b/drivers/infiniband/sw/rxe/rxe_param.h
@@ -114,7 +114,6 @@ enum rxe_device_param {
 	RXE_MAX_UCONTEXT		= 512,
 
 	RXE_NUM_PORT			= 1,
-	RXE_NUM_COMP_VECTORS		= 1,
 
 	RXE_MIN_QP_INDEX		= 16,
 	RXE_MAX_QP_INDEX		= 0x00020000,
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 4c9080c94c06..3be4cbe365c6 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1237,7 +1237,7 @@ int rxe_register_device(struct rxe_dev *rxe)
 	dev->owner = THIS_MODULE;
 	dev->node_type = RDMA_NODE_IB_CA;
 	dev->phys_port_cnt = 1;
-	dev->num_comp_vectors = RXE_NUM_COMP_VECTORS;
+	dev->num_comp_vectors = num_possible_cpus();
 	dev->dev.parent = rxe_dma_device(rxe);
 	dev->local_dma_lkey = 0;
 	dev->node_guid = rxe_node_guid(rxe);
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

On Thu, May 04, 2017 at 04:23:07PM +0300, Sagi Grimberg wrote:
> Their completely logical, so don't pose an artificial limitation.
>
> Signed-off-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> ---

Thanks a lot,
Reviewed-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Moni Shoua]

> +       dev->num_comp_vectors = num_possible_cpus();
why not num_online_cpus()?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Sagi Grimberg]

>> +       dev->num_comp_vectors = num_possible_cpus();
> why not num_online_cpus()?

Because that can change on the fly and should not effect on
the number of logical completion vectors rxe exposes.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Moni Shoua]

> Because that can change on the fly and should not effect on
> the number of logical completion vectors rxe exposes.
>
> --
BTW, what will be the affect of this if rxe driver doesn't use the
comp_vector parameter in create_cq()?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Sagi Grimberg]

>> Because that can change on the fly and should not effect on
>> the number of logical completion vectors rxe exposes.
>>
>> --
> BTW, what will be the affect of this if rxe driver doesn't use the
> comp_vector parameter in create_cq()?

Some ULPs won't create more queue-pairs than what the rdma device
supports (per multi-queue session). One example for that exists in a
patch set I submitted for exposing to ULPs IRQ affinity mappings
through the RDMA core so they in turn can make smart assignment
decisions for queue<->cpu mappings in a multi-queue environment.

I think its safe to assume that any modern HCA drivers supports a fair
share of completion vectors.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Moni Shoua]

So you say that even if rxe driver doesn't do anything with the
comp_vector parameter  passed to create cq, a ULP can benefit from rxe
reporting num_comp_vectors = number of cpus.
I can agree with that even though that IMHO the benefit deepens on how
the real eth device handles received packets.

Acked-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

thanks




On Thu, May 4, 2017 at 5:26 PM, Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> wrote:
>
>>> Because that can change on the fly and should not effect on
>>> the number of logical completion vectors rxe exposes.
>>>
>>> --
>>
>> BTW, what will be the affect of this if rxe driver doesn't use the
>> comp_vector parameter in create_cq()?
>
>
> Some ULPs won't create more queue-pairs than what the rdma device
> supports (per multi-queue session). One example for that exists in a
> patch set I submitted for exposing to ULPs IRQ affinity mappings
> through the RDMA core so they in turn can make smart assignment
> decisions for queue<->cpu mappings in a multi-queue environment.
>
> I think its safe to assume that any modern HCA drivers supports a fair
> share of completion vectors.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rxe: expose num_possible_cpus() cnum_comp_vectors

[Doug Ledford]

On Thu, 2017-05-04 at 16:23 +0300, Sagi Grimberg wrote:
> Their completely logical, so don't pose an artificial limitation.
> 
> Signed-off-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>

*They're/*impose.  Fixed those up during commit, applied for second
pull request, thanks.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
   
Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html