* [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
@ 2021-12-18 20:44 yanjun.zhu
2021-12-19 7:11 ` Mark Zhang
0 siblings, 1 reply; 5+ messages in thread
From: yanjun.zhu @ 2021-12-18 20:44 UTC (permalink / raw)
To: mustafa.ismail, shiraz.saleem, jgg, linux-rdma, yanjun.zhu
From: Zhu Yanjun <yanjun.zhu@linux.dev>
Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
get the source udp port number for a QP based on the grh.flow_label or
lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp
source port when set path") is a standard way. So it is also adopted in
this commit.
Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
V1->V2: Adopt a standard method to get udp source port.
---
drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
index 8cd5f9261692..9fe73d1db0d9 100644
--- a/drivers/infiniband/hw/irdma/verbs.c
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
return 0;
}
+
+static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+ if (!fl)
+ fl = rdma_calc_flow_label(lqpn, rqpn);
+
+ return rdma_flow_label_to_udp_sport(fl);
+}
+
/**
* irdma_modify_qp_roce - modify qp request
* @ibqp: qp's pointer for modify
@@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
+ udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
+ ibqp->qp_num,
+ roce_info->dest_qp);
+
if (attr_mask & IB_QP_AV) {
struct irdma_av *av = &iwqp->roce_ah.av;
const struct ib_gid_attr *sgid_attr;
--
2.27.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
2021-12-18 20:44 [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary yanjun.zhu
@ 2021-12-19 7:11 ` Mark Zhang
2021-12-19 8:20 ` Yanjun Zhu
0 siblings, 1 reply; 5+ messages in thread
From: Mark Zhang @ 2021-12-19 7:11 UTC (permalink / raw)
To: yanjun.zhu, mustafa.ismail, shiraz.saleem, jgg, linux-rdma
On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote:
> External email: Use caution opening links or attachments
>
>
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>
> Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html,
> get the source udp port number for a QP based on the grh.flow_label or
> lqpn/rqpn. This provides a better spread of traffic across NIC RX queues.
> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp
> source port when set path") is a standard way. So it is also adopted in
> this commit.
>
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> ---
> V1->V2: Adopt a standard method to get udp source port.
> ---
> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
> index 8cd5f9261692..9fe73d1db0d9 100644
> --- a/drivers/infiniband/hw/irdma/verbs.c
> +++ b/drivers/infiniband/hw/irdma/verbs.c
> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index,
> return 0;
> }
>
> +
> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
> +{
> + if (!fl)
> + fl = rdma_calc_flow_label(lqpn, rqpn);
> +
> + return rdma_flow_label_to_udp_sport(fl);
> +}
> +
> /**
> * irdma_modify_qp_roce - modify qp request
> * @ibqp: qp's pointer for modify
> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
>
> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
>
> + udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
> + ibqp->qp_num,
> + roce_info->dest_qp);
> +
FYI that in mlx5 driver the udp_sport is only set when address vector
and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.
> if (attr_mask & IB_QP_AV) {
> struct irdma_av *av = &iwqp->roce_ah.av;
> const struct ib_gid_attr *sgid_attr;
> --
> 2.27.0
>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
2021-12-19 7:11 ` Mark Zhang
@ 2021-12-19 8:20 ` Yanjun Zhu
2021-12-20 17:54 ` Saleem, Shiraz
0 siblings, 1 reply; 5+ messages in thread
From: Yanjun Zhu @ 2021-12-19 8:20 UTC (permalink / raw)
To: Mark Zhang, mustafa.ismail, shiraz.saleem, jgg, linux-rdma
在 2021/12/19 15:11, Mark Zhang 写道:
> On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> Based on the link
>> https://www.spinics.net/lists/linux-rdma/msg73735.html,
>> get the source udp port number for a QP based on the grh.flow_label or
>> lqpn/rqpn. This provides a better spread of traffic across NIC RX
>> queues.
>> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp
>> source port when set path") is a standard way. So it is also adopted in
>> this commit.
>>
>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>> ---
>> V1->V2: Adopt a standard method to get udp source port.
>> ---
>> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
>> 1 file changed, 13 insertions(+)
>>
>> diff --git a/drivers/infiniband/hw/irdma/verbs.c
>> b/drivers/infiniband/hw/irdma/verbs.c
>> index 8cd5f9261692..9fe73d1db0d9 100644
>> --- a/drivers/infiniband/hw/irdma/verbs.c
>> +++ b/drivers/infiniband/hw/irdma/verbs.c
>> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device
>> *ibdev, u32 port, u16 index,
>> return 0;
>> }
>>
>> +
>> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
>> +{
>> + if (!fl)
>> + fl = rdma_calc_flow_label(lqpn, rqpn);
>> +
>> + return rdma_flow_label_to_udp_sport(fl);
>> +}
>> +
>> /**
>> * irdma_modify_qp_roce - modify qp request
>> * @ibqp: qp's pointer for modify
>> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,
>> struct ib_qp_attr *attr,
>>
>> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
>>
>> + udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label,
>> + ibqp->qp_num,
>> + roce_info->dest_qp);
>> +
>
> FYI that in mlx5 driver the udp_sport is only set when address vector
> and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.
Sure. From my tests, when this function irdma_get_udp_sport is called,
dest qpn is ready.
Zhu Yanjun
>
>> if (attr_mask & IB_QP_AV) {
>> struct irdma_av *av = &iwqp->roce_ah.av;
>> const struct ib_gid_attr *sgid_attr;
>> --
>> 2.27.0
>>
>
^ permalink raw reply [flat|nested] 5+ messages in thread* RE: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
2021-12-19 8:20 ` Yanjun Zhu
@ 2021-12-20 17:54 ` Saleem, Shiraz
2021-12-21 13:43 ` Yanjun Zhu
0 siblings, 1 reply; 5+ messages in thread
From: Saleem, Shiraz @ 2021-12-20 17:54 UTC (permalink / raw)
To: Yanjun Zhu, Mark Zhang, Ismail, Mustafa, jgg@ziepe.ca,
linux-rdma@vger.kernel.org
> Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
>
>
> 在 2021/12/19 15:11, Mark Zhang 写道:
> > On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote:
> >> External email: Use caution opening links or attachments
> >>
> >>
> >> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> >>
> >> Based on the link
> >> https://www.spinics.net/lists/linux-rdma/msg73735.html,
> >> get the source udp port number for a QP based on the grh.flow_label
> >> or lqpn/rqpn. This provides a better spread of traffic across NIC RX
> >> queues.
> >> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp
> >> source port when set path") is a standard way. So it is also adopted
> >> in this commit.
> >>
> >> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> >> ---
> >> V1->V2: Adopt a standard method to get udp source port.
> >> ---
> >> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
> >> 1 file changed, 13 insertions(+)
> >>
> >> diff --git a/drivers/infiniband/hw/irdma/verbs.c
> >> b/drivers/infiniband/hw/irdma/verbs.c
> >> index 8cd5f9261692..9fe73d1db0d9 100644
> >> --- a/drivers/infiniband/hw/irdma/verbs.c
> >> +++ b/drivers/infiniband/hw/irdma/verbs.c
> >> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device
> >> *ibdev, u32 port, u16 index,
> >> return 0;
> >> }
> >>
> >> +
> >> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
> >> + if (!fl)
> >> + fl = rdma_calc_flow_label(lqpn, rqpn);
> >> +
> >> + return rdma_flow_label_to_udp_sport(fl); }
> >> +
> >> /**
> >> * irdma_modify_qp_roce - modify qp request
> >> * @ibqp: qp's pointer for modify
> >> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,
> >> struct ib_qp_attr *attr,
> >>
> >> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
> >>
> >> + udp_info->src_port =
> >> +irdma_get_udp_sport(udp_info->flow_label,
> >> + ibqp->qp_num,
> >> + roce_info->dest_qp);
> >> +
> >
> > FYI that in mlx5 driver the udp_sport is only set when address vector
> > and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.
>
> Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn
> is ready.
>
I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags.
So, in this block,
https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171
Shiraz
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
2021-12-20 17:54 ` Saleem, Shiraz
@ 2021-12-21 13:43 ` Yanjun Zhu
0 siblings, 0 replies; 5+ messages in thread
From: Yanjun Zhu @ 2021-12-21 13:43 UTC (permalink / raw)
To: Saleem, Shiraz, Yanjun Zhu, Mark Zhang, Ismail, Mustafa,
jgg@ziepe.ca, linux-rdma@vger.kernel.org
在 2021/12/21 1:54, Saleem, Shiraz 写道:
>> Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary
>>
>>
>> 在 2021/12/19 15:11, Mark Zhang 写道:
>>> On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote:
>>>> External email: Use caution opening links or attachments
>>>>
>>>>
>>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>>
>>>> Based on the link
>>>> https://www.spinics.net/lists/linux-rdma/msg73735.html,
>>>> get the source udp port number for a QP based on the grh.flow_label
>>>> or lqpn/rqpn. This provides a better spread of traffic across NIC RX
>>>> queues.
>>>> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp
>>>> source port when set path") is a standard way. So it is also adopted
>>>> in this commit.
>>>>
>>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>> ---
>>>> V1->V2: Adopt a standard method to get udp source port.
>>>> ---
>>>> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++
>>>> 1 file changed, 13 insertions(+)
>>>>
>>>> diff --git a/drivers/infiniband/hw/irdma/verbs.c
>>>> b/drivers/infiniband/hw/irdma/verbs.c
>>>> index 8cd5f9261692..9fe73d1db0d9 100644
>>>> --- a/drivers/infiniband/hw/irdma/verbs.c
>>>> +++ b/drivers/infiniband/hw/irdma/verbs.c
>>>> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device
>>>> *ibdev, u32 port, u16 index,
>>>> return 0;
>>>> }
>>>>
>>>> +
>>>> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) {
>>>> + if (!fl)
>>>> + fl = rdma_calc_flow_label(lqpn, rqpn);
>>>> +
>>>> + return rdma_flow_label_to_udp_sport(fl); }
>>>> +
>>>> /**
>>>> * irdma_modify_qp_roce - modify qp request
>>>> * @ibqp: qp's pointer for modify
>>>> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp,
>>>> struct ib_qp_attr *attr,
>>>>
>>>> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id;
>>>>
>>>> + udp_info->src_port =
>>>> +irdma_get_udp_sport(udp_info->flow_label,
>>>> + ibqp->qp_num,
>>>> + roce_info->dest_qp);
>>>> +
>>>
>>> FYI that in mlx5 driver the udp_sport is only set when address vector
>>> and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided.
>>
>> Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn
>> is ready.
>>
>
> I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags.
>
> So, in this block,
> https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171
Thanks. The latest patch has been sent following your suggestions.
Zhu Yanjun
>
> Shiraz
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-12-21 13:43 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-12-18 20:44 [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary yanjun.zhu
2021-12-19 7:11 ` Mark Zhang
2021-12-19 8:20 ` Yanjun Zhu
2021-12-20 17:54 ` Saleem, Shiraz
2021-12-21 13:43 ` Yanjun Zhu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.