[PATCH perftest] perftest: make inline send on ocrdma work better

[PATCH perftest] perftest: make inline send on ocrdma work better

[Jarod Wilson]

The ocrdma driver supports a maximum inline send size of 224, while the
current code sets this size to 236. Add an ocrdma-specific check when
setting max inline size to eliminate the need to always pass -I 224 when
running ib_send_lat on ocrdma hardware.

CC: Gil Rockah <gilr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Jarod Wilson <jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 src/perftest_parameters.c | 8 ++++++++
 src/perftest_parameters.h | 3 +++
 2 files changed, 11 insertions(+)

diff --git a/src/perftest_parameters.c b/src/perftest_parameters.c
index 3ffddd7..65ce731 100755
--- a/src/perftest_parameters.c
+++ b/src/perftest_parameters.c
@@ -1465,6 +1465,14 @@ static void ctx_set_max_inline(struct ibv_context *context,struct perftest_param
 		}
 	}
 
+	if (strncmp("ocrdma", user_param->ib_devname, 6) == 0) {
+		if (user_param->inline_size > DEF_INLINE_SEND_MAX_OCRDMA) {
+			fprintf(stdout, "lowering inline size to ocrdma supported maximum of %d\n",
+				DEF_INLINE_SEND_MAX_OCRDMA);
+			user_param->inline_size = DEF_INLINE_SEND_MAX_OCRDMA;
+		}
+	}
+
 	return;
 }
 /******************************************************************************
diff --git a/src/perftest_parameters.h b/src/perftest_parameters.h
index 1122792..a566f42 100755
--- a/src/perftest_parameters.h
+++ b/src/perftest_parameters.h
@@ -133,6 +133,9 @@
 #define DEF_INLINE_SEND_UD (188)
 #define DEF_INLINE_DC (150)
 
+/* OCRDMA has an inline max of 224 */
+#define DEF_INLINE_SEND_MAX_OCRDMA (224)
+
 /* Max and Min allowed values for perftest parameters. */
 #define MIN_TOS		(0)
 #define MAX_TOS		(256)
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH perftest] perftest: make inline send on ocrdma work better

[Steve Wise]

On 8/19/2016 2:06 PM, Jarod Wilson wrote:
> The ocrdma driver supports a maximum inline send size of 224, while the
> current code sets this size to 236. Add an ocrdma-specific check when
> setting max inline size to eliminate the need to always pass -I 224 when
> running ib_send_lat on ocrdma hardware.

<soap box>
We really should fix the RDMA uAPI to allow querying the device to get 
the max inline size for various SQ WRs (it might vary for send vs write, 
for example), instead of requiring applications to guess or hardcoding 
benchmarks based on device name...
</soap box>

> CC: Gil Rockah <gilr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Jarod Wilson <jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> ---
>   src/perftest_parameters.c | 8 ++++++++
>   src/perftest_parameters.h | 3 +++
>   2 files changed, 11 insertions(+)
>
> diff --git a/src/perftest_parameters.c b/src/perftest_parameters.c
> index 3ffddd7..65ce731 100755
> --- a/src/perftest_parameters.c
> +++ b/src/perftest_parameters.c
> @@ -1465,6 +1465,14 @@ static void ctx_set_max_inline(struct ibv_context *context,struct perftest_param
>   		}
>   	}
>   
> +	if (strncmp("ocrdma", user_param->ib_devname, 6) == 0) {
> +		if (user_param->inline_size > DEF_INLINE_SEND_MAX_OCRDMA) {
> +			fprintf(stdout, "lowering inline size to ocrdma supported maximum of %d\n",
> +				DEF_INLINE_SEND_MAX_OCRDMA);
> +			user_param->inline_size = DEF_INLINE_SEND_MAX_OCRDMA;
> +		}
> +	}
> +
>   	return;
>   }
>   /******************************************************************************
> diff --git a/src/perftest_parameters.h b/src/perftest_parameters.h
> index 1122792..a566f42 100755
> --- a/src/perftest_parameters.h
> +++ b/src/perftest_parameters.h
> @@ -133,6 +133,9 @@
>   #define DEF_INLINE_SEND_UD (188)
>   #define DEF_INLINE_DC (150)
>   
> +/* OCRDMA has an inline max of 224 */
> +#define DEF_INLINE_SEND_MAX_OCRDMA (224)
> +
>   /* Max and Min allowed values for perftest parameters. */
>   #define MIN_TOS		(0)
>   #define MAX_TOS		(256)


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH perftest] perftest: make inline send on ocrdma work better

[Jarod Wilson]

On Fri, Aug 19, 2016 at 03:45:57PM -0500, Steve Wise wrote:
> On 8/19/2016 2:06 PM, Jarod Wilson wrote:
> >The ocrdma driver supports a maximum inline send size of 224, while the
> >current code sets this size to 236. Add an ocrdma-specific check when
> >setting max inline size to eliminate the need to always pass -I 224 when
> >running ib_send_lat on ocrdma hardware.
> 
> <soap box>
> We really should fix the RDMA uAPI to allow querying the device to
> get the max inline size for various SQ WRs (it might vary for send
> vs write, for example), instead of requiring applications to guess
> or hardcoding benchmarks based on device name...
> </soap box>

This was a nice quick hack Friday afternoon hack to get it working
with existing kernels in our testing infra, but having a way to ask
the hardware would definitely be vastly superior.

-- 
Jarod Wilson
jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH perftest] perftest: make inline send on ocrdma work better

[Steve Wise]

> On Fri, Aug 19, 2016 at 03:45:57PM -0500, Steve Wise wrote:
> > On 8/19/2016 2:06 PM, Jarod Wilson wrote:
> > >The ocrdma driver supports a maximum inline send size of 224, while the
> > >current code sets this size to 236. Add an ocrdma-specific check when
> > >setting max inline size to eliminate the need to always pass -I 224 when
> > >running ib_send_lat on ocrdma hardware.
> >
> > <soap box>
> > We really should fix the RDMA uAPI to allow querying the device to
> > get the max inline size for various SQ WRs (it might vary for send
> > vs write, for example), instead of requiring applications to guess
> > or hardcoding benchmarks based on device name...
> > </soap box>
> 
> This was a nice quick hack Friday afternoon hack to get it working
> with existing kernels in our testing infra, but having a way to ask
> the hardware would definitely be vastly superior.
> 

I hear ya.  Maybe as part of the new verbs API work we could provide a way for
applications to query the max inline...

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html