max_inline_data in ibv_rc_pingpong

max_inline_data in ibv_rc_pingpong

[Amrani, Ram]

Hi Yishay, Doug,
I've noticed something irregular in ibv_rc_pingpong and I see that you are listed as the maintainers of libibverbs.
It seems that max_inline_data isn't configured prior to invoking ibv_create_qp(), but it is queried directly after.
This means that for a value of zero it will always be zero, or very small, depending on the vendor implementation.
Is there some background I'm missing?

        {
                struct ibv_qp_attr attr;
                struct ibv_qp_init_attr init_attr = {
                        .send_cq = ctx->cq,
                        .recv_cq = ctx->cq,
                        .cap     = {
                                .max_send_wr  = 1,
                                .max_recv_wr  = rx_depth,
                                .max_send_sge = 1,
                                .max_recv_sge = 1
                        },
                        .qp_type = IBV_QPT_RC
                };

                ctx->qp = ibv_create_qp(ctx->pd, &init_attr);
                if (!ctx->qp)  {
                        fprintf(stderr, "Couldn't create QP\n");
                        goto clean_cq;
                }

                ibv_query_qp(ctx->qp, &attr, IBV_QP_CAP, &init_attr);
                if (init_attr.cap.max_inline_data >= size) {
                        ctx->send_flags |= IBV_SEND_INLINE;
                }
        }


Thanks,
Ram

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: max_inline_data in ibv_rc_pingpong

[Yishai Hadas]

On 12/25/2016 1:48 PM, Amrani, Ram wrote:
> Hi Yishay, Doug,
> I've noticed something irregular in ibv_rc_pingpong and I see that you are listed as the maintainers of libibverbs.
> It seems that max_inline_data isn't configured prior to invoking ibv_create_qp(), but it is queried directly after.
> This means that for a value of zero it will always be zero, or very small, depending on the vendor implementation.
> Is there some background I'm missing?

Based on man page cap is an in/out structure
"..the values will be greater than or equal to the values requested".

The below example demonstrates a case that there is enough space for 
using inline data for the given input size. It may happen as of some 
alignment on the WQE calculated size and is based on vendor implementation.

In a real application usually an explicit value is set before calling 
ibv_create_qp to achieve a size that is required by the application for 
its needs.

>         {
>                 struct ibv_qp_attr attr;
>                 struct ibv_qp_init_attr init_attr = {
>                         .send_cq = ctx->cq,
>                         .recv_cq = ctx->cq,
>                         .cap     = {
>                                 .max_send_wr  = 1,
>                                 .max_recv_wr  = rx_depth,
>                                 .max_send_sge = 1,
>                                 .max_recv_sge = 1
>                         },
>                         .qp_type = IBV_QPT_RC
>                 };
>
>                 ctx->qp = ibv_create_qp(ctx->pd, &init_attr);
>                 if (!ctx->qp)  {
>                         fprintf(stderr, "Couldn't create QP\n");
>                         goto clean_cq;
>                 }
>
>                 ibv_query_qp(ctx->qp, &attr, IBV_QP_CAP, &init_attr);
>                 if (init_attr.cap.max_inline_data >= size) {
>                         ctx->send_flags |= IBV_SEND_INLINE;
>                 }
>         }
>
>
> Thanks,
> Ram
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: max_inline_data in ibv_rc_pingpong

[Amrani, Ram]

> Based on man page cap is an in/out structure
> "..the values will be greater than or equal to the values requested".
> 
> The below example demonstrates a case that there is enough space for
> using inline data for the given input size. It may happen as of some
> alignment on the WQE calculated size and is based on vendor
> implementation.
> 
> In a real application usually an explicit value is set before calling
> ibv_create_qp to achieve a size that is required by the application for
> its needs.


I would prefer it to use an explicit value instead of per-driver residual value.
This allows for better understanding of what is actually going on.
One of our customers used it for latency comparisons between devices and got different values, not knowing that with one driver he was using inline and with the other he isn't.
Will you be OK with a "-I" parameter as in perftest?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: max_inline_data in ibv_rc_pingpong

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Wed, Dec 28, 2016 at 09:45:08AM +0000, Amrani, Ram wrote:
> > Based on man page cap is an in/out structure
> > "..the values will be greater than or equal to the values requested".
> >
> > The below example demonstrates a case that there is enough space for
> > using inline data for the given input size. It may happen as of some
> > alignment on the WQE calculated size and is based on vendor
> > implementation.
> >
> > In a real application usually an explicit value is set before calling
> > ibv_create_qp to achieve a size that is required by the application for
> > its needs.
>
>
> I would prefer it to use an explicit value instead of per-driver residual value.
> This allows for better understanding of what is actually going on.
> One of our customers used it for latency comparisons between devices and got different values, not knowing that with one driver he was using inline and with the other he isn't.
> Will you be OK with a "-I" parameter as in perftest?

Why not? this is example application.
If the user doesn't supply "-I xxx", you will fallback to legacy mode.

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]