New providers in rdma-core

New providers in rdma-core

[Leon Romanovsky]

Hi All,

I want to remind you that latest kernel is exposing different vendor
specific structures (include/uapi/rdma/*-abi.,h) and new coming user
space drivers should use these headers directly and not copy/paste the
code from them.

In the future, we will convert already accepted drivers too.

Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Christoph Hellwig]

On Wed, Oct 19, 2016 at 07:18:04PM +0300, Leon Romanovsky wrote:
> Hi All,
> 
> I want to remind you that latest kernel is exposing different vendor
> specific structures (include/uapi/rdma/*-abi.,h) and new coming user
> space drivers should use these headers directly and not copy/paste the
> code from them.
> 
> In the future, we will convert already accepted drivers too.

Requiring installed super updtodate kernel headers isn't going to
help development.  It would be good to keep a copy of the kernel
headers in the rdma-core repo, similar to what most kernel tools
tied to bleeding edge ABIs do.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Jason Gunthorpe]

On Wed, Oct 19, 2016 at 09:39:13AM -0700, Christoph Hellwig wrote:

> Requiring installed super updtodate kernel headers isn't going to
> help development.  It would be good to keep a copy of the kernel
> headers in the rdma-core repo, similar to what most kernel tools
> tied to bleeding edge ABIs do.

Yes, this is already happening, eg rdma/rdma_user_rxe.h is cached in
the tree, and I have a patch to move the rdma_netlink.h to that scheme
as well.

cmake has a scheme where it detects if the system header is too
old/missing and if so transparently arranges things so that

 #include <rdma/rdma_user_rxe.h>

Uses the internal copy.

I also want to add a cmake option to use the kernel headers from a
kernel tree - nobody has been testing that changes to the kernel
headers work with the user space either, and eg the changes the
netlink broke stuff :/

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Adit Ranadive]

Hi Leon/Christoph,

On Wed, Oct 19, 2016 at 09:39:13AM -0700, Christoph Hellwig wrote:
> On Wed, Oct 19, 2016 at 07:18:04PM +0300, Leon Romanovsky wrote:
> > Hi All,
> > 
> > I want to remind you that latest kernel is exposing different vendor
> > specific structures (include/uapi/rdma/*-abi.,h) and new coming user
> > space drivers should use these headers directly and not copy/paste the
> > code from them.
> > 
> > In the future, we will convert already accepted drivers too.
> 
> Requiring installed super updtodate kernel headers isn't going to
> help development.  It would be good to keep a copy of the kernel
> headers in the rdma-core repo, similar to what most kernel tools
> tied to bleeding edge ABIs do.

Thanks for the heads up. I had a few questions regarding the ABI stuff.
What happens in case of providers that arent part of the kernel as yet?
I would like to send out a pull request soonish for libpvrdma. I would 
need to include the abi files in that request since cmake would fail.
Or do you want me to wait till Doug adds the PVRDMA driver to the tree?

Thanks,
Adit
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Jason Gunthorpe]

On Wed, Oct 19, 2016 at 10:28:36AM -0700, Adit Ranadive wrote:

> > Requiring installed super updtodate kernel headers isn't going to
> > help development.  It would be good to keep a copy of the kernel
> > headers in the rdma-core repo, similar to what most kernel tools
> > tied to bleeding edge ABIs do.
> 
> Thanks for the heads up. I had a few questions regarding the ABI stuff.
> What happens in case of providers that arent part of the kernel as yet?
> I would like to send out a pull request soonish for libpvrdma. I would 
> need to include the abi files in that request since cmake would fail.

Follow the pattern rxe is using:

CHECK_INCLUDE_FILE("rdma/rdma_user_rxe.h" HAVE_RDMA_USER_RXE)
RDMA_DoFixup("${HAVE_RDMA_USER_RXE}" "rdma/rdma_user_rxe.h")

Which uses the compat header file in
buildlib/fixup-include/rdma-rdma_user_rxe.h, which must be a
byte-for-byte copy of the kernel header.

> Or do you want me to wait till Doug adds the PVRDMA driver to the tree?

FWIW, we can't take a rdma-core provider until the kernel side is
accepted, but you can certainly start the process now and get it to an
OK state. Doug will be able to take both parts simultaneously.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Adit Ranadive]

On Wed, Oct 19, 2016 at 11:59:26AM -0600, Jason Gunthorpe wrote:
> On Wed, Oct 19, 2016 at 10:28:36AM -0700, Adit Ranadive wrote:
> 
> > > Requiring installed super updtodate kernel headers isn't going to
> > > help development.  It would be good to keep a copy of the kernel
> > > headers in the rdma-core repo, similar to what most kernel tools
> > > tied to bleeding edge ABIs do.
> > 
> > Thanks for the heads up. I had a few questions regarding the ABI stuff.
> > What happens in case of providers that arent part of the kernel as yet?
> > I would like to send out a pull request soonish for libpvrdma. I would 
> > need to include the abi files in that request since cmake would fail.
> 
> Follow the pattern rxe is using:
> 
> CHECK_INCLUDE_FILE("rdma/rdma_user_rxe.h" HAVE_RDMA_USER_RXE)
> RDMA_DoFixup("${HAVE_RDMA_USER_RXE}" "rdma/rdma_user_rxe.h")
> 
> Which uses the compat header file in
> buildlib/fixup-include/rdma-rdma_user_rxe.h, which must be a
> byte-for-byte copy of the kernel header.

I see that rxe still has a rxe-abi header in the providers/rxe folder.
Would that also move to the kernel as part of the conversion?

One of the things I'm unclear about what happends to the IBV structs
included in the user-level ABI headers, for example, the ibv_get_context_resp
here:
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx4/mlx4-abi.h#L48

Since these are not present in the kernel, how would the libibverbs/core
uverbs stack handle this? Would there be forthcoming changes for that too?

> > Or do you want me to wait till Doug adds the PVRDMA driver to the tree?
> 
> FWIW, we can't take a rdma-core provider until the kernel side is
> accepted, but you can certainly start the process now and get it to an
> OK state. Doug will be able to take both parts simultaneously.
> 

Makes sense.

Thanks,
Adit
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Jason Gunthorpe]

On Wed, Oct 19, 2016 at 12:51:02PM -0700, Adit Ranadive wrote:

> I see that rxe still has a rxe-abi header in the providers/rxe folder.
> Would that also move to the kernel as part of the conversion?

Ideally, but it is blocked by this:

> One of the things I'm unclear about what happends to the IBV structs
> included in the user-level ABI headers, for example, the ibv_get_context_resp
> here:
> https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx4/mlx4-abi.h#L48

This is the main thing preventing the use of the uapi headers - they
are actually *different* :(

It would be great if you have have some ideas on how to deal with
that.. This hasn't been tackled yet.

One choice is to write this:

struct mlx4_alloc_ucontext_resp_v3 {
	struct ibv_get_context_resp	ibv_resp;
	__u32				qp_tab_size;
	__u16				bf_reg_size;
	__u16				bf_regs_per_page;
};

as:

struct user_mlx4_alloc_ucontext_resp_v3 {
	struct ibv_get_context_resp	ibv_resp;
	struct mlx4_alloc_ucontext_resp_v3 udata;
};

Perhaps aided by a macro so we get high uniformity:

#define DECLARE_DRIVER_UDATA(struct_driver, struct_common)
  struct user # struct_driver {
     struct struct_common ibv_resp;
     struct struct_driver udata;
  };

Where mlx4_alloc_ucontext_resp_v3 comes from the uapi header, and is
not inlined, so each driver gets an 'abi wrapper' header that just
builds all these structs.

I'd probably advocate for that for new drivers if nothing more clever
comes forward. At least it is a step closer..

Another approach might be a different way to marshal the kernel call..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: New providers in rdma-core

[Amrani, Ram]

> Perhaps aided by a macro so we get high uniformity:
> 
> #define DECLARE_DRIVER_UDATA(struct_driver, struct_common)
>   struct user # struct_driver {
>      struct struct_common ibv_resp;
>      struct struct_driver udata;
>   };
> 
> Where mlx4_alloc_ucontext_resp_v3 comes from the uapi header, and is not
> inlined, so each driver gets an 'abi wrapper' header that just builds all these
> structs.

The idea looks good to me.
But still, I'm missing something - in order for libraries and the kernel to use the 
same headers they should reside in the same repository.
Is that the long-term goal? And wouldn't that be mixing user/kernel spaces?

Thanks,
Ram

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Jason Gunthorpe]

On Thu, Oct 20, 2016 at 09:24:30AM +0000, Amrani, Ram wrote:

> The idea looks good to me.
> But still, I'm missing something - in order for libraries and the kernel to use the 
> same headers they should reside in the same repository.
> Is that the long-term goal? And wouldn't that be mixing user/kernel spaces?

See the discussion here:

http://www.spinics.net/lists/linux-rdma/msg42067.html

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[oulijun]

在 2016/10/20 0:18, Leon Romanovsky 写道:
> Hi All,
> 
> I want to remind you that latest kernel is exposing different vendor
> specific structures (include/uapi/rdma/*-abi.,h) and new coming user
> space drivers should use these headers directly and not copy/paste the
> code from them.
> 
> In the future, we will convert already accepted drivers too.
> 
> Thanks
> 
> .
> 
Hi, Leon
  I am preparing for the userspace library of hns and have learned your email.
I have the same question with Jason Gunthorpe. Meanwhile, the new provider qedr don't
do this.
  I think that your plan will have a more clever discussion coming forward. In order to
not be blocked, I will don't something more until it is a step closer. After that, i
will fix it seraparete separately.

thanks
Lijun Ou

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: New providers in rdma-core

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Tue, Oct 25, 2016 at 01:42:44PM +0800, oulijun wrote:
> 在 2016/10/20 0:18, Leon Romanovsky 写道:
> > Hi All,
> >
> > I want to remind you that latest kernel is exposing different vendor
> > specific structures (include/uapi/rdma/*-abi.,h) and new coming user
> > space drivers should use these headers directly and not copy/paste the
> > code from them.
> >
> > In the future, we will convert already accepted drivers too.
> >
> > Thanks
> >
> > .
> >
> Hi, Leon
>   I am preparing for the userspace library of hns and have learned your email.
> I have the same question with Jason Gunthorpe. Meanwhile, the new provider qedr don't
> do this.
>   I think that your plan will have a more clever discussion coming forward. In order to
> not be blocked, I will don't something more until it is a step closer. After that, i
> will fix it seraparete separately.

Sure, no problem.

>
> thanks
> Lijun Ou
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]