Building 3.1-rc9 in kernel infiniband support with OFED libraries

Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

Seems to work mostly but some userspace libraries (mlx4) complain about
kernel version mismatch and missing XRC support.

Has XRC support not been merged? How can I build the OFED libraries
against Linux 3.1? I'd really like to get rid of the OFED kernel tree
nightmare.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Jason Gunthorpe]

On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote:

> Has XRC support not been merged? How can I build the OFED libraries
> against Linux 3.1? I'd really like to get rid of the OFED kernel tree
> nightmare.

You have to use upstream libraries with upstream kernels. Be warned
that the OFED libraries of the same SONAME are not ABI compatible with
upstream libraries.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

On Tue, 11 Oct 2011, Jason Gunthorpe wrote:

> On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote:
>
> > Has XRC support not been merged? How can I build the OFED libraries
> > against Linux 3.1? I'd really like to get rid of the OFED kernel tree
> > nightmare.
>
> You have to use upstream libraries with upstream kernels. Be warned
> that the OFED libraries of the same SONAME are not ABI compatible with
> upstream libraries.

Thats a pretty bad situation. Could we not at least get the kernel API
standardized?

Publish git trees for ofed that are based on supported upstream and distro
versions?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Bart Van Assche]

On Tue, Oct 11, 2011 at 7:39 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote:
>> Has XRC support not been merged? How can I build the OFED libraries
>> against Linux 3.1? I'd really like to get rid of the OFED kernel tree
>> nightmare.
>
> You have to use upstream libraries with upstream kernels. Be warned
> that the OFED libraries of the same SONAME are not ABI compatible with
> upstream libraries.

Why is the OFED libibverbs library binary incompatible with the
non-OFED libibverbs library ? Why hasn't XRC support been implemented
in the OFED libibverbs library such that applications built against
the upstream libibverbs headers also work with the latest OFED version
of that library ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Wendy Cheng]

On Wed, Oct 12, 2011 at 3:41 AM, Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org> wrote:
> On Tue, Oct 11, 2011 at 7:39 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>> On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote:
>>> Has XRC support not been merged? How can I build the OFED libraries
>>> against Linux 3.1? I'd really like to get rid of the OFED kernel tree
>>> nightmare.
>>
>> You have to use upstream libraries with upstream kernels. Be warned
>> that the OFED libraries of the same SONAME are not ABI compatible with
>> upstream libraries.
>
> Why is the OFED libibverbs library binary incompatible with the
> non-OFED libibverbs library ? Why hasn't XRC support been implemented
> in the OFED libibverbs library such that applications built against
> the upstream libibverbs headers also work with the latest OFED version
> of that library ?
>

I'm relatively new to OFED but happened to bump into a similar build
issue two weeks ago.

The OFED package itself does include XRC support. The issue here (my
guess) is that its build script needs to understand the running
system's kernel version to decide what should be pulled (from the
source). Linux 3.1 could be too new for OFED build script to make a
correct decision. Nevertheless, mix-matching OFED modules/libraries is
a *bad* idea.

It is difficult to love OFED build :) but it seems to work ok (so far
for me). Plus, I don't have a better proposal myself anyway .....

-- Wendy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Wed, Oct 12, 2011 at 9:32 AM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> The OFED package itself does include XRC support. The issue here (my
> guess) is that its build script needs to understand the running
> system's kernel version to decide what should be pulled (from the
> source). Linux 3.1 could be too new for OFED build script to make a
> correct decision. Nevertheless, mix-matching OFED modules/libraries is
> a *bad* idea.

No.  The same userspace build should work with all kernel versions.

We did have some ABI churn early in the history of the RDMA stack --
the last time we changed IB_USER_VERBS_ABI_VERSION was early
in 2006.  But we really need to add XRC and other support in such a
way that such ABI bumps are not necessary.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Doug Ledford]

----- Original Message -----
> On Wed, Oct 12, 2011 at 9:32 AM, Wendy Cheng
> <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > The OFED package itself does include XRC support. The issue here
> > (my
> > guess) is that its build script needs to understand the running
> > system's kernel version to decide what should be pulled (from the
> > source). Linux 3.1 could be too new for OFED build script to make a
> > correct decision. Nevertheless, mix-matching OFED modules/libraries
> > is
> > a *bad* idea.
> 
> No.  The same userspace build should work with all kernel versions.

Wendy's referring to something other than what you are thinking.  The
same libibverbs user space build should work on all kernels going back
a long way, except when you are talking about OFED their libibverbs
is hard coded to assume XRC support and fail if it isn't present, so
an OFED libibverbs won't really work without also the OFED kernel
module stack.  The script Wendy referred to is the script that checks
the running kernel's version in order to determine which backport
patches need applied to the ofa_kernel source tree in order to build
the OFED kernel modules for your running kernel.  Without that
ofa_kernel build, the OFED libibverbs will indeed fail to run on
the running kernel.  And that script hasn't been updated to support
version 3.x kernels last I checked, so she's right, the script itself
doesn't recognize the running kernel version, so ofa_kernel modules
don't get built, so OFED libibverbs won't work anyway.  So, she's
absolutely right, unless you want to start ripping hard coded
assumptions about the existence of XRC support out of things like
OFED's libibverbs, then out of qperf and a number of their other
various packages, then you have to pair the OFED kernel modules and
user space packages, they can not be separated.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Wendy Cheng]

On Fri, Oct 14, 2011 at 2:22 PM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> ----- Original Message -----
>> On Wed, Oct 12, 2011 at 9:32 AM, Wendy Cheng
>> <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > The OFED package itself does include XRC support. The issue here
>> > (my
>> > guess) is that its build script needs to understand the running
>> > system's kernel version to decide what should be pulled (from the
>> > source). Linux 3.1 could be too new for OFED build script to make a
>> > correct decision. Nevertheless, mix-matching OFED modules/libraries
>> > is
>> > a *bad* idea.
>>
>> No.  The same userspace build should work with all kernel versions.
>
> Wendy's referring to something other than what you are thinking.  The
> same libibverbs user space build should work on all kernels going back
> a long way, except when you are talking about OFED their libibverbs
> is hard coded to assume XRC support and fail if it isn't present, so
> an OFED libibverbs won't really work without also the OFED kernel
> module stack.  The script Wendy referred to is the script that checks
> the running kernel's version in order to determine which backport
> patches need applied to the ofa_kernel source tree in order to build
> the OFED kernel modules for your running kernel.  Without that
> ofa_kernel build, the OFED libibverbs will indeed fail to run on
> the running kernel.  And that script hasn't been updated to support
> version 3.x kernels last I checked, so she's right, the script itself
> doesn't recognize the running kernel version, so ofa_kernel modules
> don't get built, so OFED libibverbs won't work anyway.  So, she's
> absolutely right, unless you want to start ripping hard coded
> assumptions about the existence of XRC support out of things like
> OFED's libibverbs, then out of qperf and a number of their other
> various packages, then you have to pair the OFED kernel modules and
> user space packages, they can not be separated.
>

Yes, that (above) is exactly what I was referring to .....

The conversations in this thread remind me of the tire-swing cartoon
that has been passing around for years:
http://bibiananunes.com/user-requirements-the-tire-swing-cartoon )


-- Wendy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Hefty, Sean]

> Why is the OFED libibverbs library binary incompatible with the
> non-OFED libibverbs library ? Why hasn't XRC support been implemented
> in the OFED libibverbs library such that applications built against
> the upstream libibverbs headers also work with the latest OFED version
> of that library ?

The original XRC patches were submitted upstream, but were never accepted.  Along with those patches were modifications to ibverbs.  Those changes broke existing applications and were also never accepted upstream.  However, OFED still provided a release based on that code.  This is only a guess, but the push to release may have been a time to market decision.  (I'm just trying to relay history here.)

I have since re-worked both the kernel and user space XRC patches and submitted them for inclusion upstream.  Those patches do maintain backwards compatibility.  I believe that OFA is waiting for those patches (or some deviation) to be accepted upstream.  OFED will then be rebased on those patches, released as 2.0, with a stronger attempt made to keep everything in sync.

Hopefully XRC can make it into the 3.2 kernel.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Wed, Oct 12, 2011 at 9:37 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Hopefully XRC can make it into the 3.2 kernel.

Yes, I would really like to get them in.

Did we every resolve the controversy about the rcv QPs with
MPI users?  The design seemed sane to me, but....

Also (I'm sure you already posted this once, but...) Sean,
do you have a git tree with all the kernel patches included?

With patchwork.k.o down, it's a bit of a pain for me to
find the latest version of various patches.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Hefty, Sean]

> Did we every resolve the controversy about the rcv QPs with
> MPI users?  The design seemed sane to me, but....

Yes - I believe so.

> Also (I'm sure you already posted this once, but...) Sean,
> do you have a git tree with all the kernel patches included?

My latest patches are at:

	git://git.openfabrics.org/~shefty/rdma-dev.git xrc

though this is based on 3.0.
 
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Wed, Oct 12, 2011 at 9:46 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> though this is based on 3.0.

Thanks, that's fine.  I don't think I merged many changes for 3.1 ;)

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

> My latest patches are at:
>
>        git://git.openfabrics.org/~shefty/rdma-dev.git xrc

So I went through this and merged it to my tree (pretty much only conflicts
from 3.0->3.1-rc fixed, commit log changes and other minor cleanups).

The result is pushed out to my github  for-next branch, with the
expectation that I'll ask Linus to pull for 3.2.

However I do have one question: the last patch
("RDMA/uverbs: Export ib_open_qp() capability to user space" in
my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't
see any driver add that to its uverbs_cmd_mask... does this work?

Thanks, Roland
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1005 bytes --]

On Thu, 13 Oct 2011, Roland Dreier wrote:

> > My latest patches are at:
> >
> >        git://git.openfabrics.org/~shefty/rdma-dev.git xrc
>
> So I went through this and merged it to my tree (pretty much only conflicts
> from 3.0->3.1-rc fixed, commit log changes and other minor cleanups).

We got the XRC support via the for-next branch on github. Is that current?

> The result is pushed out to my github  for-next branch, with the
> expectation that I'll ask Linus to pull for 3.2.
>
> However I do have one question: the last patch
> ("RDMA/uverbs: Export ib_open_qp() capability to user space" in
> my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't
> see any driver add that to its uverbs_cmd_mask... does this work?
>
> Thanks, Roland

There seems to be some stuff missing in the upstream code compared to the
OFED releases:

1. Raw ethernet support (IB_QPT_RAW_ETH) is missing.

2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back
via loopback?

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

Oh yeah and can we can FDR support in for-next as well?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Thu, Oct 13, 2011 at 10:17 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
> There seems to be some stuff missing in the upstream code compared to the
> OFED releases:
>
> 1. Raw ethernet support (IB_QPT_RAW_ETH) is missing.
>
> 2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back
> via loopback?

Clean reviewed patches for this are where?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

On Thu, 13 Oct 2011, Roland Dreier wrote:

> On Thu, Oct 13, 2011 at 10:17 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
> > There seems to be some stuff missing in the upstream code compared to the
> > OFED releases:
> >
> > 1. Raw ethernet support (IB_QPT_RAW_ETH) is missing.
> >
> > 2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back
> > via loopback?
>
> Clean reviewed patches for this are where?

They are in OFED-1.5.3.1 so they were already released.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
>> Clean reviewed patches for this are where?
>
> They are in OFED-1.5.3.1 so they were already released.

OFED is neither clean nor reviewed.  Really.  The stuff in OFED always
needs a bunch
of review before it is suitable for upstream.  Ignoring ABI stability
is just one problem
that OFED code typically has.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Christoph Lameter]

On Thu, 13 Oct 2011, Roland Dreier wrote:

> On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
> >> Clean reviewed patches for this are where?
> >
> > They are in OFED-1.5.3.1 so they were already released.
>
> OFED is neither clean nor reviewed.  Really.  The stuff in OFED always
> needs a bunch
> of review before it is suitable for upstream.  Ignoring ABI stability
> is just one problem
> that OFED code typically has.

Yeah. ABI stability etc is really what we want. Thats why I would like to
see it upstream rather than continue to work with this OFED tarball
ofa_kernel mess.

So the patches need to be resubmitted for upstream inclusion from
Mellanox to you?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Doug Ledford]

----- Original Message -----
> On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org>
> wrote:
> >> Clean reviewed patches for this are where?
> >
> > They are in OFED-1.5.3.1 so they were already released.
> 
> OFED is neither clean nor reviewed.  Really.  The stuff in OFED
> always
> needs a bunch
> of review before it is suitable for upstream.  Ignoring ABI stability
> is just one problem
> that OFED code typically has.

Indeed.  Because it is in OFED means it is in testing, nothing more.
A better acronym for OFED is Open Fabrics Experimental Distribution.
These issues are the primary reason we don't follow OFED in RHEL6.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Hefty, Sean]

> The result is pushed out to my github  for-next branch, with the
> expectation that I'll ask Linus to pull for 3.2.

Thanks - I'll take a look and test again.
 
> However I do have one question: the last patch
> ("RDMA/uverbs: Export ib_open_qp() capability to user space" in
> my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't
> see any driver add that to its uverbs_cmd_mask... does this work?

ib_open_qp() is implemented entirely in the verbs layer (verbs.c).  The OFED API compatibility support that I added to libibverbs makes use of this call, which I tested by running OSU's mvapich2.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Thu, Oct 13, 2011 at 10:31 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> ib_open_qp() is implemented entirely in the verbs layer (verbs.c).  The OFED API compatibility support that I added to libibverbs makes use of this call, which I tested by running OSU's mvapich2.

Wait, now I'm baffled by the patch (ie
http://git.openfabrics.org/git?p=~shefty/rdma-dev.git;a=commitdiff;h=1ec4e62a6e967ddc258e7c4e674168debb727d39)

I don't see anything that calls ib_uverbs_open_qp().  Am I missing something??

Does the OFED API compatibility actually call this function and care
if it fails?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Hefty, Sean]

> Wait, now I'm baffled by the patch (ie
> http://git.openfabrics.org/git?p=~shefty/rdma-
> dev.git;a=commitdiff;h=1ec4e62a6e967ddc258e7c4e674168debb727d39)
> 
> I don't see anything that calls ib_uverbs_open_qp().  Am I missing something??
> 
> Does the OFED API compatibility actually call this function and care
> if it fails?

I'm confused now.  The patch definitely looks like it's missing a change to uverbs_main.c to setup the command table.  (I thought you were referring to needing a change to struct ib_device before.)

I'm looking at the code on my test system and trying to understand what exactly is happening.  The OFED compatibility should call this routine and care if it fails, and it looks like mpi cares too, but I don't see how it can work...  Give me some time to look into this.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Roland Dreier]

On Thu, Oct 13, 2011 at 11:32 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> I'm confused now.  The patch definitely looks like it's missing a change to uverbs_main.c to setup the command table.  (I thought you were referring to needing a change to struct ib_device before.)

Right, we need to add it to the command table, and also the hardware driver
needs to opt into the function in its uverbs cmd mask.  (Even if the call never
hits the hardware driver ... we could clean that up maybe, but I'm just talking
about how things work at the moment)

> I'm looking at the code on my test system and trying to understand what exactly is happening.  The OFED compatibility should call this routine and care if it fails, and it looks like mpi cares too, but I don't see how it can work...  Give me some time to look into this

No problem.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries

[Doug Ledford]

----- Original Message -----
> On Tue, Oct 11, 2011 at 7:39 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote:
> >> Has XRC support not been merged? How can I build the OFED
> >> libraries
> >> against Linux 3.1? I'd really like to get rid of the OFED kernel
> >> tree
> >> nightmare.
> >
> > You have to use upstream libraries with upstream kernels. Be warned
> > that the OFED libraries of the same SONAME are not ABI compatible
> > with
> > upstream libraries.
> 
> Why is the OFED libibverbs library binary incompatible with the
> non-OFED libibverbs library ?

Well, as an example:

Upstream (Roland's) libibverbs (and this may not be entirely clean,
I do have the original RoCE support in here, and plan to switch over
to IBoE when it's available): /usr/include/infiniband/verbs.h

struct ibv_srq {
        struct ibv_context     *context;
        void                   *srq_context;
        struct ibv_pd          *pd;
        uint32_t                handle;

        pthread_mutex_t         mutex;
        pthread_cond_t          cond;
        uint32_t                events_completed;
};

>From OFED's libibverbs: /usr/include/infiniband/verbs.h

struct ibv_srq {
        struct ibv_context     *context;
        void                   *srq_context;
        struct ibv_pd          *pd;
        uint32_t                handle;

        uint32_t                events_completed;

        uint32_t                xrc_srq_num;
        struct ibv_xrc_domain  *xrc_domain;
        struct ibv_cq          *xrc_cq;

        pthread_mutex_t         mutex;
        pthread_cond_t          cond;
};

So, while they added a few new things, they also reordered existing
items.  This is a public structure passed back to the app and which
the app might read any of the items if they are interested.  So, with
a struct change like this you would normally require an update to
the library symbol map and have a compat wrapper so that binaries
compiled with either the old or new struct definition can link
against the library.  However, here is the library symbol maps for
ibv_create_srq in each library source:

Roland's: ibv_create_srq{IBVERBS_1.0}
OFED's: ibv_create_srq{IBVERBS_1.0}

This is despite the fact that whenever an ibverbs interface has
needed an update, Roland has done the right thing and updated
the symbol version to the IBVERBS_1.1 space in the map and
created a back compat wrapper in the 1.0 space.  So it's not even
like the OFED folks didn't have examples of how to do this
properly staring them in the face, they just didn't care to do it
properly.

> Why hasn't XRC support been implemented
> in the OFED libibverbs library such that applications built against
> the upstream libibverbs headers also work with the latest OFED
> version
> of that library ?

Because OFED exists.  That's the *real* reason they don't bother to
write this stuff right in the first place.  OFED is released as one
single unit, and there is no guarantee of API or ABI compatibility
from one release to the next.  And there is no guarantee of API/ABI
compatibility between any given OFED released library and any other
upstream library (aka, no guarantee that the OFED libibverbs will
work with the upstream librdmacm).  If something doesn't work,
then their answer is "Upgrade to the latest OFED on all machines,
and recompile your apps as needed".  If you had your own tree and
didn't have to answer to any upstream review, especially on
controversial new features or difficult designs, would you bother?

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html