OFED 1.5.2: libibumad.so version bump

OFED 1.5.2: libibumad.so version bump

[Yevgeny Kliteynik]

Hi all,

In order to support RoCEE, a while ago I've added
a new field to umad, thus introduced an ABI change.

There already was a discussion on the linux-rdma list,
but due to the proximity of the upcoming OFED 1.5.2
release these concerns were raised again.

So my question is, other that *general* concerns about
changing ABI, is anybody aware of the *actual* problem
that will be caused by this? Any customer/3rd party
solution that would be affected by this?

-- Yevgeny

Re: OFED 1.5.2: libibumad.so version bump

[Ira Weiny]

On Tue, 31 Aug 2010 02:27:29 -0700
Yevgeny Kliteynik <kliteyn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi all,
> 
> In order to support RoCEE, a while ago I've added
> a new field to umad, thus introduced an ABI change.
> 
> There already was a discussion on the linux-rdma list,
> but due to the proximity of the upcoming OFED 1.5.2
> release these concerns were raised again.
> 
> So my question is, other that *general* concerns about
> changing ABI, is anybody aware of the *actual* problem
> that will be caused by this? Any customer/3rd party
> solution that would be affected by this?

Because our MVAPICH depends on umad, libibumad.so.1 to be exact.[*]  These ABI
changes (to v2 and v3) would have forced our users to recompile their codes.
We are maintaining the old ABI here until our next major release of CHAOS[#]
to prevent this.

I think the thing to remember is that many people are using Open Fabrics
software, but are _not_ using "OFED".  What is tested with OFED is not the
only thing which might be using these libraries.  Our version of MVAPICH is a
good example.

I am certainly not the expert in this area and I know that many people have
tried to make this point in the past, but I will say it here again.  Each of
these Open Fabrics packages _must_ be maintained to stand on their own.
Roland did this a long time ago with ibverbs.

I think now is a good time to start discussing breaking up the "management" git
tree so that these libraries can live on their own.  I will write a separate
email regarding this.

Ira

[*] We are looking into removing the dependency.
[#] Shameless plug: http://code.google.com/p/chaos-release/wiki/CHAOS_Description

> 
> -- Yevgeny
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://*vger.kernel.org/majordomo-info.html
> 


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED 1.5.2: libibumad.so version bump

[Hal Rosenstock]

Hi Ira,

On Tue, Aug 31, 2010 at 3:49 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> On Tue, 31 Aug 2010 02:27:29 -0700
> Yevgeny Kliteynik <kliteyn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Hi all,
>>
>> In order to support RoCEE, a while ago I've added
>> a new field to umad, thus introduced an ABI change.
>>
>> There already was a discussion on the linux-rdma list,
>> but due to the proximity of the upcoming OFED 1.5.2
>> release these concerns were raised again.
>>
>> So my question is, other that *general* concerns about
>> changing ABI, is anybody aware of the *actual* problem
>> that will be caused by this? Any customer/3rd party
>> solution that would be affected by this?
>
> Because our MVAPICH depends on umad, libibumad.so.1 to be exact.[*]  These ABI
> changes (to v2 and v3) would have forced our users to recompile their codes.
> We are maintaining the old ABI here until our next major release of CHAOS[#]
> to prevent this.
>
> I think the thing to remember is that many people are using Open Fabrics
> software, but are _not_ using "OFED".  What is tested with OFED is not the
> only thing which might be using these libraries.  Our version of MVAPICH is a
> good example.
>
> I am certainly not the expert in this area and I know that many people have
> tried to make this point in the past, but I will say it here again.  Each of
> these Open Fabrics packages _must_ be maintained to stand on their own.
> Roland did this a long time ago with ibverbs.
>
> I think now is a good time to start discussing breaking up the "management" git
> tree so that these libraries can live on their own.

How does breaking up the "management" git tree help with this issue ?
To me, that's the admin part and is separate from the ABI issue
raised. The ABI compatibility is not achieved by administrative means
(separate repos, etc.) but rather than review and discipline to
achieve this as a unmutable goal.

-- Hal

> I will write a separate email regarding this.
>
> Ira
>
> [*] We are looking into removing the dependency.
> [#] Shameless plug: http://code.google.com/p/chaos-release/wiki/CHAOS_Description
>
>>
>> -- Yevgeny
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://*vger.kernel.org/majordomo-info.html
>>
>
>
> --
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED 1.5.2: libibumad.so version bump

[Ira Weiny]

On Thu, 2 Sep 2010 13:35:44 -0700
Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi Ira,
> 
> On Tue, Aug 31, 2010 at 3:49 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> > On Tue, 31 Aug 2010 02:27:29 -0700
> > Yevgeny Kliteynik <kliteyn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> >> Hi all,
> >>
> >> In order to support RoCEE, a while ago I've added
> >> a new field to umad, thus introduced an ABI change.
> >>
> >> There already was a discussion on the linux-rdma list,
> >> but due to the proximity of the upcoming OFED 1.5.2
> >> release these concerns were raised again.
> >>
> >> So my question is, other that *general* concerns about
> >> changing ABI, is anybody aware of the *actual* problem
> >> that will be caused by this? Any customer/3rd party
> >> solution that would be affected by this?
> >
> > Because our MVAPICH depends on umad, libibumad.so.1 to be exact.[*]  These ABI
> > changes (to v2 and v3) would have forced our users to recompile their codes.
> > We are maintaining the old ABI here until our next major release of CHAOS[#]
> > to prevent this.
> >
> > I think the thing to remember is that many people are using Open Fabrics
> > software, but are _not_ using "OFED".  What is tested with OFED is not the
> > only thing which might be using these libraries.  Our version of MVAPICH is a
> > good example.
> >
> > I am certainly not the expert in this area and I know that many people have
> > tried to make this point in the past, but I will say it here again.  Each of
> > these Open Fabrics packages _must_ be maintained to stand on their own.
> > Roland did this a long time ago with ibverbs.
> >
> > I think now is a good time to start discussing breaking up the "management" git
> > tree so that these libraries can live on their own.
> 
> How does breaking up the "management" git tree help with this issue ?

Creating and tracking of branches to maintain ABI would be easier with
separate git trees.  Furthermore, separate trees will help "force" the use of
consistent ABI's and interfaces.

For example, if I currently want OpenSM version 3.3.6 I get a management tree
with version libibumad 1.3.5.  But this last ABI change to umad was only
required for the latest infiniband-diags (ibstat utility).  Why do I get all
this "cruft" when pulling the latest OpenSM?

> To me, that's the admin part and is separate from the ABI issue
> raised.

Yes it is separate.  That is why I created another thread to discuss those
issues.

>
> The ABI compatibility is not achieved by administrative means
> (separate repos, etc.) but rather than review and discipline to
> achieve this as a unmutable goal.

I agree that ABI compatibility will require more discipline.  That is what
made me think of the separate git trees.  I feel it will be _easier_ to
maintain this discipline when the trees are separate.

Ira

> 
> -- Hal
> 
> > I will write a separate email regarding this.
> >
> > Ira
> >
> > [*] We are looking into removing the dependency.
> > [#] Shameless plug: http://*code.google.com/p/chaos-release/wiki/CHAOS_Description
> >
> >>
> >> -- Yevgeny
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> More majordomo info at  http://**vger.kernel.org/majordomo-info.html
> >>
> >
> >
> > --
> > Ira Weiny
> > Math Programmer/Computer Scientist
> > Lawrence Livermore National Lab
> > 925-423-8008
> > weiny2-i2BcT+NCU+M@public.gmane.org
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://*vger.kernel.org/majordomo-info.html
> >
> 


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org

Re: OFED 1.5.2: libibumad.so version bump

[Jason Gunthorpe]

On Thu, Sep 02, 2010 at 02:05:02PM -0700, Ira Weiny wrote:

> > The ABI compatibility is not achieved by administrative means
> > (separate repos, etc.) but rather than review and discipline to
> > achieve this as a unmutable goal.
> 
> I agree that ABI compatibility will require more discipline.  That is what
> made me think of the separate git trees.  I feel it will be _easier_ to
> maintain this discipline when the trees are separate.

I would almost think the reverse.. It feels like there are *too many*
GIT trees and too few maintainers. I do argee that including the core
MAD libraries with opensm is kinda silly.

Consider bundling all the core libraries into a single git tree and
considering a group of people to watch for these issues for all the
libraries, together.

It feels like ongoing development of many of these libs has stopped,
plus they are so tiny, it might be a better use of everyones time this
way?

When I say combine together, I really mean that, shared configure
script, etc, etc. Not like we see in the management.git.

As it is, building the core OFA userspace from source is a huge
PITA. Just finding the proper git trees is often hard..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED 1.5.2: libibumad.so version bump

[Hal Rosenstock]

On Thu, Sep 2, 2010 at 5:05 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> On Thu, 2 Sep 2010 13:35:44 -0700
> Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Hi Ira,
>>
>> On Tue, Aug 31, 2010 at 3:49 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
>> > On Tue, 31 Aug 2010 02:27:29 -0700
>> > Yevgeny Kliteynik <kliteyn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >
>> >> Hi all,
>> >>
>> >> In order to support RoCEE, a while ago I've added
>> >> a new field to umad, thus introduced an ABI change.
>> >>
>> >> There already was a discussion on the linux-rdma list,
>> >> but due to the proximity of the upcoming OFED 1.5.2
>> >> release these concerns were raised again.
>> >>
>> >> So my question is, other that *general* concerns about
>> >> changing ABI, is anybody aware of the *actual* problem
>> >> that will be caused by this? Any customer/3rd party
>> >> solution that would be affected by this?
>> >
>> > Because our MVAPICH depends on umad, libibumad.so.1 to be exact.[*]  These ABI
>> > changes (to v2 and v3) would have forced our users to recompile their codes.
>> > We are maintaining the old ABI here until our next major release of CHAOS[#]
>> > to prevent this.
>> >
>> > I think the thing to remember is that many people are using Open Fabrics
>> > software, but are _not_ using "OFED".  What is tested with OFED is not the
>> > only thing which might be using these libraries.  Our version of MVAPICH is a
>> > good example.
>> >
>> > I am certainly not the expert in this area and I know that many people have
>> > tried to make this point in the past, but I will say it here again.  Each of
>> > these Open Fabrics packages _must_ be maintained to stand on their own.
>> > Roland did this a long time ago with ibverbs.
>> >
>> > I think now is a good time to start discussing breaking up the "management" git
>> > tree so that these libraries can live on their own.
>>
>> How does breaking up the "management" git tree help with this issue ?
>
> Creating and tracking of branches to maintain ABI would be easier with
> separate git trees.  Furthermore, separate trees will help "force" the use of
> consistent ABI's and interfaces.
>
> For example, if I currently want OpenSM version 3.3.6 I get a management tree
> with version libibumad 1.3.5.  But this last ABI change to umad was only
> required for the latest infiniband-diags (ibstat utility).  Why do I get all
> this "cruft" when pulling the latest OpenSM?

That's a totally different issue than which packages a particular OFED
release picks up.

>
>> To me, that's the admin part and is separate from the ABI issue
>> raised.
>
> Yes it is separate.  That is why I created another thread to discuss those
> issues.
>
>>
>> The ABI compatibility is not achieved by administrative means
>> (separate repos, etc.) but rather than review and discipline to
>> achieve this as a unmutable goal.
>
> I agree that ABI compatibility will require more discipline.  That is what
> made me think of the separate git trees.  I feel it will be _easier_ to
> maintain this discipline when the trees are separate.

Call me a skeptic but I think the same thing would've occurred with
separate git trees.

I have no real preference one way or the other. In fact, this was
discussed in the early days which are long forgotten. The libraries
are small and umad is a lot more stable than mad. It would just mean a
lot of busy work for everyone with internal trees.

-- Hal

>
> Ira
>
>>
>> -- Hal
>>
>> > I will write a separate email regarding this.
>> >
>> > Ira
>> >
>> > [*] We are looking into removing the dependency.
>> > [#] Shameless plug: http://*code.google.com/p/chaos-release/wiki/CHAOS_Description
>> >
>> >>
>> >> -- Yevgeny
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> >> More majordomo info at  http://**vger.kernel.org/majordomo-info.html
>> >>
>> >
>> >
>> > --
>> > Ira Weiny
>> > Math Programmer/Computer Scientist
>> > Lawrence Livermore National Lab
>> > 925-423-8008
>> > weiny2-i2BcT+NCU+M@public.gmane.org
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> > More majordomo info at  http://*vger.kernel.org/majordomo-info.html
>> >
>>
>
>
> --
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html