ib mad definitions

ib mad definitions

[Hefty, Sean]

This has probably been discussed before, but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband?

This appears to be the only place where IB MAD definitions are available for user space applications, and having them available at the libibumad level makes sense to me.

(I'm trying to port madeye to user space as a diag, and want all IB MAD definitions.)

- Sean 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Mike Heinz]

Works for me.

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Hefty, Sean
Sent: Monday, October 18, 2010 6:25 PM
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Sasha Khapyorsky
Subject: ib mad definitions

This has probably been discussed before, but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband?

This appears to be the only place where IB MAD definitions are available for user space applications, and having them available at the libibumad level makes sense to me.

(I'm trying to port madeye to user space as a diag, and want all IB MAD definitions.)

- Sean 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Hal Rosenstock]

On Mon, Oct 18, 2010 at 6:24 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> This has probably been discussed before,

Yes, several times AFAIR.

> but is there a strong reason why ib_types.h can't be moved from opensm/include/iba to libibumad/include/infiniband?

Why does this need to be moved ?

> This appears to be the only place where IB MAD definitions are available for user space applications, and having them available at the libibumad level makes sense to me.
>
> (I'm trying to port madeye to user space as a diag, and want all IB MAD definitions.)

There already are diags including ib_types.h (saquery for one).

-- Hal

> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 742 bytes --]

> > but is there a strong reason why ib_types.h can't be moved from
> opensm/include/iba to libibumad/include/infiniband?
> 
> Why does this need to be moved ?

The dependency should be on libibumad, not opensm.  libibumad is pretty much useless without these definitions.  Why wouldn't you move them?

> There already are diags including ib_types.h (saquery for one).

Yes, but we're either stuck with everything that needs ib_types.h to be part of the management.git tree, or the app needs to depend on opensm.  Currently, ibacm duplicates definitions because they aren't available anywhere else.
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: ib mad definitions

[Hal Rosenstock]

On Tue, Oct 19, 2010 at 11:28 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> > but is there a strong reason why ib_types.h can't be moved from
>> opensm/include/iba to libibumad/include/infiniband?
>>
>> Why does this need to be moved ?
>
> The dependency should be on libibumad, not opensm.  libibumad is pretty much useless without these definitions.  Why wouldn't you move them?

Off the top of my head, OpenSM is layered on top of libibumad but
doesn't need/use libibmad. I think that was the main reason although
that could be changed if ib_types.h were to be moved. I'm not sure
what other reasons came up in the previous discussions.

>
>> There already are diags including ib_types.h (saquery for one).
>
> Yes, but we're either stuck with everything that needs ib_types.h to be part of the management.git tree, or the app needs to depend on opensm.  Currently, ibacm duplicates definitions because they aren't available anywhere else.

I agree ib_types.h is more generic than opensm. Moving to libibmad and
making opensm depend on this is probably better than all the
duplication. There have been viewpoints that libibumad and libibmad
shouldn't be separate (as they are small) but they were never combined
into a single library.

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

> I agree ib_types.h is more generic than opensm. Moving to libibmad and
> making opensm depend on this is probably better than all the
> duplication. There have been viewpoints that libibumad and libibmad
> shouldn't be separate (as they are small) but they were never combined
> into a single library.

My motivation with these changes is for ibacm to receive and use notification of CM timeouts to update its path record cache.  ibacm already defines the basic mad structure, multicast record, and path record.  It would also need the CM mad format.  I'd happily remove these definitions if they were already available.

Porting madeye to user space is a side benefit to the proposed kernel changes.

ibacm only depends on libibumad.  The madeye port also only depends on libibumad.  Honestly, I find the libibmad APIs confusing.  I'd much rather libibumad provide mad definitions.

Sasha/Ira, do either of you have opinions on this?

Re: ib mad definitions

[Hal Rosenstock]

On Tue, Oct 19, 2010 at 12:48 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> I agree ib_types.h is more generic than opensm. Moving to libibmad and
>> making opensm depend on this is probably better than all the
>> duplication. There have been viewpoints that libibumad and libibmad
>> shouldn't be separate (as they are small) but they were never combined
>> into a single library.

The other thing I just recalled was the OpenSM portability issue.
ib_types.h is needed here and libibmad/libibumad is not in all those
environments. As you''re all too well aware, this was even the case in
Windows until very recently. There may still be others we care about
where moving ib_types.h might be problematic.

-- Hal

> My motivation with these changes is for ibacm to receive and use notification of CM timeouts to update its path record cache.  ibacm already defines the basic mad structure, multicast record, and path record.  It would also need the CM mad format.  I'd happily remove these definitions if they were already available.
>
> Porting madeye to user space is a side benefit to the proposed kernel changes.
>
> ibacm only depends on libibumad.  The madeye port also only depends on libibumad.  Honestly, I find the libibmad APIs confusing.  I'd much rather libibumad provide mad definitions.
>
> Sasha/Ira, do either of you have opinions on this?
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Ira Weiny]

On Tue, 19 Oct 2010 08:43:22 -0700
Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Tue, Oct 19, 2010 at 11:28 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> >> > but is there a strong reason why ib_types.h can't be moved from
> >> opensm/include/iba to libibumad/include/infiniband?
> >>
> >> Why does this need to be moved ?
> >
> > The dependency should be on libibumad, not opensm.  libibumad is pretty much useless without these definitions.  Why wouldn't you move them?
> 
> Off the top of my head, OpenSM is layered on top of libibumad but
> doesn't need/use libibmad. I think that was the main reason although
> that could be changed if ib_types.h were to be moved. I'm not sure
> what other reasons came up in the previous discussions.

I think ib_types.h should be part of ibumad.  Everything depends on libibumad
at some point.[*]  Therefore common mad definitions should be in ib_types.h and
packaged with libibumad.

[*] ok OpenSM does not strictly, see below.

> 
> >
> >> There already are diags including ib_types.h (saquery for one).
> >
> > Yes, but we're either stuck with everything that needs ib_types.h to be part of the management.git tree, or the app needs to depend on opensm.  Currently, ibacm duplicates definitions because they aren't available anywhere else.
> 
> I agree ib_types.h is more generic than opensm. Moving to libibmad and
> making opensm depend on this is probably better than all the
> duplication. There have been viewpoints that libibumad and libibmad
> shouldn't be separate (as they are small) but they were never combined
> into a single library.

The opposing view is that libibumad is only an interface to the kernel umad
module, where libibmad is more abstract.


As far as moving ib_types, I suggested this a while back.
http://www.mail-archive.com/general-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org/msg27439.html

Let's see if I can summarize the thread.

- Sean was workiong on libibacm and redefined ib_types.h definitions.
- I suggested moving ib_types.h to umad so he would not have a dependancy on
  OpenSM.
- Sean brought up that ib_types.h is large and probably should be split
- I agreed, and asked Sasha if such a patch would be acceptable, or create a
  new library to deal with the inline functions in ib_types.h
- Hal said that ibutils requires ib_types.h but does not want a dependancy on
  libibumad...
- I suggested a separate library to solve this problem.
- Hal corrected himself saying that ibutils requires osm_vendor_ibumad.
  However, OpenSM does not always use libibumad (depending on the underlying
  stack) so it would need to get ib_types somewhere else.  Hal was also
  concerned about a library with little more than a header file in it.
- Jason chimed in with "Please no more libraries"...  :-)  (and digressed with
  Sean in to PR queries, MPI, and other useful, but unrelated, stuff)
- Sean says "libibumad is pretty useless without some network structure
  definitions."
- I state that it looks like ibutils dependancy is on the static functions in
  ib_types.h only.
- Hal says yes ibutils depends on OpenSM for the vendor layer and that
  Mellanox is better able to answer questions regarding ibutils support.
- Hal says he thinks ib_types is "more akin to what is in libibmad rather than
  libibumad"
- Sean finds that ib_types.h includes complib headers.
- I submit a "rough hack" to remove complib headers.
- Jason, Sean, and myself discuss ugly byteswapping functions.
- Sasha agrees that he is not sure that umad is the right place for ib_types
- Sean says we should split the file up and at least some of the definitions
  should be in umad...


We all get busy...


I think we need to move ib_types (mad definitions to umad).

Basic MAD definitions should be provided at the lowest possible level so all
software can use them.

The issues (solutions) are:

ib_types depends on complib at the moment (fixable)
ibutils depends on OpenSM (it will anyway -- non-issue)
somethings in ib_types are ugly, byteswapping (non-issue; deal with it later)
OpenSM may _not_ include umad and therefore miss defines. (fixable?)

As for this last item, would it be a big deal to require umad for the header
only?  Does umad not compile somewhere that other vendor layers are used?  I
think it is much better for OpenSM to require umad than for other MAD
processing software to require OpenSM.  Also, would splitting ib_types help
this at all?


Ira

> 
> -- Hal
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://BLOCKEDvger.kernel.org/majordomo-info.html
> 


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

> ib_types depends on complib at the moment (fixable)
> ibutils depends on OpenSM (it will anyway -- non-issue)
> somethings in ib_types are ugly, byteswapping (non-issue; deal with it
> later)
> OpenSM may _not_ include umad and therefore miss defines. (fixable?)
> 
> As for this last item, would it be a big deal to require umad for the
> header
> only?  Does umad not compile somewhere that other vendor layers are used?
> I
> think it is much better for OpenSM to require umad than for other MAD
> processing software to require OpenSM.  Also, would splitting ib_types help
> this at all?

I'll propose the following:

1. Add to libibumad/include/infiniband:

   umad_types.h - basic mad, rmpp headers
   umad_sa.h    - SA attributes
   umad_cm.h    - CM messages

2. Include umad_types.h and umad_sa.h from ib_types.h
3. Include umad_cm.h from ib_cm_types.h

We start with a minimal set of definitions to umad and add/move other definitions later as needed, creating new header files where appropriate (umad_smi.h, umad_pm.h, etc.)

If we can get some basic agreement on this, I'll start on the patches immediately.  In an ideal world, the new header files would work on any platform.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Ira Weiny]

On Tue, 19 Oct 2010 11:50:46 -0700
"Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> > ib_types depends on complib at the moment (fixable)
> > ibutils depends on OpenSM (it will anyway -- non-issue)
> > somethings in ib_types are ugly, byteswapping (non-issue; deal with it
> > later)
> > OpenSM may _not_ include umad and therefore miss defines. (fixable?)
> > 
> > As for this last item, would it be a big deal to require umad for the
> > header
> > only?  Does umad not compile somewhere that other vendor layers are used?
> > I
> > think it is much better for OpenSM to require umad than for other MAD
> > processing software to require OpenSM.  Also, would splitting ib_types help
> > this at all?
> 
> I'll propose the following:
> 
> 1. Add to libibumad/include/infiniband:
> 
>    umad_types.h - basic mad, rmpp headers
>    umad_sa.h    - SA attributes
>    umad_cm.h    - CM messages
> 
> 2. Include umad_types.h and umad_sa.h from ib_types.h
> 3. Include umad_cm.h from ib_cm_types.h
> 
> We start with a minimal set of definitions to umad and add/move other definitions later as needed, creating new header files where appropriate (umad_smi.h, umad_pm.h, etc.)
> 
> If we can get some basic agreement on this, I'll start on the patches immediately.  In an ideal world, the new header files would work on any platform.

I agree,
Ira

> 
> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://BLOCKEDvger.kernel.org/majordomo-info.html
> 


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Smith, Stan]

Ira Weiny wrote:
> On Tue, 19 Oct 2010 11:50:46 -0700
> "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>
>>> ib_types depends on complib at the moment (fixable)
>>> ibutils depends on OpenSM (it will anyway -- non-issue)
>>> somethings in ib_types are ugly, byteswapping (non-issue; deal with
>>> it later) OpenSM may _not_ include umad and therefore miss defines.
>>> (fixable?)
>>>
>>> As for this last item, would it be a big deal to require umad for
>>> the header only?  Does umad not compile somewhere that other vendor
>>> layers are used? I think it is much better for OpenSM to require
>>> umad than for other MAD processing software to require OpenSM.
>>> Also, would splitting ib_types help this at all?
>>
>> I'll propose the following:
>>
>> 1. Add to libibumad/include/infiniband:
>>
>>    umad_types.h - basic mad, rmpp headers
>>    umad_sa.h    - SA attributes
>>    umad_cm.h    - CM messages
>>
>> 2. Include umad_types.h and umad_sa.h from ib_types.h
>> 3. Include umad_cm.h from ib_cm_types.h
>>
>> We start with a minimal set of definitions to umad and add/move
>> other definitions later as needed, creating new header files where
>> appropriate (umad_smi.h, umad_pm.h, etc.)
>>
>> If we can get some basic agreement on this, I'll start on the
>> patches immediately.  In an ideal world, the new header files would
>> work on any platform.
>
> I agree,
> Ira

Just to be painfully clear ...
A user-mode application would then only need to include ib_types.h + CM flavor of choice .h files ?

>
>>
>> - Sean
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at
>> http://BLOCKEDvger.kernel.org/majordomo-info.html
>>
>
>
> --
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

> Just to be painfully clear ...
> A user-mode application would then only need to include ib_types.h + CM
> flavor of choice .h files ?

For compatibility, ib_types.h would include whatever files any definitions were moved to.  An application that includes ib_types.h today wouldn't need additional includes.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Jason Gunthorpe]

On Tue, Oct 19, 2010 at 11:50:46AM -0700, Hefty, Sean wrote:

> We start with a minimal set of definitions to umad and add/move
> other definitions later as needed, creating new header files where
> appropriate (umad_smi.h, umad_pm.h, etc.)
 
> If we can get some basic agreement on this, I'll start on the
> patches immediately.  In an ideal world, the new header files would
> work on any platform.

Can we at least agree on the usage of these structures first? Are the
constants going to be in host or network byte order?

Are you going to make something like the kernel where there is a
native structure and pack/unpack function set?

Something macro-based like foo = GET_MEMBER(*pr,preference)

Network byte order casting structures?

Host byte order casting structures? (my favorite)

bitfields?

For years now I've had a set of data files that describe all the IB
structures bitfield layouts. I think I can contribute the data files
but not the generator script.

Since they all have various merits, maybe the smartest thing is to just
codegen all of the above permutations from single data source?

ie
// network endian bitfield casting structure
struct MADHeader_NE x = {};
x.status = htons(1);

// host endian bitfield casting structure
struct MADHeader_HE x = {};
x.status = 1
to_network(&x,sizeof(x)); // x[i] = htonl(x[i]) for i in len/4

/* Non-bitfield macro access structure
   (using the 1 byte = 1 bit helper structure technique) */
struct MADHeader_M x = {}
SET_MEMBER(x,status,1);  

// Pack/unpack function structure
struct MADHeader_UP x = {};
x.status = htons(1);
pack_MADHeader(&x,mad_buf,sizeof(mad_buf));

I'd like to think we don't need the last one, but people seem to like
that scheme ..

I also like to codegen structure printing functions, that is
surprisingly useful - and implements a good chunk of madeye.
 
What do you think?

I've also very recently been thinking that I'd like python bindings
for MADs for some projects. I was planning on building it out with the
code gen scheme.

Ira, I think the cleanest answer is that OSM keeps its type file, and
umad gets a new one that is cleaner, more capable and probably
incompatible. I'd hate to see us stick to the OSM scheme for umad just
for code compatability.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

> Can we at least agree on the usage of these structures first? Are the
> constants going to be in host or network byte order?

I was simply suggesting to 'move' some of the existing structures and defines.

> Are you going to make something like the kernel where there is a
> native structure and pack/unpack function set?

This would not be my preference.

> Something macro-based like foo = GET_MEMBER(*pr,preference)
> 
> Network byte order casting structures?
> 
> Host byte order casting structures? (my favorite)
> 
> bitfields?

again - not my preference

> Ira, I think the cleanest answer is that OSM keeps its type file, and
> umad gets a new one that is cleaner, more capable and probably
> incompatible. I'd hate to see us stick to the OSM scheme for umad just
> for code compatability.

Whatever is done must fit within the windows development framework that we use.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Jason Gunthorpe]

On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote:
> > Can we at least agree on the usage of these structures first? Are the
> > constants going to be in host or network byte order?
> 
> I was simply suggesting to 'move' some of the existing structures and defines.

But they are horrible and little used outside opensm right now, you
really want to commit to that forever?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Ira Weiny]

On Tue, 19 Oct 2010 18:09:58 -0700
Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote:
> > > Can we at least agree on the usage of these structures first? Are the
> > > constants going to be in host or network byte order?
> > 
> > I was simply suggesting to 'move' some of the existing structures and defines.
> 
> But they are horrible and little used outside opensm right now, you
> really want to commit to that forever?

Not everything is horrible.  And if it is we can fix it.  But I think
defining "yet another" header with the same functionality is worse.  Like it or
not ib_types is there.  If you don't remove/fix it, someone will find it and use
it.  How does that make things cleaner just because there is something "clean"
somewhere else?  Someone will find ib_types use it.  I still feel this is the
best first step at getting rid of ib_types.h (at least as it currently stands).

Ira

> 
> Jason


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Jason Gunthorpe]

On Tue, Oct 19, 2010 at 06:32:57PM -0700, Ira Weiny wrote:
> On Tue, 19 Oct 2010 18:09:58 -0700
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> > On Tue, Oct 19, 2010 at 06:00:51PM -0700, Hefty, Sean wrote:
> > > > Can we at least agree on the usage of these structures first? Are the
> > > > constants going to be in host or network byte order?
> > > 
> > > I was simply suggesting to 'move' some of the existing structures and defines.
> > 
> > But they are horrible and little used outside opensm right now, you
> > really want to commit to that forever?
 
> Not everything is horrible.  And if it is we can fix it.  But I think
> defining "yet another" header with the same functionality is worse.
> Like it or

libibumad is a system library. It needs to have a stable ABI, low
churn and ideally be 'complete'.

My database of IB structs has 117 structures, all with wakky alignment
and all manner of strangeness. IMHO, it is infeasible to keep with the
ad hoc approach in ibtypes.h and generate a complete header set
without a lot of churn. This is why it is horrible.

There are things worse than 'yet another' header - for instance a
system library being churned again and again for cleanups. Figure out
what you want, do it once, do it right, be done.

If we could all agree what these structs should look like I can
provide my database and someone can write the codegen AND WE CAN BE
DONE FOREVER. How is this not much better??

Don't treat the API of a system library as some casual thing. :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Ira Weiny]

On Tue, 19 Oct 2010 18:00:51 -0700
"Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> > Can we at least agree on the usage of these structures first? Are the
> > constants going to be in host or network byte order?
> 
> I was simply suggesting to 'move' some of the existing structures and defines.
> 
> > Are you going to make something like the kernel where there is a
> > native structure and pack/unpack function set?
> 
> This would not be my preference.
> 
> > Something macro-based like foo = GET_MEMBER(*pr,preference)
> > 
> > Network byte order casting structures?
> > 
> > Host byte order casting structures? (my favorite)
> > 
> > bitfields?
> 
> again - not my preference
> 
> > Ira, I think the cleanest answer is that OSM keeps its type file, and
> > umad gets a new one that is cleaner, more capable and probably
> > incompatible. I'd hate to see us stick to the OSM scheme for umad just
> > for code compatability.
> 
> Whatever is done must fit within the windows development framework that we use.

I am all for "cleaner, more capable..." but why incompatible?  If we want to
start fresh and then convert OpenSM later, fine.  But _don't_ forget to go
back and convert OpenSM, because if you leave ib_types.h out there someone is
going to use it and we are back to where we started...  :-(  Same for ibmad,
when these definitions become available in umad, mad can be simplified.

What I would like right now is to get the definitions in 1 place!

Right now there are 3 headers I find path record in.

libibverbs: sa.h
libibmad: mad.h
opensm: ib_types.h


Node type is defined in:

libibverbs: verbs.h
opensm: ib_types.h
libibmad: mad.h

I could go on.

What Sean is offering to do is move ib_types to umad.  From there I can use
those definitions in mad (thus removing them from mad and consolidating at
least 2 of the 3 above).  Perhaps use them in ibverbs as well?  As a first
step I think we should take Sean up on his offer to start cleaning things up.
But we have to remove stuff as we go or we will just be defining yet another
place to look for these.  After this we can look at making things cleaner
(perhaps even combining mad and umad, and including some of the ideas you have
above).  As Sean said in another email, after this change; including
"ib_types.h" will be the same for anyone using it.  The exception is that we
have simplified the code.  I think this is a win-win with minimal work.

Ira

-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Jason Gunthorpe]

On Tue, Oct 19, 2010 at 06:12:56PM -0700, Ira Weiny wrote:

> I am all for "cleaner, more capable..." but why incompatible?  If we want to
> start fresh and then convert OpenSM later, fine.  But _don't_ forget to go
> back and convert OpenSM, because if you leave ib_types.h out there someone is
> going to use it and we are back to where we started...  :-(  Same for ibmad,
> when these definitions become available in umad, mad can be simplified.

ib_types.h should not be installed in /usr/include, stop doing that
and that risk goes away.

ibmad can't really be changed, it is system library with a defined
API. Maybe ibmad.2 or something, I don't know. I tried to use some of
the PR APIs in it, and I've found them not useful :(

For instance we can't just abandon the mad_get_fields approach because
we have real, usuable field access in umad, it has to stay.

> Right now there are 3 headers I find path record in.

> libibverbs: sa.h

This isn't a MAD path record, this is the kernel version, which is
unpacked. What we really needs is MAD 2 kernel and vice versa
conversion in a library. I already have code that does this in
several places :(

> libibmad: mad.h

You mean mad_get_fields IB_SA_PR_DGID_F, etc? It doesn't even have all
the fields :(

> opensm: ib_types.h

Yep.
 
> Node type is defined in:
> 
> libibverbs: verbs.h
> opensm: ib_types.h
> libibmad: mad.h
> 
> I could go on.

Keep in mind that for the most part libibmad is someones attempt to
make a set of accessors and structures for mads. It is incomplete. It
is largely unusable. I certainly haven't been able to use its PR
structure parsing functions for any real app. Was it just pulled out
of opensm? I don't know, I'd just as soon see that part of it be
discarded, and a complete set of structures added to umad.

opensm has unique problems because they want to remain independent of the
OFA stack, I don't think they have a choice but to duplicate.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib mad definitions

[Hefty, Sean]

> > Right now there are 3 headers I find path record in.
> 
> > libibverbs: sa.h
> 
> This isn't a MAD path record, this is the kernel version, which is
> unpacked. What we really needs is MAD 2 kernel and vice versa
> conversion in a library. I already have code that does this in
> several places :(

This contains both.  A wire structure was added to libibverbs to support librdmacm, which doesn't depend on umad.  There are probably only a few of structures that make sense as part of ibverbs - path record and maybe multicast record.  But, we'd have to get agreement that umad should depend on ibverbs, or we'll always have duplicate definitions.

Jason, can you share a portion of the final output looks like for your codegen?

Currently, I'd be happy just starting by having things like the mgmt classes, methods, attribute IDs, status values, common mad header, rmpp header, and sa header listed in a single place.  Eventually, I need the CM MADs defined, which aren't available AFAICT.

ib_types uses a lot of typedefs and #defines, so we may be able to use those to provide backwards compatability.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Jason Gunthorpe]

On Wed, Oct 20, 2010 at 09:06:54AM -0700, Hefty, Sean wrote:

> This contains both.  A wire structure was added to libibverbs to
> support librdmacm, which doesn't depend on umad.  There are probably
> only a few of structures that make sense as part of ibverbs - path
> record and maybe multicast record.  But, we'd have to get agreement
> that umad should depend on ibverbs, or we'll always have duplicate
> definitions.

Duplication doesn't concern me too much, as long as the structures are
cast compatible it doesn't matter too much, there is nothing in verbs
that uses the type, for instance?

I think umad should depend on ibverbs (and use ibv_context to open the
device, but that is another subject), and I think the *structures*
should not have a ABI dependency on umad - static inlines and
structure definitions. So you can compile against it without a link
time dependency. This would let rdmacm use the structures, just have
to compile libumad first.

> Jason, can you share a portion of the final output looks like for your codegen?

Well, my codegen is tailored to what I've been doing. I would not use
this approach to target C code, you need to use macros instead of
templates, etc, etc. But that doesn't matter too much, you can codegen
whatever you want, be it mad_get_field, or pack/unpack or the
non-bitfield approach in ib_types.h.

I have something in mind for C that is a combination of bitfields and
the 1-byte=1-bit structure map technique. I'd codegen both host-endian
overlays and network endian overlays. There is good reason for both, IMHO.
[This email is so long already, I can outline my thoughts on this
 seperately if you are interested]

What I've done has great properties, it is very difficult to make an
endian error using it, and it has zero overhead on BE machines.

/* [15.2.5.16] Information on paths through the subnet */
struct SAPathRecord {
    enum { attributeID = 0x35 };
    uint32_t rsv0;

    uint32_t rsv1;

    struct HdrIPv6Addr DGID;

    struct HdrIPv6Addr SGID;

    uint16_t DLID;
    uint16_t SLID;

    uint32_t rawTraffic:1;
    uint32_t rsv2:3;
    uint32_t flowLabel:20;
    uint32_t hopLimit:8;

    uint8_t TClass;
    uint8_t reversible:1;
    uint8_t numbPath:7;
    uint16_t PKey;

    uint16_t rsv3:12;
    uint16_t SL:4;
    uint8_t MTUSelector:2;
    uint8_t MTU:6;
    uint8_t rateSelector:2;
    uint8_t rate:6;

    uint8_t packetLifeTimeSelector:2;
    uint8_t packetLifeTime:6;
    uint8_t preference;
    uint16_t rsv4;

    uint32_t rsv5;
};

This is a host-endian bitfield overlay. Two are output, this is the
host == BE version, the host == LE version is re-ordered:

    uint32_t hopLimit:8;
    uint32_t flowLabel:20;
    uint32_t rsv2:3;
    uint32_t rawTraffic:1;

etc. The neat thing about the codegen is that is analyzes the
arrangement to choose an optimal basic type. Ie TClass is not a
bitfield, but hopLimit is because flowLabel spans a 16 bit quantity.
This helps avoid any bit slicing code overheads by letting the
compiler produce short and byte loads whenever possible.

Use is sort of like:

uint32_t madBuf[256/4];
for (int I = 0; I != 64; I++)
   madBuf[I] = ntohl(madBuf[I]);
SAPathRecord *pr = madBuf;
pr.DLID = 1;
pr.SLID = 2;
pr.TClss = 3;
for (int I = 0; I != 64; I++)
   madBuf[I] = htonl(madBuf[I]);

This avoids all ntoh/hton on types <= 32, and completely avoids all
overhead on BE machines (which is important for me). In most of my
code I stick the swap in the packet RX/TX path.

The down side is that byte arrays, 64 bits and GIDs get messed up, but
those are compartively rare. With the network endian overlays those 3
things are OK but lot of stuff like flowlabel is hoplessly messed up
(need a ntoh20bits() function). I think both arrangements are useful..

The codgen uses C++ templates and anonymous unions to fix up some of
the ugly stuff in the spec, like the byte arrays and the different
labels for the MADHeaders in some cases.. These are special cases in
the script.

/* [14.2.1.2] SMP Format - Direct Routed */
struct SMPFormatDirected {
    enum { mgmtClass = 0x81 };
    union {
        struct MADHeader MADHeader;
        struct {
            uint32_t MADHeader1;
            uint16_t D:1;
            uint16_t status:15;
            uint8_t hopPointer;
            uint8_t hopCount;
            uint32_t MADHeader2[4];
        };
    };

    struct IBA64bit MKey;

    uint16_t drSLID;
    uint16_t drDLID;

    uint32_t rsv0[7];

    uint32_t data[16];

    IBABitFieldArray<8,64> initialPath;
    IBABitFieldArray<8,64> returnPath;
};

Or the 3 bit wide arrays in PMPortSamplesCtl (so gross):

/* [16.1.3.2] Port Performance Data Sampling Control */
struct PMPortSamplesCtl {
    enum { attributeID = 0x10 };
    uint8_t opCode;
    uint8_t portSelect;
    uint8_t tick;
    uint8_t rsv0:5;
    uint8_t counterWidth:3;

    union {
        IBABitFieldArray<3,15,CounterMaskTraits> counterMask;
        struct {
            uint32_t rsv1:2;
            uint32_t counterMask0:3;
            uint32_t counterMask1:3;
            uint32_t counterMask2:3;
            uint32_t counterMask3:3;
            uint32_t counterMask4:3;
            uint32_t counterMask5:3;
            uint32_t counterMask6:3;
            uint32_t counterMask7:3;
            uint32_t counterMask8:3;
            uint32_t counterMask9:3;

            uint16_t rsv2:1;
            uint16_t counterMask10:3;
            uint16_t counterMask11:3;
            uint16_t counterMask12:3;
            uint16_t counterMask13:3;
            uint16_t counterMask14:3;
            uint8_t sampleMechanisms;
            uint8_t rsv3:6;
            uint8_t sampleStatus:2;
        };
    };

The counterMask template's operator [] magically inlines the right
code for foo.counterMask[x] - again this is all about reducing the
possibility for coding error through automation.

Last time this was brought up Roland mentioned he thought the codegen
in gcc was poor for bitfields, I saw something recently that seems to
indicate bitfields are similar/better than slicing:

http://lwn.net/Articles/406067/

> Currently, I'd be happy just starting by having things like the mgmt
> classes, methods, attribute IDs, status values, common mad header,
> rmpp header, and sa header listed in a single place.  Eventually, I
> need the CM MADs defined, which aren't available AFAICT.

The main data I lack is constants for things like error codes in
obscure mads and so forth.

You can codegen things like the componetMask constants quiet easially,
etc.

Since the database annotates structure arrangements and attribute IDs
pretty well the codegen also produces an any-mad printer function with
considerable field decoding. This is incredibly useful for
debugging. ie:

   void print(const CMMRA & S, const char *prefix)
   {
      const uint32_t *p = reinterpret_cast < const uint32_t * >(&S);
      printf("%sGMP :  %08x  LCID=%u\n", prefix, *p++, (unsigned int) S.LCID);
      printf("%sGMP :  %08x  RCID=%u\n", prefix, *p++, (unsigned int) S.RCID);
      printf
         ("%sGMP :  %08x  messageMRAed=%u rsv0=%u serviceTimeout=%u rsv1=%u rsv2=%u\n",
          prefix, *p++, (unsigned int) S.messageMRAed, (unsigned int) S.rsv0,
          (unsigned int) S.serviceTimeout, (unsigned int) S.rsv1,
          (unsigned int) S.rsv2);

      for (unsigned int i = 0; i < 1760 / 32; i++)
         printf("%sGMP :  %08x  %11s[%2u]=%u\n", prefix, *p++,
                i == 0 ? "privateData" : "", i, S.privateData[i]);
   }

This code gen is 500 lines of perl for the structs and another 350 for
the printer code.

> ib_types uses a lot of typedefs and #defines, so we may be able to
> use those to provide backwards compatability.

Maybe? I don't know.. To me the most important thing is to have a
uniform conversion of what is in the spec to code, and from there if
API preserving wrappers are possible then great, otherwise.. ?

A big quibble I have with the naming in ibtypes is I think that
network endian constants are a mistake. Very confusing.

As far as Windows goes.. The great thing about codegen is you can
codegen anything :) As long as the Windows compiler behaves
predictably, we can codegen bitfields for it. There are only two
possible ways to lay out bitfields (high bit first or low bit first)
and the ABI specifies it. Linux ELF arm, ia32, ia64, pcc, mips 32/64
at least all use the same layout.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib mad definitions

[Ira Weiny]

On Tue, 19 Oct 2010 20:28:18 -0700
Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> On Tue, Oct 19, 2010 at 06:12:56PM -0700, Ira Weiny wrote:
> 
> > I am all for "cleaner, more capable..." but why incompatible?  If we want to
> > start fresh and then convert OpenSM later, fine.  But _don't_ forget to go
> > back and convert OpenSM, because if you leave ib_types.h out there someone is
> > going to use it and we are back to where we started...  :-(  Same for ibmad,
> > when these definitions become available in umad, mad can be simplified.
> 
> ib_types.h should not be installed in /usr/include, stop doing that
> and that risk goes away.

That is one way to solve it, but only after those definitions (or equivalent) are available in /usr/include.

> 
> ibmad can't really be changed, it is system library with a defined
> API. Maybe ibmad.2 or something, I don't know. I tried to use some of
> the PR APIs in it, and I've found them not useful :(

ibmad.2 is the way to go.  I have expressed my distaste for ibmad in the past and I agree, it is too established to be changed.  But I have not had the time.  Mainly I am trying to support Sean who has the time.

Ira

> 
> For instance we can't just abandon the mad_get_fields approach because
> we have real, usuable field access in umad, it has to stay.
> 
> > Right now there are 3 headers I find path record in.
> 
> > libibverbs: sa.h
> 
> This isn't a MAD path record, this is the kernel version, which is
> unpacked. What we really needs is MAD 2 kernel and vice versa
> conversion in a library. I already have code that does this in
> several places :(
> 
> > libibmad: mad.h
> 
> You mean mad_get_fields IB_SA_PR_DGID_F, etc? It doesn't even have all
> the fields :(
> 
> > opensm: ib_types.h
> 
> Yep.
>  
> > Node type is defined in:
> > 
> > libibverbs: verbs.h
> > opensm: ib_types.h
> > libibmad: mad.h
> > 
> > I could go on.
> 
> Keep in mind that for the most part libibmad is someones attempt to
> make a set of accessors and structures for mads. It is incomplete. It
> is largely unusable. I certainly haven't been able to use its PR
> structure parsing functions for any real app. Was it just pulled out
> of opensm? I don't know, I'd just as soon see that part of it be
> discarded, and a complete set of structures added to umad.
> 
> opensm has unique problems because they want to remain independent of the
> OFA stack, I don't think they have a choice but to duplicate.
> 
> Jason


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html