[RFC] – Proposal for new process for OFED releases

[RFC] – Proposal for new process for OFED releases

[Tziporet Koren]

We propose a new process for the OFED releases starting from next OFED release:
- OFED content will be the relevant kernel.org modules and user space released packages
- OFED will offer only backports to the distros  (no fixes)
- OFED package will be used for easy installation of all packages in a friendly manner

The main goals of this change:
1. Ensure OFED and the upstream kernel are the same
2. Provide customers a way to use the new features in latest kernels on existing distros
3. OFED qualification will contribute to the stability of the upstream code 
 
We think that at this point of the RDMA technology maturity this is the right way to go.
In this way OFED is not conflicting with the kernel or the distros, and still provide a valuable value for early adopters of new features.

Versions:
We suggest that the OFED version will be the same as kernel.org
For example, for kernel 3.2 the OFED release would be OFED-3.2. 
This would make it easy for people to associate the OFED code with the corresponding kernel.org code.

Some open questions that we should consider:
- How to handle experimental features?
- Need to follow up kernel stable releases if bug fixes are relevant to OFA modules
- Should we have a release for every kernel release (I think yes)
- What should we do with modules like SDP that are not in kernel?

Comments and responses are welcome
The decision will be taken in the next EWG meeting.

Tziporet & Woody

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC] – Proposal for new process for OFED releases

[Hefty, Sean]

> We propose a new process for the OFED releases starting from next OFED
> release:
> - OFED content will be the relevant kernel.org modules and user space released
> packages
> - OFED will offer only backports to the distros  (no fixes)

I think this point needs to be clarified - at least to me anyway. :)

> - OFED package will be used for easy installation of all packages in a
> friendly manner
> 
> The main goals of this change:
> 1. Ensure OFED and the upstream kernel are the same
> 2. Provide customers a way to use the new features in latest kernels on
> existing distros
> 3. OFED qualification will contribute to the stability of the upstream code

I like this approach.
 
> We think that at this point of the RDMA technology maturity this is the right
> way to go.
> In this way OFED is not conflicting with the kernel or the distros, and still
> provide a valuable value for early adopters of new features.
> 
> Versions:
> We suggest that the OFED version will be the same as kernel.org
> For example, for kernel 3.2 the OFED release would be OFED-3.2.
> This would make it easy for people to associate the OFED code with the
> corresponding kernel.org code.
> 
> Some open questions that we should consider:
> - How to handle experimental features?
> - Need to follow up kernel stable releases if bug fixes are relevant to OFA
> modules
> - Should we have a release for every kernel release (I think yes)

This would help test upstream submissions, which would be good.  It may be desirable to have stable versions of previous releases, so that customers can get bug fixes without pulling in a bunch of new features.  E.g. OFED-3.2.1, OFED-3.2.2, etc.  If maintaining stable releases of every version is too expensive, maybe mark specific versions (i.e. 3.2) as stable, with intermediate releases (i.e. 3.3, 3.4) as experimental.  Just some ideas.

> - What should we do with modules like SDP that are not in kernel?

Either remove them or carry them forward as experimental features.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC] - Proposal for new process for OFED releases

[Woodruff, Robert J]

Sean wrote,
>> - OFED will offer only backports to the distros  (no fixes)
> I think this point needs to be clarified - at least to me anyway. :)

What this means is that the OFED code base will be identical to what
is included in the upstream kernel and libs. 
OFED will provide only backports to allow that 
code to be run on other kernels.

If people find a bug or want a new feature, they need to submit the patch for the bug
or new code upstream and have it accepted there before it will go into any OFED
release.

We may need some other way for people to try out experimental new feature/code
that is not upstream, but we won't include any non-upstream code in the production
OFED release. 

Hope this helps clarify the proposal.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] - Proposal for new process for OFED releases

[Jason Gunthorpe]

On Thu, Dec 01, 2011 at 04:13:41PM -0800, Woodruff, Robert J wrote:
> Sean wrote,
> >> - OFED will offer only backports to the distros  (no fixes)
> > I think this point needs to be clarified - at least to me anyway. :)
> 
> What this means is that the OFED code base will be identical to what
> is included in the upstream kernel and libs. 
> OFED will provide only backports to allow that 
> code to be run on other kernels.
> 
> If people find a bug or want a new feature, they need to submit the
> patch for the bug or new code upstream and have it accepted there
> before it will go into any OFED release.
> 
> We may need some other way for people to try out experimental new
> feature/code that is not upstream, but we won't include any
> non-upstream code in the production OFED release.
> 
> Hope this helps clarify the proposal.

This is what many of us, including myself, have been asking OFED to be
for some time now. I hope the EWG adopts this proposal.

FWIW, I think it would be ideal if code that is thought important
enough to flow into OFED is also marked for the kernel.org stable
series. This increases the chance important work is brought into the
disto kernel updates quickly.

I think it would be fairly simple for vendors looking to beta features
to provide that via their own packaging, possibly derived from OFED,
pretty much exactly as we've seen to date anyhow.

Non-upstreamable features like SDP could be split into separate
packaging, and built after the appropriate OFED/distro headers are
setup.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Bart Van Assche]

On Fri, Dec 2, 2011 at 1:04 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > - What should we do with modules like SDP that are not in kernel?
>
> Either remove them or carry them forward as experimental features.

Wat I expect is that reworking the SDP implementation such that it can
be included upstream will take less work in the long term than
maintaining it as out-of-tree code.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Christoph Lameter]

On Fri, 2 Dec 2011, Bart Van Assche wrote:

> On Fri, Dec 2, 2011 at 1:04 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > > - What should we do with modules like SDP that are not in kernel?
> >
> > Either remove them or carry them forward as experimental features.
>
> Wat I expect is that reworking the SDP implementation such that it can
> be included upstream will take less work in the long term than
> maintaining it as out-of-tree code.

What were the issues that prevented the merging of the SDP
implementation?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Bart Van Assche]

On Fri, Dec 2, 2011 at 7:12 PM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:
> What were the issues that prevented the merging of the SDP
> implementation?

At least AF_INET_SDP - there might have been other issues. See e.g.
http://lkml.org/lkml/2006/3/6/70.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Or Gerlitz]

On 12/2/2011 2:04 AM, Hefty, Sean wrote:
>> We propose a new process for the OFED releases starting from next OFED release:
>> - OFED content will be the relevant kernel.org modules and user space released packages
>> - OFED will offer only backports to the distros  (no fixes)
>
> I think this point needs to be clarified

Sean, Yes, I agree, we have to be precise here, the term back-porting 
has to be made clear:

The kernel is a large piece that keeps moving on - its made of many 
smaller pieces/components, but with very sharp and well defined 
dependencies and interactions.  The rdma stack is far from being an 
isolated piece which you can pull from kernel X and plug into kernel Y - 
this applies all over the place in varying extents - e.g from the RDMA 
HW drivers, through the IB core and up to the ULPs.

The latter is the easiest to explain, as Roland once commented, each IB 
ULP driver is a fish and a eel - so SRP/iSER/rNFS/IPoIB are all IB 
fishes working with the IB core services and with the HW through the 
verbs, but the are part from higher-level constructs in the kernel, such 
as being network device (IPoIB), SCSI Low Level drivers (1st two), iscsi 
transport provider (iser) and RPC provider (rNFS).  Now, BACKPORTING 
these stacks (SCSI, iSCSI, RPC/NFS, etc) isn't something that OFA can 
carry, and it would be self-damaging to create notion with end-users 
that OFED does so. I tend to think this is done now for rNFS, and its a 
mistake. Distributions do that, by the way, but its part of their bread 
and butter.

This argument applies also to the core, yes. The core and IPoIB has 
interactions with the networking stack, e.g around route and neighbour 
lookups, and alike.  The networking stack is something that sits deeply 
into the built in part of the kernel and changes every now and then. 
BACKPORTING here would mean to simply remove sets of patches from the 
core and ipoib.

In the HW drivers space, things could be simpler, but I haven't thought 
about it deeply, though.

To be concrete/constructive here, a per IB stack module individual has 
to be assigned for that backporting, which doesn't mean "make IB code 
from kernel X to build under kernel Y" - lets
see if we have people to actually do that.

For example, on the iser space, and for the stack provided by Mellanox 
to customers - I took the approach of iser_backport(X,Y) = ~Y  --- which 
means that if I have to backport the iser code from kernel X into kernel 
Y, I simply use the iser code that comes with Y

I do that since Y has well/tight integrated iscsi stack for which the 
maintainers worked very hard to produce, and I can't re-invent 
backporting that stack.

The tilde in ~Y stands for slight verb changes that could arise from the 
backporting the rest of the IB stack has gone through, from X to Y, so 
if the verb to create CQ has another param in X vs what it had in Y- I 
add it under the ~ umbrella, is that clear?

Or.

Re: [RFC] – Proposal for new process for OFED releases

[Steve Wise]

On 12/05/2011 01:05 AM, Or Gerlitz wrote:
>
> To be concrete/constructive here, a per IB stack module individual has to be assigned for that backporting, which 
> doesn't mean "make IB code from kernel X to build under kernel Y" - lets
> see if we have people to actually do that.
>
> For example, on the iser space, and for the stack provided by Mellanox to customers - I took the approach of 
> iser_backport(X,Y) = ~Y  --- which means that if I have to backport the iser code from kernel X into kernel Y, I 
> simply use the iser code that comes with Y
>
> I do that since Y has well/tight integrated iscsi stack for which the maintainers worked very hard to produce, and I 
> can't re-invent backporting that stack.
>
> The tilde in ~Y stands for slight verb changes that could arise from the backporting the rest of the IB stack has gone 
> through, from X to Y, so if the verb to create CQ has another param in X vs what it had in Y- I add it under the ~ 
> umbrella, is that clear?
>

What if you had some feature in kernel X's iser that you wanted in the kernel Y backport?  By your definition, this 
wouldn't be possible it seems.  If we're only using the functionality of kernel Y, then what's the point?   Maybe I'm 
confused.

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Roland Dreier]

> The kernel is a large piece that keeps moving on - its made of many smaller
> pieces/components, but with very sharp and well defined dependencies and
> interactions.  The rdma stack is far from being an isolated piece which you
> can pull from kernel X and plug into kernel Y - this applies all over the
> place in varying extents - e.g from the RDMA HW drivers, through the IB core
> and up to the ULPs.
>
> The latter is the easiest to explain, as Roland once commented, each IB ULP
> driver is a fish and a eel - so SRP/iSER/rNFS/IPoIB are all IB fishes
> working with the IB core services and with the HW through the verbs, but the
> are part from higher-level constructs in the kernel, such as being network
> device (IPoIB), SCSI Low Level drivers (1st two), iscsi transport provider
> (iser) and RPC provider (rNFS).  Now, BACKPORTING these stacks (SCSI, iSCSI,
> RPC/NFS, etc) isn't something that OFA can carry, and it would be
> self-damaging to create notion with end-users that OFED does so. I tend to
> think this is done now for rNFS, and its a mistake. Distributions do that,
> by the way, but its part of their bread and butter.

https://lkml.org/lkml/2011/9/9/327 and its links have interesting and possibly
useful information about the wireless-compat backporting project.  In fact
it might make sense to try and merge OFED efforts into that compat project.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Christoph Lameter]

On Thu, 1 Dec 2011, Tziporet Koren wrote:

> We propose a new process for the OFED releases starting from next OFED release:
> - OFED content will be the relevant kernel.org modules and user space released packages
> - OFED will offer only backports to the distros  (no fixes)
> - OFED package will be used for easy installation of all packages in a friendly manner
>
> The main goals of this change:
> 1. Ensure OFED and the upstream kernel are the same
> 2. Provide customers a way to use the new features in latest kernels on existing distros
> 3. OFED qualification will contribute to the stability of the upstream code

Ohh. A Christmas (or whatever your favorite holidays label is) present.

Great, we may now be able to stop torturing our support departments to get
OFED running with this and that.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ewg] [RFC] – Proposal for new process for OFED releases

[Doug Ledford]

On 12/01/2011 02:53 PM, Tziporet Koren wrote:
> 
> We propose a new process for the OFED releases starting from next OFED release:
> - OFED content will be the relevant kernel.org modules and user space released packages
> - OFED will offer only backports to the distros  (no fixes)
> - OFED package will be used for easy installation of all packages in a friendly manner

Yay!!!  I'm all in favor.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] – Proposal for new process for OFED releases

[Bart Van Assche]

[-- Attachment #1.1: Type: text/plain, Size: 2153 bytes --]

On Thu, Dec 1, 2011 at 7:53 PM, Tziporet Koren <tziporet-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>wrote:

>
> We propose a new process for the OFED releases starting from next OFED
> release:
> - OFED content will be the relevant kernel.org modules and user space
> released packages
> - OFED will offer only backports to the distros  (no fixes)
> - OFED package will be used for easy installation of all packages in a
> friendly manner
>
> The main goals of this change:
> 1. Ensure OFED and the upstream kernel are the same
> 2. Provide customers a way to use the new features in latest kernels on
> existing distros
> 3. OFED qualification will contribute to the stability of the upstream code
>
> We think that at this point of the RDMA technology maturity this is the
> right way to go.
> In this way OFED is not conflicting with the kernel or the distros, and
> still provide a valuable value for early adopters of new features.
>
> Versions:
> We suggest that the OFED version will be the same as kernel.org
> For example, for kernel 3.2 the OFED release would be OFED-3.2.
> This would make it easy for people to associate the OFED code with the
> corresponding kernel.org code.
>
> Some open questions that we should consider:
> - How to handle experimental features?
> - Need to follow up kernel stable releases if bug fixes are relevant to
> OFA modules
> - Should we have a release for every kernel release (I think yes)
> - What should we do with modules like SDP that are not in kernel?
>
> Comments and responses are welcome
>

Personally I would appreciate it a lot if everything that is not a kernel
module would be moved out of the kernel-ib RPM. That would make it a lot
easier to use the OFED user space components in combination with upstream
or distro-provided kernel IB modules. The relevant files are:

# rpm -ql kernel-ib | grep -v lib/modules
/etc/infiniband
/etc/infiniband/connectx.conf
/etc/infiniband/info
/etc/infiniband/openib.conf
/etc/init.d/openibd
/etc/modprobe.d/ib_ipoib.conf
/etc/modprobe.d/mlx4_en.conf
/etc/udev/rules.d/90-ib.rules
/sbin/connectx_port_config
/sbin/sysctl_perf_tuning
/usr/bin/ibdev2netdev

Bart.

[-- Attachment #1.2: Type: text/html, Size: 2641 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
ewg mailing list
ewg-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg