Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Mike Snitzer]

On 2/2/07, Chuck Lever <chuck.lever@oracle.com> wrote:
> Roland Dreier wrote:
> >  > They are mostly from Chuck Level and make preparating for IPv6 support
> >  > in the NFS server.
> >  > They are *not* for 2.6.20, but should be ok for .21.
> >
> > Out of curiousity, does this patch series reduce the delta between the
> > NFS/RDMA tree and mainline Linux?  In other words does this bring
> > NFS/RDMA closer to merging?
>
> Hi Roland-
>
> The client side support for an RPC/RDMA module is almost completely
> integrated into mainline.  There is still a minimal set of patches
> required to support alternate transports in loadable kernel modules
> which Trond has indicated he will integrate when the RPC/RDMA transport
> is ready to be integrated.

Hi Chuck,

I must be missing something because I don't see _any_ trace of the
core RPC over RDMA support (xprtrdma et al), your RPC Transport
Switch, or any of the other supporting changes in mainline.  Could
you, or others, please clarify the plan for merging RPC/RDMA?

> At this time I'm not aware of a plan to integrate server-side support
> for NFS/RDMA.  Perhaps the NetApp RDMA developers could respond.

Do you mean the NFS/RDMA server code from Tom Tucker of Open Grid
Computing?  Why wouldn't this code be included?

please advise, thanks.

Re: [NFS] Merge plans for RPC/RDMA? (Was: Re: [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Trond Myklebust]

On Fri, 2007-04-13 at 16:47 -0400, Mike Snitzer wrote:
> I must be missing something because I don't see _any_ trace of the
> core RPC over RDMA support (xprtrdma et al), your RPC Transport
> Switch, or any of the other supporting changes in mainline.  Could
> you, or others, please clarify the plan for merging RPC/RDMA?

That would be a question for the actual RPC/RDMA developers (i.e. James
Lentini and Tom Talpey). They haven't submitted any code for review yet.


> Do you mean the NFS/RDMA server code from Tom Tucker of Open Grid
> Computing?  Why wouldn't this code be included?

That too has never been submitted for review AFAIK. Why would we include
it?

Cheers
  Trond

Re: Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Mike Snitzer wrote:
> On 2/2/07, Chuck Lever <chuck.lever@oracle.com> wrote:
>> Roland Dreier wrote:
>> >  > They are mostly from Chuck Level and make preparating for IPv6 
>> support
>> >  > in the NFS server.
>> >  > They are *not* for 2.6.20, but should be ok for .21.
>> >
>> > Out of curiousity, does this patch series reduce the delta between the
>> > NFS/RDMA tree and mainline Linux?  In other words does this bring
>> > NFS/RDMA closer to merging?
>>
>> Hi Roland-
>>
>> The client side support for an RPC/RDMA module is almost completely
>> integrated into mainline.  There is still a minimal set of patches
>> required to support alternate transports in loadable kernel modules
>> which Trond has indicated he will integrate when the RPC/RDMA transport
>> is ready to be integrated.
> 
> Hi Chuck,
> 
> I must be missing something because I don't see _any_ trace of the
> core RPC over RDMA support (xprtrdma et al), your RPC Transport
> Switch, or any of the other supporting changes in mainline.  Could
> you, or others, please clarify the plan for merging RPC/RDMA?

The RPC transport switch patches are almost fully integrated into 
mainline.  The xprtrdma piece is what is not there yet.

>> At this time I'm not aware of a plan to integrate server-side support
>> for NFS/RDMA.  Perhaps the NetApp RDMA developers could respond.
> 
> Do you mean the NFS/RDMA server code from Tom Tucker of Open Grid
> Computing?  Why wouldn't this code be included?

Tom's code hasn't been reviewed by the community.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Mike Snitzer]

On 4/13/07, Chuck Lever <chuck.lever@oracle.com> wrote:
> Mike Snitzer wrote:
> > On 2/2/07, Chuck Lever <chuck.lever@oracle.com> wrote:
> >> Roland Dreier wrote:
> >> >  > They are mostly from Chuck Level and make preparating for IPv6
> >> support
> >> >  > in the NFS server.
> >> >  > They are *not* for 2.6.20, but should be ok for .21.
> >> >
> >> > Out of curiousity, does this patch series reduce the delta between the
> >> > NFS/RDMA tree and mainline Linux?  In other words does this bring
> >> > NFS/RDMA closer to merging?
> >>
> >> Hi Roland-
> >>
> >> The client side support for an RPC/RDMA module is almost completely
> >> integrated into mainline.  There is still a minimal set of patches
> >> required to support alternate transports in loadable kernel modules
> >> which Trond has indicated he will integrate when the RPC/RDMA transport
> >> is ready to be integrated.
> >
> > Hi Chuck,
> >
> > I must be missing something because I don't see _any_ trace of the
> > core RPC over RDMA support (xprtrdma et al), your RPC Transport
> > Switch, or any of the other supporting changes in mainline.  Could
> > you, or others, please clarify the plan for merging RPC/RDMA?
>
> The RPC transport switch patches are almost fully integrated into
> mainline.  The xprtrdma piece is what is not there yet.

OK, has the xprtrdma piece been reviewed by the community?  What, if
anything, is preventing the code from being included in -mm for wider
testing?

Has xprtrdma inclusion/progress been discussed on some other mailing
list?  Any insight would be appreciated.

Re: Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]

Mike Snitzer wrote:
> On 4/13/07, Chuck Lever <chuck.lever@oracle.com> wrote:
>> Mike Snitzer wrote:
>> > On 2/2/07, Chuck Lever <chuck.lever@oracle.com> wrote:
>> >> Roland Dreier wrote:
>> >> >  > They are mostly from Chuck Level and make preparating for IPv6
>> >> support
>> >> >  > in the NFS server.
>> >> >  > They are *not* for 2.6.20, but should be ok for .21.
>> >> >
>> >> > Out of curiousity, does this patch series reduce the delta 
>> between the
>> >> > NFS/RDMA tree and mainline Linux?  In other words does this bring
>> >> > NFS/RDMA closer to merging?
>> >>
>> >> Hi Roland-
>> >>
>> >> The client side support for an RPC/RDMA module is almost completely
>> >> integrated into mainline.  There is still a minimal set of patches
>> >> required to support alternate transports in loadable kernel modules
>> >> which Trond has indicated he will integrate when the RPC/RDMA 
>> transport
>> >> is ready to be integrated.
>> >
>> > Hi Chuck,
>> >
>> > I must be missing something because I don't see _any_ trace of the
>> > core RPC over RDMA support (xprtrdma et al), your RPC Transport
>> > Switch, or any of the other supporting changes in mainline.  Could
>> > you, or others, please clarify the plan for merging RPC/RDMA?
>>
>> The RPC transport switch patches are almost fully integrated into
>> mainline.  The xprtrdma piece is what is not there yet.
> 
> OK, has the xprtrdma piece been reviewed by the community?  What, if
> anything, is preventing the code from being included in -mm for wider
> testing?

Some of the main NFS client developers (Trond, the CITI-UM developers, 
myself) have reviewed previous versions of this code, and have 
recommended a number of changes.  I haven't heard anything recently from 
the NetApp engineers working on this.  We are waiting for them to 
publish again, but I think they are waiting for the last of the 
transport switch patches to trickle into mainline (which should be ready 
with the release of 2.6.21).  I know they have been working with a 
handful of test sites and have achieved good performance results.

> Has xprtrdma inclusion/progress been discussed on some other mailing
> list?

Not yet, but the place that is likely to occur is nfs@lists.sourceforge.net.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: Merge plans for RPC/RDMA? (Was: Re: [NFS] [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

Mike Snitzer wrote:
> On 4/13/07, Chuck Lever <chuck.lever@oracle.com> wrote:
>> Mike Snitzer wrote:
>> > On 2/2/07, Chuck Lever <chuck.lever@oracle.com> wrote:
>> >> Roland Dreier wrote:
>> >> >  > They are mostly from Chuck Level and make preparating for IPv6
>> >> support
>> >> >  > in the NFS server.
>> >> >  > They are *not* for 2.6.20, but should be ok for .21.
>> >> >
>> >> > Out of curiousity, does this patch series reduce the delta 
>> between the
>> >> > NFS/RDMA tree and mainline Linux?  In other words does this bring
>> >> > NFS/RDMA closer to merging?
>> >>
>> >> Hi Roland-
>> >>
>> >> The client side support for an RPC/RDMA module is almost completely
>> >> integrated into mainline.  There is still a minimal set of patches
>> >> required to support alternate transports in loadable kernel modules
>> >> which Trond has indicated he will integrate when the RPC/RDMA 
>> transport
>> >> is ready to be integrated.
>> >
>> > Hi Chuck,
>> >
>> > I must be missing something because I don't see _any_ trace of the
>> > core RPC over RDMA support (xprtrdma et al), your RPC Transport
>> > Switch, or any of the other supporting changes in mainline.  Could
>> > you, or others, please clarify the plan for merging RPC/RDMA?
>>
>> The RPC transport switch patches are almost fully integrated into
>> mainline.  The xprtrdma piece is what is not there yet.
> 
> OK, has the xprtrdma piece been reviewed by the community?  What, if
> anything, is preventing the code from being included in -mm for wider
> testing?

OK, there is something holding up the client-side piece of NFS/RDMA. 
There is some hackery required for the NFS client to recognize that the 
RDMA transport should be used instead of the standard socket transport.

We did discuss this a bit at Connectathon '07 six weeks ago, but my 
impression is that more discussion is required.

I'm working on some changes to NFS mounting that would move NFS mount 
option parsing into the kernel.  This would make it very simple to add 
RDMA transport support, but it's rather a while in coming.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

Re: [NFS] Merge plans for RPC/RDMA? (Was: Re: [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Mike Snitzer]

On 4/13/07, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> On Fri, 2007-04-13 at 16:47 -0400, Mike Snitzer wrote:
> > I must be missing something because I don't see _any_ trace of the
> > core RPC over RDMA support (xprtrdma et al), your RPC Transport
> > Switch, or any of the other supporting changes in mainline.  Could
> > you, or others, please clarify the plan for merging RPC/RDMA?
>
> That would be a question for the actual RPC/RDMA developers (i.e. James
> Lentini and Tom Talpey). They haven't submitted any code for review yet.

The reason I asked is there seems to be a catch-22 going on here if
(as Chuck indicated) the NetApp engineers are waiting for the
remaining rpc transport switch patches to be merged.  My naive
understanding is that those remaining transport switch patches aren't
_really_ needed without the RPC/RDMA patches.

Essentially I'm echoing Or Gerlitz's post to openib-general back in December:
http://lists.openfabrics.org/pipermail/general/2006-December/029721.html

If so, it begs the question: why the hold up from the NetApp
engineers?  In the hopes of getting some insight I've cc'd James
Lentini and Tom Talpey.

I'm assuming that all of you obviously want your code merged mainline.
 The thing that is puzzling is the non-traditional release management
of this RPC/RDMA and NFS/RDMA code.  Aside from the periodic
announcements of the tarball updates to the sf.net nfs-rdma project;
I've not found any posting/discussion of associated patches to the
various mailing lists (nfs, nfsv4, ofa's general, lkml, etc).

I'm interested in understanding why the reluctance to push for a merge
now (let alone some months ago) given the various successes that have
been seen with the NFS/RDMA effort (Sandia, SC '06, as Chuck noted:
good performance at various test sites).  If the code is holding up
well why the delay in review and merging?  And rather than wait for
the remaining transport switch patches; why not look to merge all of
NFS/RDMA (client) work at the same time?

regards,
Mike

Re: [NFS] Merge plans for RPC/RDMA? (Was: Re: [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[James Lentini]

On Sat, 14 Apr 2007, Mike Snitzer wrote:

> On 4/13/07, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > On Fri, 2007-04-13 at 16:47 -0400, Mike Snitzer wrote:
> > > I must be missing something because I don't see _any_ trace of the
> > > core RPC over RDMA support (xprtrdma et al), your RPC Transport
> > > Switch, or any of the other supporting changes in mainline.  Could
> > > you, or others, please clarify the plan for merging RPC/RDMA?
> > 
> > That would be a question for the actual RPC/RDMA developers (i.e. James
> > Lentini and Tom Talpey). They haven't submitted any code for review yet.
> 
> The reason I asked is there seems to be a catch-22 going on here if
> (as Chuck indicated) the NetApp engineers are waiting for the
> remaining rpc transport switch patches to be merged.  My naive
> understanding is that those remaining transport switch patches aren't
> _really_ needed without the RPC/RDMA patches.
> 
> Essentially I'm echoing Or Gerlitz's post to openib-general back in December:
> http://lists.openfabrics.org/pipermail/general/2006-December/029721.html
> 
> If so, it begs the question: why the hold up from the NetApp
> engineers?  In the hopes of getting some insight I've cc'd James
> Lentini and Tom Talpey.
> 
> I'm assuming that all of you obviously want your code merged mainline.
> The thing that is puzzling is the non-traditional release management
> of this RPC/RDMA and NFS/RDMA code.  Aside from the periodic
> announcements of the tarball updates to the sf.net nfs-rdma project;
> I've not found any posting/discussion of associated patches to the
> various mailing lists (nfs, nfsv4, ofa's general, lkml, etc).
> 
> I'm interested in understanding why the reluctance to push for a merge
> now (let alone some months ago) given the various successes that have
> been seen with the NFS/RDMA effort (Sandia, SC '06, as Chuck noted:
> good performance at various test sites).  If the code is holding up
> well why the delay in review and merging?  And rather than wait for
> the remaining transport switch patches; why not look to merge all of
> NFS/RDMA (client) work at the same time?

There is no reluctance to submit the NFS-RDMA code for review. There 
have been several upstream changes to the NFS server recently. We 
(NetApp and Tom Tucker at OGC) are working on a new patchset for 
NFS-RDMA that is compatible with these changes. We are testing the 
code now. 

We will post the patches for review when we finish updating them.

Re: [NFS] Merge plans for RPC/RDMA? (Was: Re: [PATCH 000 of 14] knfsd: Preparation for IPv6 support in NFS server.)

[Mike Snitzer]

On 4/16/07, James Lentini <jlentini@netapp.com> wrote:
>
>
> On Sat, 14 Apr 2007, Mike Snitzer wrote:
>
> > I'm interested in understanding why the reluctance to push for a merge
> > now (let alone some months ago) given the various successes that have
> > been seen with the NFS/RDMA effort (Sandia, SC '06, as Chuck noted:
> > good performance at various test sites).  If the code is holding up
> > well why the delay in review and merging?  And rather than wait for
> > the remaining transport switch patches; why not look to merge all of
> > NFS/RDMA (client) work at the same time?
>
> There is no reluctance to submit the NFS-RDMA code for review. There
> have been several upstream changes to the NFS server recently. We
> (NetApp and Tom Tucker at OGC) are working on a new patchset for
> NFS-RDMA that is compatible with these changes. We are testing the
> code now.
>
> We will post the patches for review when we finish updating them.

Thanks for the clarification.  It was unfair of me to imply any amount
of "reluctance" to submit the NFS/RDMA code for review.  I was just
trying to gauge the "real soon now" type posts we've seen over the
past few months.  If anything I'd imagine the latest NFS churn has
given you additional incentive to merge.

Thanks for your continued hard work; NFS/RDMA looks to be a huge
improvement for modern NFS deployments.

regards,
Mike