Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Trond Myklebust]

On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote: 
> Hi,
> 
> There seem to be legitimate reasons for an (NFSv4.1) client and/or server to prefer a dedicated callback channel.

That would be an assertion that is missing a lot of context and
explanation. Why should my client bother to support such a server?

> If a server wants this result, it seems from the language of 18.36.3 that it should indicate it by not setting CREATE_SESSION4_FLAG_CONN_BACK_CHAN in csr_flags in the CREATE_SESSION response, presuming the flag is set in the corresponding csa_flags argument (it's not allowed to set it otherwise).  The client may respond with BIND_CONN_TO_SESSION on a new channel, setting bctsa_dir to CDFC4_BACK.

Nope. The exact wording is:

         If CREATE_SESSION4_FLAG_CONN_BACK_CHAN is set in csa_flags, the
         client is requesting that the connection over which the
         CREATE_SESSION operation arrived be associated with the
         session's backchannel in addition to its fore channel.  If the
         server agrees, it sets CREATE_SESSION4_FLAG_CONN_BACK_CHAN in
         the result field csr_flags.  If
         CREATE_SESSION4_FLAG_CONN_BACK_CHAN is not set in csa_flags,
         then CREATE_SESSION4_FLAG_CONN_BACK_CHAN MUST NOT be set in
         csr_flags.

I see nothing there to indicate that the server is able to request a
dedicated backchannel. All it says is that the server may refuse a
backchannel on this particular connection.

> Currently, the Linux and I believe also the CITI Windows client always propose channels in both directions.  The Linux mainline Linux client doesn't know how to BIND_CONN_TO_SESSION, so trivially it won't negotiate any back channel if a server didn't agree to both directions today, either.  I've experimentally implemented a "fallback" model in a Linux client and (partly in a) Ganesha server.  I'd appreciate any feedback on the idea.

Yep. As I said, why should we bother adding support for servers that
don't? I can function perfectly well without pNFS support or delegation
support in such a case. Performance will suck, but why do I care?

Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Trond Myklebust]

On Wed, 2011-10-05 at 23:28 -0400, Trond Myklebust wrote: 
> On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote: 
> > Currently, the Linux and I believe also the CITI Windows client always propose channels in both directions.  The Linux mainline Linux client doesn't know how to BIND_CONN_TO_SESSION, so trivially it won't negotiate any back channel if a server didn't agree to both directions today, either.  I've experimentally implemented a "fallback" model in a Linux client and (partly in a) Ganesha server.  I'd appreciate any feedback on the idea.
> 
> Yep. As I said, why should we bother adding support for servers that
> don't? I can function perfectly well without pNFS support or delegation
> support in such a case. Performance will suck, but why do I care?

To put it in more basic terms: what you are proposing will add
development costs to the client and and an extra code burden to maintain
long term. So what is in it for me?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Matt W. Benjamin]

Hi Trond,

The rationale for making it possible to arrive at a dedicated back channel for a session is, I think, mainly that it can yield greater throughput, a benefit which grows as workload increases.  Conversely, channel negotiation appears to not have many downsides.

My reasoning would be mainly that the dedicated configuration appears to minimise the throughput potentially lost to flow control or head-of-line blocking, in particular when the the underlying transport is TCP.  A slightly different way to look at it is, that it enables a simpler RPC dispatcher at both endpoints.  (I doubt the latter is more than a short-term consideration.  Still, having an ability to work around a class of transport/interop problems transparently, which the fallback strategy could permit, might be viewed as a win.)

If one is persuaded of the potential utility of a dedicated session back channel, it still might not appear clearly better to have the client initially propose a mixed channel and assume the server will object if it cares (as opposed to making a dedicated fore channel its initial proposal).  I think the fallback strategy is slightly better.  It offers both client and server input into the transport setup, and it has the benefit of being closest to what 41 servers currently do and the Linux client currently does.

As you correctly state, the client could proceed without a back channel in the presence of a "disagreeing" server and still be conformant with RFC 5661 (if perhaps not in spirit), but actually, I'm not certain that the Linux client actually does proceed as specified in this case, as presently implemented (3.1.0-rc8)--I think it may incorrectly assume that a mixed channel has been agreed on.

Back channel negotiation does what it does, seemingly, at very little cost in development (a few hundred lines, mostly structure formulas).  Maintenance costs are said to be proportional to code size (or so I have read, e.g., Page-Jones).  I'm not making light of this point, obviously everything has a cost.

Matt

----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:

> On Wed, 2011-10-05 at 23:28 -0400, Trond Myklebust wrote: 
> > On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote: 
> > > Currently, the Linux and I believe also the CITI Windows client
> always propose channels in both directions.  The Linux mainline Linux
> client doesn't know how to BIND_CONN_TO_SESSION, so trivially it won't
> negotiate any back channel if a server didn't agree to both directions
> today, either.  I've experimentally implemented a "fallback" model in
> a Linux client and (partly in a) Ganesha server.  I'd appreciate any
> feedback on the idea.
> > 
> > Yep. As I said, why should we bother adding support for servers
> that
> > don't? I can function perfectly well without pNFS support or
> delegation
> > support in such a case. Performance will suck, but why do I care?
> 
> To put it in more basic terms: what you are proposing will add
> development costs to the client and and an extra code burden to
> maintain
> long term. So what is in it for me?
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

RE: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Myklebust, Trond]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBNYXR0IFcuIEJlbmphbWluIFtt
YWlsdG86bWF0dEBsaW51eGJveC5jb21dDQo+IFNlbnQ6IFRodXJzZGF5LCBPY3RvYmVyIDA2LCAy
MDExIDExOjExIEFNDQo+IFRvOiBNeWtsZWJ1c3QsIFRyb25kDQo+IENjOiBuZnN2NDsgbGludXgt
bmZzOyBuZnMtZ2FuZXNoYS1kZXZlbA0KPiBTdWJqZWN0OiBSZTogW25mc3Y0XSBiYWNrIGNoYW5u
ZWwgZmxhZ3MsIENSRUFURV9TRVNTSU9OLA0KPiBCSU5EX0NPTk5fVE9fU0VTU0lPTg0KPiANCj4g
SGkgVHJvbmQsDQo+IA0KPiBUaGUgcmF0aW9uYWxlIGZvciBtYWtpbmcgaXQgcG9zc2libGUgdG8g
YXJyaXZlIGF0IGEgZGVkaWNhdGVkIGJhY2sgY2hhbm5lbCBmb3IgYQ0KPiBzZXNzaW9uIGlzLCBJ
IHRoaW5rLCBtYWlubHkgdGhhdCBpdCBjYW4geWllbGQgZ3JlYXRlciB0aHJvdWdocHV0LCBhIGJl
bmVmaXQgd2hpY2gNCj4gZ3Jvd3MgYXMgd29ya2xvYWQgaW5jcmVhc2VzLiAgQ29udmVyc2VseSwg
Y2hhbm5lbCBuZWdvdGlhdGlvbiBhcHBlYXJzIHRvDQo+IG5vdCBoYXZlIG1hbnkgZG93bnNpZGVz
Lg0KDQpXaHkgZG8gd2UgbmVlZCBzdWNoIGEgaHVnZSB2b2x1bWUgb2YgY2FsbGJhY2tzPyBUaGV5
IHNob3VsZCBiZSB0aGUgZXhjZXB0aW9uIHJhdGhlciB0aGFuIHRoZSBydWxlLCBzaW5jZSB0aGVp
ciBlZmZlY3QgaXMgYWx3YXlzIHRvIHNsb3cgZG93biBwZXJmb3JtYW5jZS4gVGhyb3VnaHB1dCBp
cyBpbiBhZGRpdGlvbiBsaW1pdGVkIGJ5IHRoZSBudW1iZXIgb2Ygc2Vzc2lvbiBzbG90cyAoY3Vy
cmVudGx5IDEpLg0KDQo+IE15IHJlYXNvbmluZyB3b3VsZCBiZSBtYWlubHkgdGhhdCB0aGUgZGVk
aWNhdGVkIGNvbmZpZ3VyYXRpb24gYXBwZWFycyB0bw0KPiBtaW5pbWlzZSB0aGUgdGhyb3VnaHB1
dCBwb3RlbnRpYWxseSBsb3N0IHRvIGZsb3cgY29udHJvbCBvciBoZWFkLW9mLWxpbmUNCj4gYmxv
Y2tpbmcsIGluIHBhcnRpY3VsYXIgd2hlbiB0aGUgdGhlIHVuZGVybHlpbmcgdHJhbnNwb3J0IGlz
IFRDUC4gIEEgc2xpZ2h0bHkNCj4gZGlmZmVyZW50IHdheSB0byBsb29rIGF0IGl0IGlzLCB0aGF0
IGl0IGVuYWJsZXMgYSBzaW1wbGVyIFJQQyBkaXNwYXRjaGVyIGF0IGJvdGgNCj4gZW5kcG9pbnRz
LiAgKEkgZG91YnQgdGhlIGxhdHRlciBpcyBtb3JlIHRoYW4gYSBzaG9ydC10ZXJtIGNvbnNpZGVy
YXRpb24uICBTdGlsbCwNCj4gaGF2aW5nIGFuIGFiaWxpdHkgdG8gd29yayBhcm91bmQgYSBjbGFz
cyBvZiB0cmFuc3BvcnQvaW50ZXJvcCBwcm9ibGVtcw0KPiB0cmFuc3BhcmVudGx5LCB3aGljaCB0
aGUgZmFsbGJhY2sgc3RyYXRlZ3kgY291bGQgcGVybWl0LCBtaWdodCBiZSB2aWV3ZWQgYXMgYQ0K
PiB3aW4uKQ0KDQpGb3Igd2hvbT8gV2UgaGF2ZSBhbHJlYWR5IGltcGxlbWVudGVkIGEgY2FsbGJh
Y2sgbW9kZWwgdGhhdCBpcyB3b3JraW5nIGZpbmUgZm9yIHVzLiBJIGhhdmUgeWV0IHRvIHNlZSBh
bnkgZXZpZGVuY2Ugb2YgdGhlIGNhbGxiYWNrIHNjYWxhYmlsaXR5IGJyZWFrZG93biBzY2VuYXJp
byB0aGF0IHlvdSBjbGFpbSBhcyBhIG1vdGl2YXRpb24uIFdoYXQgSSBoYXZlIChmcmVxdWVudGx5
KSBzZWVuIGlzIHNjYWxhYmlsaXR5IGlzc3VlcyBkdWUgdG8gY2xpZW50cyBhbmQgc2VydmVycyBy
dW5uaW5nIG91dCBvZiBmcmVlIFRDUCBwb3J0cy4NCg0KPiBJZiBvbmUgaXMgcGVyc3VhZGVkIG9m
IHRoZSBwb3RlbnRpYWwgdXRpbGl0eSBvZiBhIGRlZGljYXRlZCBzZXNzaW9uIGJhY2sNCj4gY2hh
bm5lbCwgaXQgc3RpbGwgbWlnaHQgbm90IGFwcGVhciBjbGVhcmx5IGJldHRlciB0byBoYXZlIHRo
ZSBjbGllbnQgaW5pdGlhbGx5DQo+IHByb3Bvc2UgYSBtaXhlZCBjaGFubmVsIGFuZCBhc3N1bWUg
dGhlIHNlcnZlciB3aWxsIG9iamVjdCBpZiBpdCBjYXJlcyAoYXMNCj4gb3Bwb3NlZCB0byBtYWtp
bmcgYSBkZWRpY2F0ZWQgZm9yZSBjaGFubmVsIGl0cyBpbml0aWFsIHByb3Bvc2FsKS4gIEkgdGhp
bmsgdGhlDQo+IGZhbGxiYWNrIHN0cmF0ZWd5IGlzIHNsaWdodGx5IGJldHRlci4gIEl0IG9mZmVy
cyBib3RoIGNsaWVudCBhbmQgc2VydmVyIGlucHV0IGludG8NCj4gdGhlIHRyYW5zcG9ydCBzZXR1
cCwgYW5kIGl0IGhhcyB0aGUgYmVuZWZpdCBvZiBiZWluZyBjbG9zZXN0IHRvIHdoYXQgNDEgc2Vy
dmVycw0KPiBjdXJyZW50bHkgZG8gYW5kIHRoZSBMaW51eCBjbGllbnQgY3VycmVudGx5IGRvZXMu
DQo+DQo+IEFzIHlvdSBjb3JyZWN0bHkgc3RhdGUsIHRoZSBjbGllbnQgY291bGQgcHJvY2VlZCB3
aXRob3V0IGEgYmFjayBjaGFubmVsIGluIHRoZQ0KPiBwcmVzZW5jZSBvZiBhICJkaXNhZ3JlZWlu
ZyIgc2VydmVyIGFuZCBzdGlsbCBiZSBjb25mb3JtYW50IHdpdGggUkZDIDU2NjEgKGlmDQo+IHBl
cmhhcHMgbm90IGluIHNwaXJpdCksIGJ1dCBhY3R1YWxseSwgSSdtIG5vdCBjZXJ0YWluIHRoYXQg
dGhlIExpbnV4IGNsaWVudCBhY3R1YWxseQ0KPiBkb2VzIHByb2NlZWQgYXMgc3BlY2lmaWVkIGlu
IHRoaXMgY2FzZSwgYXMgcHJlc2VudGx5IGltcGxlbWVudGVkICgzLjEuMC1yYzgpLS0NCj4gSSB0
aGluayBpdCBtYXkgaW5jb3JyZWN0bHkgYXNzdW1lIHRoYXQgYSBtaXhlZCBjaGFubmVsIGhhcyBi
ZWVuIGFncmVlZCBvbi4NCg0KSSBhZ3JlZSB0aGF0IHdlIHNob3VsZCBmaXggdGhhdCBidWcuIEkg
ZGlzYWdyZWUgdGhhdCB3ZSBuZWVkIGEgc2Vjb25kIGNhbGxiYWNrIG1vZGVsIHRvIGRvIHNvLiBK
dXN0IHR1cm4gb2ZmIHBORlMgKGFuZCBwb3NzaWJseSBXQU5UcyBpZiB3ZSBldmVyIGltcGxlbWVu
dCB0aGVtKSwgYW5kIHdlIHNob3VsZCBiZSBmaW5lLg0KDQo+IEJhY2sgY2hhbm5lbCBuZWdvdGlh
dGlvbiBkb2VzIHdoYXQgaXQgZG9lcywgc2VlbWluZ2x5LCBhdCB2ZXJ5IGxpdHRsZSBjb3N0IGlu
DQo+IGRldmVsb3BtZW50IChhIGZldyBodW5kcmVkIGxpbmVzLCBtb3N0bHkgc3RydWN0dXJlIGZv
cm11bGFzKS4NCj4gTWFpbnRlbmFuY2UgY29zdHMgYXJlIHNhaWQgdG8gYmUgcHJvcG9ydGlvbmFs
IHRvIGNvZGUgc2l6ZSAob3Igc28gSSBoYXZlIHJlYWQsDQo+IGUuZy4sIFBhZ2UtSm9uZXMpLiAg
SSdtIG5vdCBtYWtpbmcgbGlnaHQgb2YgdGhpcyBwb2ludCwgb2J2aW91c2x5IGV2ZXJ5dGhpbmcg
aGFzDQo+IGEgY29zdC4NCg0KTXkgb2JqZWN0aW9uIGlzIHRvIHRoZSBsYWNrIG9mIGEgY29uc2Vu
c3VzIG9uIGEgc2luZ2xlIHN5c3RlbSBmb3IgaW1wbGVtZW50aW5nIGZlYXR1cmVzLiBJJ20gdGly
ZWQgb2YgYmVpbmcgdG9sZCB0aGF0IHRoZSBzcGVjIGFsbG93cyB5b3UgdG8gZG8gdGhlIHNhbWUg
dGhpbmcgaW4gMTAgZGlmZmVyZW50IHdheXMsIHdpdGggYW4gZXhwZWN0YXRpb24gdGhhdCB3ZSBz
aG91bGQgaW1wbGVtZW50IGFsbCAxMCB3YXlzLg0KSWYgd2UgY2FuJ3QgZmluZCBhIHNpbmdsZSBn
b29kIHdheSBvZiBpbXBsZW1lbnRpbmcgYSBmZWF0dXJlLCB0aGVuIG15IHByZWZlcmVuY2UgaXMg
dG8gZHJvcCBzdXBwb3J0IGZvciB0aGF0IGZlYXR1cmUgYWx0b2dldGhlciAob3IgdG8gY2hvb3Nl
IG9uZSBpbXBsZW1lbnRhdGlvbiBhbmQgdG8gc3RpY2sgd2l0aCBpdCkuIE15IGV4cGVjdGF0aW9u
IGZvciBhIHN0YW5kYXJkIGlzIHRoYXQgaXQgc2hvdWxkIGFpbSB0byBfcmVkdWNlXyB0aGUgbnVt
YmVyIG9mIGRpZmZlcmVudCBpbXBsZW1lbnRhdGlvbnMgc28gdGhhdCB3ZSBjYW4gY29uY2VudHJh
dGUgb3VyIGRldmVsb3BtZW50IGFuZCB0ZXN0aW5nIGVmZm9ydHMuIChCVFc6IHBORlMgaXMgZGVm
aW5pdGVseSBub3QgYmV5b25kIGNyaXRpY2lzbSBpbiB0aGlzIHJlc3BlY3QuKQ0KDQpJT1c6IEkg
Y2FuIHNlZSB2YWxpZCByZWFzb25zIGZvciB3aHkgd2Ugc2hvdWxkIHRlc3QgdGhlIGNhc2Ugd2hl
cmUgdGhlIHNlcnZlciByZWZ1c2VzIGEgbWl4ZWQgZm9yZStiYWNrIGNoYW5uZWwsIGJ1dCBJIGRv
bid0IHNlZSB0aGF0IGFzIGEgcmVhc29uIHRvIGltcGxlbWVudCBhIHNlY29uZCBiYWNrIGNoYW5u
ZWwgbW9kZWwuIFRoYXQgcmVxdWlyZXMgdXMgdG8gYWRkIGNvZGUgKyB0ZXN0cyAoYW5kIHBlcmZv
cm0gcmVndWxhciByZWdyZXNzaW9uIHRlc3RzKSBmb3IgdGhhdCBiYWNrIGNoYW5uZWwgbW9kZSwg
YXMgd2VsbCBhcyBkZWFsaW5nIHdpdGggdGhlIGNhc2Ugd2hlcmUgdGhhdCBtb2RlbCBvZiBvcGVy
YXRpb24gdG9vIGlzIHJlamVjdGVkIGJ5IHRoZSBzZXJ2ZXIuDQoNClRyb25kDQo=

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Matt W. Benjamin]

Hi Trond,

> For whom? We have already implemented a callback model that is working
> fine for us. I have yet to see any evidence of the callback
> scalability breakdown scenario that you claim as a motivation. What I
> have (frequently) seen is scalability issues due to clients and
> servers running out of free TCP ports.

For whom?  Well, I work primarily on servers.  I have worked experimentally on the Linux v41 client.  I am pretty sure I wasn't speaking of a "callback scalability breakdown."  I was discussing a tradeoff of ports for reduced overhead of flow control, lock contention, etc.  I wasn't trying to start an inflated theory discussion about these, I don't feel fully at ease doing so.  Still, I don't intuit that potential for port exhaustion trumps other considerations, it is one consideration among several.  I also mentioned some potential benefit for server implementations, such as the one we have been collaborating on.  I'd like to maximize its potential for success.

----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:
>
> My objection is to the lack of a consensus on a single system for
> implementing features. I'm tired of being told that the spec allows
> you to do the same thing in 10 different ways, with an expectation
> that we should implement all 10 ways.
> If we can't find a single good way of implementing a feature, then my
> preference is to drop support for that feature altogether (or to
> choose one implementation and to stick with it). My expectation for a
> standard is that it should aim to _reduce_ the number of different
> implementations so that we can concentrate our development and testing
> efforts. (BTW: pNFS is definitely not beyond criticism in this
> respect.)

You think the standard should influence, so as to reduce, the number of competing implementations?  Maybe you meant 'implementation choices for <x> feature.'

My expectation from NFSv4 is that there is some room for innovation and experimentation.  I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.

>
> IOW: I can see valid reasons for why we should test the case where the
> server refuses a mixed fore+back channel, but I don't see that as a
> reason to implement a second back channel model. That requires us to
> add code + tests (and perform regular regression tests) for that back
> channel mode, as well as dealing with the case where that model of
> operation too is rejected by the server.

I find I need to keep reminding my self what "assume" does.  But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and perhaps was implemented in some environment.  If so, perhaps such a person will chime in with an opinion.

>
> Trond

Thanks,

Matt

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Rick Macklem]

Trond Myklebust wrote:
> On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote:
> > Hi,
> >
> > There seem to be legitimate reasons for an (NFSv4.1) client and/or
> > server to prefer a dedicated callback channel.
> 
> That would be an assertion that is missing a lot of context and
> explanation. Why should my client bother to support such a server?
> 
> > If a server wants this result, it seems from the language of 18.36.3
> > that it should indicate it by not setting
> > CREATE_SESSION4_FLAG_CONN_BACK_CHAN in csr_flags in the
> > CREATE_SESSION response, presuming the flag is set in the
> > corresponding csa_flags argument (it's not allowed to set it
> > otherwise). The client may respond with BIND_CONN_TO_SESSION on a
> > new channel, setting bctsa_dir to CDFC4_BACK.
> 
> Nope. The exact wording is:
> 
> If CREATE_SESSION4_FLAG_CONN_BACK_CHAN is set in csa_flags, the
> client is requesting that the connection over which the
> CREATE_SESSION operation arrived be associated with the
> session's backchannel in addition to its fore channel. If the
> server agrees, it sets CREATE_SESSION4_FLAG_CONN_BACK_CHAN in
> the result field csr_flags. If
> CREATE_SESSION4_FLAG_CONN_BACK_CHAN is not set in csa_flags,
> then CREATE_SESSION4_FLAG_CONN_BACK_CHAN MUST NOT be set in
> csr_flags.
> 
> I see nothing there to indicate that the server is able to request a
> dedicated backchannel. All it says is that the server may refuse a
> backchannel on this particular connection.
> 
> > Currently, the Linux and I believe also the CITI Windows client
> > always propose channels in both directions. The Linux mainline Linux
> > client doesn't know how to BIND_CONN_TO_SESSION, so trivially it
> > won't negotiate any back channel if a server didn't agree to both
> > directions today, either. I've experimentally implemented a
> > "fallback" model in a Linux client and (partly in a) Ganesha server.
> > I'd appreciate any feedback on the idea.
> 
> Yep. As I said, why should we bother adding support for servers that
> don't? I can function perfectly well without pNFS support or
> delegation
> support in such a case. Performance will suck, but why do I care?
> 
Just fyi, what I have implemented (and I don't intend to change it) is
what Trond currently has done. For the FreeBSD 4.1 client, it will set
CREATE_SESSION4_FLAG_CONN_BACK_CHAN in csa_flags, but if the server
doesn`t set it in csr_flags, I simply assume `no backchannel`. I
don`t plan on implementing a dedicated back channel. (I suppose that
could change if there were strong evidence that a dedicated back channel
did improve performance significantly.)

I thought I`d post this mainly to show that there are actually times when I
agree with Trond;-) rick

RE: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Myklebust, Trond]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBSaWNrIE1hY2tsZW0gW21haWx0
bzpybWFja2xlbUB1b2d1ZWxwaC5jYV0NCj4gU2VudDogVGh1cnNkYXksIE9jdG9iZXIgMDYsIDIw
MTEgOTo0MiBQTQ0KPiBUbzogTXlrbGVidXN0LCBUcm9uZA0KPiBDYzogbGludXgtbmZzOyBuZnMt
Z2FuZXNoYS1kZXZlbDsgbmZzdjQ7IE1hdHQgVy4gQmVuamFtaW4NCj4gU3ViamVjdDogUmU6IFtu
ZnN2NF0gYmFjayBjaGFubmVsIGZsYWdzLCBDUkVBVEVfU0VTU0lPTiwNCj4gQklORF9DT05OX1RP
X1NFU1NJT04NCj4gDQo+IFRyb25kIE15a2xlYnVzdCB3cm90ZToNCj4gPiBPbiBXZWQsIDIwMTEt
MTAtMDUgYXQgMTk6MjEgLTA0MDAsIE1hdHQgVy4gQmVuamFtaW4gd3JvdGU6DQo+IEp1c3QgZnlp
LCB3aGF0IEkgaGF2ZSBpbXBsZW1lbnRlZCAoYW5kIEkgZG9uJ3QgaW50ZW5kIHRvIGNoYW5nZSBp
dCkgaXMgd2hhdA0KPiBUcm9uZCBjdXJyZW50bHkgaGFzIGRvbmUuIEZvciB0aGUgRnJlZUJTRCA0
LjEgY2xpZW50LCBpdCB3aWxsIHNldA0KPiBDUkVBVEVfU0VTU0lPTjRfRkxBR19DT05OX0JBQ0tf
Q0hBTiBpbiBjc2FfZmxhZ3MsIGJ1dCBpZiB0aGUgc2VydmVyDQo+IGRvZXNuYHQgc2V0IGl0IGlu
IGNzcl9mbGFncywgSSBzaW1wbHkgYXNzdW1lIGBubyBiYWNrY2hhbm5lbGAuIEkgZG9uYHQgcGxh
biBvbg0KPiBpbXBsZW1lbnRpbmcgYSBkZWRpY2F0ZWQgYmFjayBjaGFubmVsLiAoSSBzdXBwb3Nl
IHRoYXQgY291bGQgY2hhbmdlIGlmDQo+IHRoZXJlIHdlcmUgc3Ryb25nIGV2aWRlbmNlIHRoYXQg
YSBkZWRpY2F0ZWQgYmFjayBjaGFubmVsIGRpZCBpbXByb3ZlDQo+IHBlcmZvcm1hbmNlIHNpZ25p
ZmljYW50bHkuKQ0KPiANCj4gSSB0aG91Z2h0IElgZCBwb3N0IHRoaXMgbWFpbmx5IHRvIHNob3cg
dGhhdCB0aGVyZSBhcmUgYWN0dWFsbHkgdGltZXMgd2hlbiBJDQo+IGFncmVlIHdpdGggVHJvbmQ7
LSkgcmljaw0KDQpIZXkuLi4gRG9uJ3QgeW91IGFsd2F5cz8gOy0pDQoNCiAgICAgVHJvbmQNCg==

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Trond Myklebust]

On Thu, 2011-10-06 at 16:12 -0400, Matt W. Benjamin wrote: 
> Hi Trond,
> 
> > For whom? We have already implemented a callback model that is working
> > fine for us. I have yet to see any evidence of the callback
> > scalability breakdown scenario that you claim as a motivation. What I
> > have (frequently) seen is scalability issues due to clients and
> > servers running out of free TCP ports.
> 
> For whom?  Well, I work primarily on servers.  I have worked experimentally on the Linux v41 client.  I am pretty sure I wasn't speaking of a "callback scalability breakdown."  I was discussing a tradeoff of ports for reduced overhead of flow control, lock contention, etc.  I wasn't trying to start an inflated theory discussion about these, I don't feel fully at ease doing so.  Still, I don't intuit that potential for port exhaustion trumps other considerations, it is one consideration among several.  I also mentioned some potential benefit for server implementations, such as the one we have been collaborating on.  I'd like to maximize its potential for success.
> 
> ----- "Trond Myklebust" <Trond.Myklebust@netapp.com> wrote:
> >
> > My objection is to the lack of a consensus on a single system for
> > implementing features. I'm tired of being told that the spec allows
> > you to do the same thing in 10 different ways, with an expectation
> > that we should implement all 10 ways.
> > If we can't find a single good way of implementing a feature, then my
> > preference is to drop support for that feature altogether (or to
> > choose one implementation and to stick with it). My expectation for a
> > standard is that it should aim to _reduce_ the number of different
> > implementations so that we can concentrate our development and testing
> > efforts. (BTW: pNFS is definitely not beyond criticism in this
> > respect.)
> 
> You think the standard should influence, so as to reduce, the number of competing implementations?  Maybe you meant 'implementation choices for <x> feature.'

I meant to say exactly what I said above: that a standard should reduce
the number of _different_ ways to implement the exact same feature. Find
something that works, and then add it to the standard. Do not waste my
time by adding experimental features and then coming back 9 months after
the standard has been published and stating that 'I just discovered that
feature A is better implemented using method X rather than method Y'.
That's not a standard: it's an experiment in progress.

> My expectation from NFSv4 is that there is some room for innovation and experimentation.  I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.

I'm not trying to compete with ad-hoc storage protocols: they can always
find a niche market to peddle their ideas. Once somebody starts talking
about a standard, then the main concept is that there is room for
cooperation in order to achieve mutual benefit. I see no benefit here.

> > IOW: I can see valid reasons for why we should test the case where the
> > server refuses a mixed fore+back channel, but I don't see that as a
> > reason to implement a second back channel model. That requires us to
> > add code + tests (and perform regular regression tests) for that back
> > channel mode, as well as dealing with the case where that model of
> > operation too is rejected by the server.
> 
> I find I need to keep reminding my self what "assume" does.  But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and perhaps was implemented in some environment.  If so, perhaps such a person will chime in with an opinion.

Then you need to communicate that value to me. I never signed an
agreement to implement all of RFC5661.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Matt W. Benjamin]

Hi Trond,

I appreciate your point of view.  I appreciate Rick's comments.  I hope some other comments will trickle in.  Do I think you and Rick are saying the same thing?  Not really.  Do I think you're making sense when you imply I want to change the standard?  No, I don't follow that.  BIND_CONN_TO_SESSION is in the published standard.  Can I show a dedicated back channel improves performance?  Not at present.  That's a sensible question.  I think if the protocol in general is doing what it is intended to do, it should be possible for some workloads, at some point.  (If callbacks are, somehow generally detrimental to performance, as you state, I think maybe we have some more work to do.)  (I note only one more thing--I did raise this topic on list 12 months ago.  Only Bruce commented, at the time.)

Thanks again,

Matt

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

RE: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Myklebust, Trond]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBNYXR0IFcuIEJlbmphbWluIFtt
YWlsdG86bWF0dEBsaW51eGJveC5jb21dDQo+IFNlbnQ6IFRodXJzZGF5LCBPY3RvYmVyIDA2LCAy
MDExIDEwOjU1IFBNDQo+IFRvOiBNeWtsZWJ1c3QsIFRyb25kDQo+IENjOiBsaW51eC1uZnM7IG5m
cy1nYW5lc2hhLWRldmVsOyBuZnN2NA0KPiBTdWJqZWN0OiBSZTogW25mc3Y0XSBiYWNrIGNoYW5u
ZWwgZmxhZ3MsIENSRUFURV9TRVNTSU9OLA0KPiBCSU5EX0NPTk5fVE9fU0VTU0lPTg0KPiANCj4g
SGkgVHJvbmQsDQo+IA0KPiBJIGFwcHJlY2lhdGUgeW91ciBwb2ludCBvZiB2aWV3LiAgSSBhcHBy
ZWNpYXRlIFJpY2sncyBjb21tZW50cy4gIEkgaG9wZSBzb21lDQo+IG90aGVyIGNvbW1lbnRzIHdp
bGwgdHJpY2tsZSBpbi4gIERvIEkgdGhpbmsgeW91IGFuZCBSaWNrIGFyZSBzYXlpbmcgdGhlIHNh
bWUNCj4gdGhpbmc/ICBOb3QgcmVhbGx5LiAgRG8gSSB0aGluayB5b3UncmUgbWFraW5nIHNlbnNl
IHdoZW4geW91IGltcGx5IEkgd2FudCB0bw0KPiBjaGFuZ2UgdGhlIHN0YW5kYXJkPyAgTm8sIEkg
ZG9uJ3QgZm9sbG93IHRoYXQuICBCSU5EX0NPTk5fVE9fU0VTU0lPTiBpcyBpbg0KPiB0aGUgcHVi
bGlzaGVkIHN0YW5kYXJkLiAgQ2FuIEkgc2hvdyBhIGRlZGljYXRlZCBiYWNrIGNoYW5uZWwgaW1w
cm92ZXMNCg0KSSBuZXZlciBhY2N1c2VkIHlvdSBvZiB0cnlpbmcgdG8gY2hhbmdlIHRoZSBzdGFu
ZGFyZC4gV2hhdCBJIHNhaWQgd2FzIHRoYXQgeW91IG5lZWQgdG8gY29udmluY2UgbWUgdGhhdCB0
aGVyZSBpcyBhIGdvb2QgcmVhc29uIHRvIGFjY2VwdCB0d28gaW1wbGVtZW50YXRpb25zIG9mIHRo
ZSBzYW1lIGZlYXR1cmUgaW4gdGhlIGNsaWVudC4NCg0KPiBwZXJmb3JtYW5jZT8gIE5vdCBhdCBw
cmVzZW50LiAgVGhhdCdzIGEgc2Vuc2libGUgcXVlc3Rpb24uICBJIHRoaW5rIGlmIHRoZQ0KPiBw
cm90b2NvbCBpbiBnZW5lcmFsIGlzIGRvaW5nIHdoYXQgaXQgaXMgaW50ZW5kZWQgdG8gZG8sIGl0
IHNob3VsZCBiZSBwb3NzaWJsZSBmb3INCj4gc29tZSB3b3JrbG9hZHMsIGF0IHNvbWUgcG9pbnQu
ICAoSWYgY2FsbGJhY2tzIGFyZSwgc29tZWhvdyBnZW5lcmFsbHkNCj4gZGV0cmltZW50YWwgdG8g
cGVyZm9ybWFuY2UsIGFzIHlvdSBzdGF0ZSwgSSB0aGluayBtYXliZSB3ZSBoYXZlIHNvbWUgbW9y
ZQ0KPiB3b3JrIHRvIGRvLikgIChJIG5vdGUgb25seSBvbmUgbW9yZSB0aGluZy0tSSBkaWQgcmFp
c2UgdGhpcyB0b3BpYyBvbiBsaXN0IDEyDQo+IG1vbnRocyBhZ28uICBPbmx5IEJydWNlIGNvbW1l
bnRlZCwgYXQgdGhlIHRpbWUuKQ0KDQpGYWlyIGVub3VnaCwgYnV0IDEyIG1vbnRocyBhZ28sIHJm
YzU2NjEgd2FzIGluIGxhc3QgY2FsbCBzdGF0dXMsIGFuZCB3ZSB3ZXJlIGFsbCB0cnlpbmcgdG8g
bW92ZSBvbiB0byBmaW5hbCBpbXBsZW1lbnRhdGlvbnMuDQoNCkkgaGF2ZSBfYWx3YXlzXyBhcmd1
ZWQgdGhhdCBjYWxsYmFja3Mgc2hvdWxkIGJlIHRoZSBleGNlcHRpb24gYW5kIG5vdCB0aGUgcnVs
ZSwgZHVlIHRvIHRoZSBmYWN0IHRoYXQgaW4gcHJldHR5IG11Y2ggYWxsIGNhc2VzIHRoZSBzZXJ2
ZXIgd2lsbCBoYXZlIHRvIHNlbmQgb3V0IGEgTkZTNEVSUl9ERUxBWSByZXN1bHQgdG8gd2hhdGV2
ZXIgUlBDIGNhbGwgaXQgaXMgdGhhdCB0cmlnZ2VyZWQgYSBjYWxsYmFjay4gTm90aGluZyBuZXcg
dGhlcmUgaW4gTkZTdjQuMS4gSSdsbCBhZGQgdGhhdCBub3IgaGFzIGFueWJvZHkgc3VjY2Vzc2Z1
bGx5IGFyZ3VlZCB0aGF0IHRoZXJlIGlzIGFueSBjaGFuZ2UgaW4gTkZTdjQuMSBjb21wYXJlZCB0
byBvdXIgZXhpc3RpbmcgTkZTdjQgaW1wbGVtZW50YXRpb25zLiBORlN2NCBkZWxlZ2F0aW9uIGNh
bGxiYWNrcyBhcmUgKG1lYXN1cmFibHkpIGV4dHJlbWVseSBzbG93LCBhbmQgc28gSSdkIGV4cGVj
dCB0aGUgZXhhY3Qgc2FtZSBiZWhhdmlvciBmb3IgTkZTdjQuMSBsYXlvdXQgcmVjYWxscyBldGMu
IE5vdGhpbmcgaGFzIGNoYW5nZWQgaW4gTkZTdjQgY2FsbGJhY2sgdGVjaG5vbG9neSB0byB3YXJy
YW50IGFuIGV4cGVjdGF0aW9uIG9mIHBlcmZvcm1hbmNlIGltcHJvdmVtZW50cy4NCg0KSU9XOiBO
b2JvZHkgKGluY2x1ZGluZyB5b3UpIGhhcyBldmVyIGNvbnZpbmNlZCBtZSB0aGF0IHdlIG5lZWQg
dG8gY29uc2lkZXIgY2FsbGJhY2tzIGFzIGEgc2NhbGFiaWxpdHkgaXNzdWUuIEknZCByZXF1aXJl
IHNvbWUgcHJldHR5IGZpcm0gZXZpZGVuY2UgYmVmb3JlIEkgY291bGQgYWNjZXB0IHRoYXQgYXMg
YSBmYWN0LiBTbyBmYXIsIGFsbCBJIGhhdmUgc2VlbiBpcyBtb3JlIG9yIGxlc3MgZWxvcXVlbnQg
cHJvc2UuIE5vIGhhcmQgbnVtYmVycy4NCg0KVHJvbmQNCg==

RE: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[david.noveck]

> I have _always_ argued that callbacks should be the exception and not the rule, 

Sounds right.

> due to the fact that in pretty much all cases the server will have to send 
> out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. 

I may disagree depending on what you mean by "petty much" always.

V4.1 does have callbacks that are specified so as to useful asynchronously, and
if people take advantage of them, they won't have this NFS4ERR_DELAY problem.

Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG.  Over time, future versions may
expand this list and make callbacks less exceptional, although not the rule. 

-----Original Message-----
From: nfsv4-bounces@ietf.org [mailto:nfsv4-bounces@ietf.org] On Behalf Of Myklebust, Trond
Sent: Thursday, October 06, 2011 11:39 PM
To: Matt W. Benjamin
Cc: linux-nfs; nfs-ganesha-devel; nfsv4
Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

> -----Original Message-----
> From: Matt W. Benjamin [mailto:matt@linuxbox.com]
> Sent: Thursday, October 06, 2011 10:55 PM
> To: Myklebust, Trond
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> BIND_CONN_TO_SESSION
> 
> Hi Trond,
> 
> I appreciate your point of view.  I appreciate Rick's comments.  I hope some
> other comments will trickle in.  Do I think you and Rick are saying the same
> thing?  Not really.  Do I think you're making sense when you imply I want to
> change the standard?  No, I don't follow that.  BIND_CONN_TO_SESSION is in
> the published standard.  Can I show a dedicated back channel improves

I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.

> performance?  Not at present.  That's a sensible question.  I think if the
> protocol in general is doing what it is intended to do, it should be possible for
> some workloads, at some point.  (If callbacks are, somehow generally
> detrimental to performance, as you state, I think maybe we have some more
> work to do.)  (I note only one more thing--I did raise this topic on list 12
> months ago.  Only Bruce commented, at the time.)

Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.

I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.

IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.

Trond
_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4

Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[Trond Myklebust]

On Tue, 2011-10-18 at 17:28 -0400, david.noveck@emc.com wrote: 
> > I have _always_ argued that callbacks should be the exception and not the rule, 
> 
> Sounds right.
> 
> > due to the fact that in pretty much all cases the server will have to send 
> > out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. 
> 
> I may disagree depending on what you mean by "petty much" always.
> 
> V4.1 does have callbacks that are specified so as to useful asynchronously, and
> if people take advantage of them, they won't have this NFS4ERR_DELAY problem.
> 
> Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG.  Over time, future versions may
> expand this list and make callbacks less exceptional, although not the rule. 

I'm not sure why you included CB_RECALL_ANY in the above list. Did you
mean CB_RECALLABLE_OBJ_AVAIL?

The main consumer of callbacks in NFSv4.1 is expected by some to be
layouts, which do not really have an asynchronous mode to notify you
when a particular layout has been made available.
Ditto for the NFSv4.0 number 1 consumer: delegations. The new
CB_PUSH_DELEG operation notifies you that you may now request a
delegation, but it doesn't help when you are just trying to OPEN a file,
and the server has to recall a delegation from someone else.

So I still think that NFS4ERR_DELAY will still be needed in order to
slow things down, unless we're careful about getting into these
situations where recalls are needed.

> -----Original Message-----
> From: nfsv4-bounces@ietf.org [mailto:nfsv4-bounces@ietf.org] On Behalf Of Myklebust, Trond
> Sent: Thursday, October 06, 2011 11:39 PM
> To: Matt W. Benjamin
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
> 
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:matt@linuxbox.com]
> > Sent: Thursday, October 06, 2011 10:55 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> > Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> > BIND_CONN_TO_SESSION
> > 
> > Hi Trond,
> > 
> > I appreciate your point of view.  I appreciate Rick's comments.  I hope some
> > other comments will trickle in.  Do I think you and Rick are saying the same
> > thing?  Not really.  Do I think you're making sense when you imply I want to
> > change the standard?  No, I don't follow that.  BIND_CONN_TO_SESSION is in
> > the published standard.  Can I show a dedicated back channel improves
> 
> I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.
> 
> > performance?  Not at present.  That's a sensible question.  I think if the
> > protocol in general is doing what it is intended to do, it should be possible for
> > some workloads, at some point.  (If callbacks are, somehow generally
> > detrimental to performance, as you state, I think maybe we have some more
> > work to do.)  (I note only one more thing--I did raise this topic on list 12
> > months ago.  Only Bruce commented, at the time.)
> 
> Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.
> 
> I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.
> 
> IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.
> 
> Trond
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

RE: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

[david.noveck]

I meant CB_RECALL_ANY.  The server can use this to make sure that it is 
Likely to have an extra recallable object available in hand.  If he indicates
the desire early, than the interaction is asynchronous, and so the first
request requesting the layout/delegation get that one and you start another
asynchronous operation to have another given back.

In some cases, not having space does not man DELAY but rather not getting the
delegation you want, with that being provided to the client by CB_RECALL_OBJ_AVAIL.

This sort of interaction suggests to me that requests and callbacks may wind up
being more connected than we have thought before and that we might address this in
some v4.x for x >= 3.  


-----Original Message-----
From: Trond Myklebust [mailto:Trond.Myklebust@netapp.com] 
Sent: Tuesday, October 18, 2011 6:39 PM
To: Noveck, David
Cc: matt@linuxbox.com; linux-nfs@vger.kernel.org; nfs-ganesha-devel@lists.sourceforge.net; nfsv4@ietf.org
Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

On Tue, 2011-10-18 at 17:28 -0400, david.noveck@emc.com wrote: 
> > I have _always_ argued that callbacks should be the exception and not the rule, 
> 
> Sounds right.
> 
> > due to the fact that in pretty much all cases the server will have to send 
> > out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. 
> 
> I may disagree depending on what you mean by "petty much" always.
> 
> V4.1 does have callbacks that are specified so as to useful asynchronously, and
> if people take advantage of them, they won't have this NFS4ERR_DELAY problem.
> 
> Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG.  Over time, future versions may
> expand this list and make callbacks less exceptional, although not the rule. 

I'm not sure why you included CB_RECALL_ANY in the above list. Did you
mean CB_RECALLABLE_OBJ_AVAIL?

The main consumer of callbacks in NFSv4.1 is expected by some to be
layouts, which do not really have an asynchronous mode to notify you
when a particular layout has been made available.
Ditto for the NFSv4.0 number 1 consumer: delegations. The new
CB_PUSH_DELEG operation notifies you that you may now request a
delegation, but it doesn't help when you are just trying to OPEN a file,
and the server has to recall a delegation from someone else.

So I still think that NFS4ERR_DELAY will still be needed in order to
slow things down, unless we're careful about getting into these
situations where recalls are needed.

> -----Original Message-----
> From: nfsv4-bounces@ietf.org [mailto:nfsv4-bounces@ietf.org] On Behalf Of Myklebust, Trond
> Sent: Thursday, October 06, 2011 11:39 PM
> To: Matt W. Benjamin
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
> 
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:matt@linuxbox.com]
> > Sent: Thursday, October 06, 2011 10:55 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> > Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> > BIND_CONN_TO_SESSION
> > 
> > Hi Trond,
> > 
> > I appreciate your point of view.  I appreciate Rick's comments.  I hope some
> > other comments will trickle in.  Do I think you and Rick are saying the same
> > thing?  Not really.  Do I think you're making sense when you imply I want to
> > change the standard?  No, I don't follow that.  BIND_CONN_TO_SESSION is in
> > the published standard.  Can I show a dedicated back channel improves
> 
> I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.
> 
> > performance?  Not at present.  That's a sensible question.  I think if the
> > protocol in general is doing what it is intended to do, it should be possible for
> > some workloads, at some point.  (If callbacks are, somehow generally
> > detrimental to performance, as you state, I think maybe we have some more
> > work to do.)  (I note only one more thing--I did raise this topic on list 12
> > months ago.  Only Bruce commented, at the time.)
> 
> Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.
> 
> I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.
> 
> IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.
> 
> Trond
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> nfsv4@ietf.org
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com