Confused by pnfs LAYOUTRETURN - seeking clarity.

Confused by pnfs LAYOUTRETURN - seeking clarity.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2334 bytes --]


I've been trying to understand how LAYOUTRETURN is used in pNFS,
primarily because our SLE12-SP1 kernel (based on 3.12) appears
to have a very different opinion than some Netapp filers.

My reading of RFC-5661 suggests that the client needs to call
LAYOUTRETURN for every layout that it received from the server.  A
single LAYOUTRETURN can cover a whole file or a whole filesystem, so it
doesn't need to be 1-for-1, but there is no implicit return.

However RFC-5663 contains the text

   A LAYOUTRETURN operation represents an explicit release of resources
   by the client, usually done for the purpose of avoiding unnecessary
   CB_LAYOUTRECALL operations in the future.

This seems to imply that LAYOUTRETURN is only an optimisation.  If you
don't want to avoid CB_LAYOUTRECALL, there is not much call for
LAYOUTRETURN.  It seems to suggest (without explicitly saying) that the
CB_LAYOUTRECALL will effect the return of a layout without the client
explicitly sending LAYOUTRETURN in response.  RFC-5661 says LAYOUTRETURN
does need to be sent in response.

The code in 3.12 doesn't send LAYOUTRETURN in response to
CB_LAYOUTRECALL, nor does it send LAYOUTRETURN when it closes a file
marked as "return layouts on close".  The one place I have seen evidence
of it returning layouts is when a file is unlinked, though I think there
are others (chmod, IO error).

The current upstream code seems to call LAYOUTRETURN more correctly, but
it is hard to be sure because I couldn't find a commit which acknowledged
the specific problem and corrected it - just commits that claim to be
making improvements and avoiding races and things like that.

Questions:
 - Am I correct that all layouts need to be explicitly returned by the
   client, and so the text from RFC-5663 is misleading?

 - If so, what is the earliest kernel that is believed to correctly
   return layouts in response to CB_LAYOUTRECALL, or a 'roc' file being
   closed?


I was advised that Netapp are considering a change (netapp issue
955835):
  An enhancement will be added in future versions of Ontap to clear out
  the corresponding layout states after a file has been closed in the
  event the client does not return them.

This sounds like a mistake, unless "clear out" means "send
CB_LAYOUTRECALL for".  Should we advice Netapp against this?

Thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Confused by pnfs LAYOUTRETURN - seeking clarity.

[Christoph Hellwig]

Hi Neil,

sorry for the sparse reply, I'm travelling right now,

take a look at the NFS4ERR_NOMATCHING_LAYOUT error in NFSv4.2
and maybe try to do an internet search for forgetful clients - that's
the way how the Linux pNFS client operates using the above error
to signal it doesn't have any outstanding layouts when responding
to a layoutrecall.

Re: Confused by pnfs LAYOUTRETURN - seeking clarity.

[Jeff Layton]

On Fri, 2017-03-03 at 08:24 +1100, NeilBrown wrote:
> I've been trying to understand how LAYOUTRETURN is used in pNFS,
> primarily because our SLE12-SP1 kernel (based on 3.12) appears
> to have a very different opinion than some Netapp filers.
> 
> My reading of RFC-5661 suggests that the client needs to call
> LAYOUTRETURN for every layout that it received from the server.  A
> single LAYOUTRETURN can cover a whole file or a whole filesystem, so it
> doesn't need to be 1-for-1, but there is no implicit return.
> 
> However RFC-5663 contains the text
> 
>    A LAYOUTRETURN operation represents an explicit release of resources
>    by the client, usually done for the purpose of avoiding unnecessary
>    CB_LAYOUTRECALL operations in the future.
> 
> This seems to imply that LAYOUTRETURN is only an optimisation.  If you
> don't want to avoid CB_LAYOUTRECALL, there is not much call for
> LAYOUTRETURN.  It seems to suggest (without explicitly saying) that the
> CB_LAYOUTRECALL will effect the return of a layout without the client
> explicitly sending LAYOUTRETURN in response.  RFC-5661 says LAYOUTRETURN
> does need to be sent in response.
> 
> The code in 3.12 doesn't send LAYOUTRETURN in response to
> CB_LAYOUTRECALL, nor does it send LAYOUTRETURN when it closes a file
> marked as "return layouts on close".  The one place I have seen evidence
> of it returning layouts is when a file is unlinked, though I think there
> are others (chmod, IO error).
> 
> The current upstream code seems to call LAYOUTRETURN more correctly, but
> it is hard to be sure because I couldn't find a commit which acknowledged
> the specific problem and corrected it - just commits that claim to be
> making improvements and avoiding races and things like that.
> 
> Questions:
>  - Am I correct that all layouts need to be explicitly returned by the
>    client, and so the text from RFC-5663 is misleading?
> 

There is one special case...

When the server does a CB_LAYOUTRECALL, the client can reply with
NFS4ERR_NOMATCHING_LAYOUT if it's not actively using the layout at the
time. With that, the client isn't expected to do a LAYOUTRETURN.

Any other status on the CB_LAYOUTRECALL however does require a
LAYOUTRETURN.

>  - If so, what is the earliest kernel that is believed to correctly
>    return layouts in response to CB_LAYOUTRECALL, or a 'roc' file being
>    closed?
> 
> I was advised that Netapp are considering a change (netapp issue
> 955835):
>   An enhancement will be added in future versions of Ontap to clear out
>   the corresponding layout states after a file has been closed in the
>   event the client does not return them.
> 
> This sounds like a mistake, unless "clear out" means "send
> CB_LAYOUTRECALL for".  Should we advice Netapp against this?
> 
> Thanks,
> NeilBrown
> 

Re: Confused by pnfs LAYOUTRETURN - seeking clarity.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On Fri, Mar 03 2017, Jeff Layton wrote:

>
> There is one special case...
>
> When the server does a CB_LAYOUTRECALL, the client can reply with
> NFS4ERR_NOMATCHING_LAYOUT if it's not actively using the layout at the
> time. With that, the client isn't expected to do a LAYOUTRETURN.
>
> Any other status on the CB_LAYOUTRECALL however does require a
> LAYOUTRETURN.

Thanks, both to you and Christoph, for this pointer.
Things are starting to make sense.

However RFC5661 says under
   Operation 5: CB_LAYOUTRECALL - Recall Layout from Client

   While the client responds to the CB_LAYOUTRECALL immediately, the
   operation is not considered complete (i.e., considered pending) until
   all affected layouts are returned to the server via the LAYOUTRETURN
   operation.

which seems to strongly imply that a LAYOUTRETURN is expected.  No
alternative is listed.

Further:

    The NFS4ERR_NOMATCHING_LAYOUT error is only returned when the client
    does not hold layouts for the file or if the client does not have
    any overlapping layouts for the specification in the layout recall.

This is an *error* condition.   The client really does still hold
the layouts if it hasn't sent "LAYOUTRETURN".  The fact that the linux
client chooses to forget them and won't ever use them again isn't really
the same thing as "not hold[ing] layouts".

If I were a server author and I got NFS4ERR_NOMATCHING_LAYOUT for a
CB_LAYOUTRECALL where a I knew for certain that the client really did
have the layout, then I would probably feel justified in discarding all
state held by that confused client, and requiring to re-establish
everything (because I cannot trust anything any more).

So the Netapp filer is clearly doing the wrong thing, as we doesn't send
CB_LAYOUTRECALL.  But I'm far from convinced that Linux is doing the
right thing by replying NFS4ERR_NOMATCHING_LAYOUT.

However, I'm not going to try to "fix" anything here.  Hopefully we can
manage to stumble forward.

Thanks again,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Confused by pnfs LAYOUTRETURN - seeking clarity.

[Trond Myklebust]

SGkgTmVpbCwNCg0KT24gTW9uLCAyMDE3LTAzLTA2IGF0IDE0OjU0ICsxMTAwLCBOZWlsQnJvd24g
d3JvdGU6DQo+IE9uIEZyaSwgTWFyIDAzIDIwMTcsIEplZmYgTGF5dG9uIHdyb3RlOg0KPiANCj4g
PiANCj4gPiBUaGVyZSBpcyBvbmUgc3BlY2lhbCBjYXNlLi4uDQo+ID4gDQo+ID4gV2hlbiB0aGUg
c2VydmVyIGRvZXMgYSBDQl9MQVlPVVRSRUNBTEwsIHRoZSBjbGllbnQgY2FuIHJlcGx5IHdpdGgN
Cj4gPiBORlM0RVJSX05PTUFUQ0hJTkdfTEFZT1VUIGlmIGl0J3Mgbm90IGFjdGl2ZWx5IHVzaW5n
IHRoZSBsYXlvdXQgYXQNCj4gPiB0aGUNCj4gPiB0aW1lLiBXaXRoIHRoYXQsIHRoZSBjbGllbnQg
aXNuJ3QgZXhwZWN0ZWQgdG8gZG8gYSBMQVlPVVRSRVRVUk4uDQo+ID4gDQo+ID4gQW55IG90aGVy
IHN0YXR1cyBvbiB0aGUgQ0JfTEFZT1VUUkVDQUxMIGhvd2V2ZXIgZG9lcyByZXF1aXJlIGENCj4g
PiBMQVlPVVRSRVRVUk4uDQo+IA0KPiBUaGFua3MsIGJvdGggdG8geW91IGFuZCBDaHJpc3RvcGgs
IGZvciB0aGlzIHBvaW50ZXIuDQo+IFRoaW5ncyBhcmUgc3RhcnRpbmcgdG8gbWFrZSBzZW5zZS4N
Cj4gDQo+IEhvd2V2ZXIgUkZDNTY2MSBzYXlzIHVuZGVyDQo+IMKgwqDCoE9wZXJhdGlvbiA1OiBD
Ql9MQVlPVVRSRUNBTEwgLSBSZWNhbGwgTGF5b3V0IGZyb20gQ2xpZW50DQo+IA0KPiDCoMKgwqBX
aGlsZSB0aGUgY2xpZW50IHJlc3BvbmRzIHRvIHRoZSBDQl9MQVlPVVRSRUNBTEwgaW1tZWRpYXRl
bHksIHRoZQ0KPiDCoMKgwqBvcGVyYXRpb24gaXMgbm90IGNvbnNpZGVyZWQgY29tcGxldGUgKGku
ZS4sIGNvbnNpZGVyZWQgcGVuZGluZykNCj4gdW50aWwNCj4gwqDCoMKgYWxsIGFmZmVjdGVkIGxh
eW91dHMgYXJlIHJldHVybmVkIHRvIHRoZSBzZXJ2ZXIgdmlhIHRoZQ0KPiBMQVlPVVRSRVRVUk4N
Cj4gwqDCoMKgb3BlcmF0aW9uLg0KPiANCj4gd2hpY2ggc2VlbXMgdG8gc3Ryb25nbHkgaW1wbHkg
dGhhdCBhIExBWU9VVFJFVFVSTiBpcyBleHBlY3RlZC7CoMKgTm8NCj4gYWx0ZXJuYXRpdmUgaXMg
bGlzdGVkLg0KPiANCj4gRnVydGhlcjoNCj4gDQo+IMKgwqDCoMKgVGhlIE5GUzRFUlJfTk9NQVRD
SElOR19MQVlPVVQgZXJyb3IgaXMgb25seSByZXR1cm5lZCB3aGVuIHRoZQ0KPiBjbGllbnQNCj4g
wqDCoMKgwqBkb2VzIG5vdCBob2xkIGxheW91dHMgZm9yIHRoZSBmaWxlIG9yIGlmIHRoZSBjbGll
bnQgZG9lcyBub3QgaGF2ZQ0KPiDCoMKgwqDCoGFueSBvdmVybGFwcGluZyBsYXlvdXRzIGZvciB0
aGUgc3BlY2lmaWNhdGlvbiBpbiB0aGUgbGF5b3V0DQo+IHJlY2FsbC4NCj4gDQo+IFRoaXMgaXMg
YW4gKmVycm9yKiBjb25kaXRpb24uwqDCoMKgVGhlIGNsaWVudCByZWFsbHkgZG9lcyBzdGlsbCBo
b2xkDQo+IHRoZSBsYXlvdXRzIGlmIGl0IGhhc24ndCBzZW50ICJMQVlPVVRSRVRVUk4iLsKgwqBU
aGUgZmFjdCB0aGF0IHRoZQ0KPiBsaW51eA0KPiBjbGllbnQgY2hvb3NlcyB0byBmb3JnZXQgdGhl
bSBhbmQgd29uJ3QgZXZlciB1c2UgdGhlbSBhZ2FpbiBpc24ndA0KPiByZWFsbHkNCj4gdGhlIHNh
bWUgdGhpbmcgYXMgIm5vdCBob2xkW2luZ10gbGF5b3V0cyIuDQo+IA0KPiBJZiBJIHdlcmUgYSBz
ZXJ2ZXIgYXV0aG9yIGFuZCBJIGdvdCBORlM0RVJSX05PTUFUQ0hJTkdfTEFZT1VUIGZvciBhDQo+
IENCX0xBWU9VVFJFQ0FMTCB3aGVyZSBhIEkga25ldyBmb3IgY2VydGFpbiB0aGF0IHRoZSBjbGll
bnQgcmVhbGx5IGRpZA0KPiBoYXZlIHRoZSBsYXlvdXQsIHRoZW4gSSB3b3VsZCBwcm9iYWJseSBm
ZWVsIGp1c3RpZmllZCBpbiBkaXNjYXJkaW5nDQo+IGFsbA0KPiBzdGF0ZSBoZWxkIGJ5IHRoYXQg
Y29uZnVzZWQgY2xpZW50LCBhbmQgcmVxdWlyaW5nIHRvIHJlLWVzdGFibGlzaA0KPiBldmVyeXRo
aW5nIChiZWNhdXNlIEkgY2Fubm90IHRydXN0IGFueXRoaW5nIGFueSBtb3JlKS4NCj4gDQo+IFNv
IHRoZSBOZXRhcHAgZmlsZXIgaXMgY2xlYXJseSBkb2luZyB0aGUgd3JvbmcgdGhpbmcsIGFzIHdl
IGRvZXNuJ3QNCj4gc2VuZA0KPiBDQl9MQVlPVVRSRUNBTEwuwqDCoEJ1dCBJJ20gZmFyIGZyb20g
Y29udmluY2VkIHRoYXQgTGludXggaXMgZG9pbmcgdGhlDQo+IHJpZ2h0IHRoaW5nIGJ5IHJlcGx5
aW5nIE5GUzRFUlJfTk9NQVRDSElOR19MQVlPVVQuDQo+IA0KPiBIb3dldmVyLCBJJ20gbm90IGdv
aW5nIHRvIHRyeSB0byAiZml4IiBhbnl0aGluZyBoZXJlLsKgwqBIb3BlZnVsbHkgd2UNCj4gY2Fu
DQo+IG1hbmFnZSB0byBzdHVtYmxlIGZvcndhcmQuDQo+IA0KDQpQbGVhc2Ugc2VlIFJGQzU2NjEs
IFNlY3Rpb24gMTIuNS41LjEgaHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzU2Ng0KMSNz
ZWN0aW9uLTEyLjUuNS4xDQoNClRoYXQgc2VjdGlvbiBoYXMgYSBmdWxsIGRvY3VtZW50YXRpb24g
b2Ygd2hhdCB0aGUgc2VydmVyIHNob3VsZA0KaW50ZXJwcmV0IE5GUzRFUlJfTk9NQVRDSElOR19M
QVlPVVQgdG8gbWVhbiwgYW5kIHNob3VsZCBleHBsYWluIHdoeSB0aGUNCkxpbnV4IGNsaWVudCBi
ZWhhdmlvdXIgaXMgcXVpdGUgY29ycmVjdCB3LnIudC4gdGhlIHNwZWMuDQpUaGVyZSBhcmUgYSBj
b3VwbGUgb2YgZXJyYXRhIHRvIGNvbnNpZGVyIHRvbywgYnV0IHRoZXkgbWFpbmx5IHRyeSB0bw0K
bmFpbCBkb3duIHRoZSBleHBlY3RlZCBiZWhhdmlvdXIgZm9yIGEgY291cGxlIG9mIGNvcm5lciBj
YXNlcy4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5l
ciwgUHJpbWFyeURhdGENCnRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5jb20NCg==

Re: Confused by pnfs LAYOUTRETURN - seeking clarity.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Tue, Mar 07 2017, Trond Myklebust wrote:

> Hi Neil,
>
> On Mon, 2017-03-06 at 14:54 +1100, NeilBrown wrote:
...
>> 
>> So the Netapp filer is clearly doing the wrong thing, as we doesn't
>> send
>> CB_LAYOUTRECALL.  But I'm far from convinced that Linux is doing the
>> right thing by replying NFS4ERR_NOMATCHING_LAYOUT.
>> 
>> However, I'm not going to try to "fix" anything here.  Hopefully we
>> can
>> manage to stumble forward.
>> 
>
> Please see RFC5661, Section 12.5.5.1 https://tools.ietf.org/html/rfc566
> 1#section-12.5.5.1
>
> That section has a full documentation of what the server should
> interpret NFS4ERR_NOMATCHING_LAYOUT to mean, and should explain why the
> Linux client behaviour is quite correct w.r.t. the spec.
> There are a couple of errata to consider too, but they mainly try to
> nail down the expected behaviour for a couple of corner cases.

Thanks for that.  The last too sentences to suggest that
NFS4ERR_NOMATCHING_LAYOUT isn't really an error, and the whole
discusses how the server and client can reasonably have different
understandings of the current set of layouts.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]