NFS client behavior on WRITE_DELEGATION

NFS client behavior on WRITE_DELEGATION

[Soumya Koduri]

Hi,

In our NFSv4.1 Delegations testing we observed that NFS client sends 
WRITE requests to the server even when granted WRITE_DELEGATION. I see 
that only OPEN/CLOSE and LOCK operations are served locally by the client.

Please confirm if this behavior is expected and if yes, can it be 
improved to avoid sending WRITEs to the server until the application 
does fsync.

Thanks,
Soumya

Re: NFS client behavior on WRITE_DELEGATION

[Anna Schumaker]

(Resending due to accidentally using HTML instead of plain text)

On Wed, 2018-12-05 at 15:22 -0500, Anna Schumaker wrote:
> Hi Soumya,
> 
> What conditions are you seeing the writes?  It could be the kernel trying to
> flush out data to free up memory for other processes.
> 
> I don't think preemptively writing back data is necessarily a bad thing
> though, even with a write delegation.  If the server decides to recall the
> delegation, we wouldn't keep it waiting as long while we write back local
> data.  Additionally, if the client crashes then any updates might have already
> had a chance to sync to the server.
> 
> I hope this helps!
> Anna
> 
> On Mon, Dec 3, 2018 at 2:55 AM Soumya Koduri <skoduri@redhat.com> wrote:
> > Hi,
> > 
> > In our NFSv4.1 Delegations testing we observed that NFS client sends 
> > WRITE requests to the server even when granted WRITE_DELEGATION. I see 
> > that only OPEN/CLOSE and LOCK operations are served locally by the client.
> > 
> > Please confirm if this behavior is expected and if yes, can it be 
> > improved to avoid sending WRITEs to the server until the application 
> > does fsync.
> > 
> > Thanks,
> > Soumya

Re: NFS client behavior on WRITE_DELEGATION

[Soumya Koduri]

Hi Anna,

Thanks a lot for the response. To understand client behavior under 
various scenarios, I started with a simple "C" program which does below 
operations in a loop on a nfsv4.1 mount point-

while(..)
{
OPEN,
LOCK,
READ/WRITE,
UNLCK,
CLOSE
}

While examining the pkt traces, I observed that even when granted 
WRITE_DELEGATION, there are WRITEs (though asynchronous & followed by 
COMMIT) sent to the server for each of those iterations. So could it be 
because of the kernel flushing out the data?

Also I noticed that the behavior of WRITE_DELEGATION granted for 
OPEN(O_RDWR) is different when compared to the one granted to OPEN(O_WRONLY)

1) When the file is opened with OPEN4_SHARE_ACCESS_WRITE and a 
OPEN_DELEGATE_WRITE delegation is granted, subsequent OPENs are avoided 
and not sent to the server as expected, whereas
If the file is opened with OPEN4_SHARE_ACCESS_BOTH and granted 
OPEN_DELEGATE_WRITE, the next open is sent to server resulting in server 
recalling the delegation it granted to the same client.

2) And to that recall request, client responded back with EBADHANDLE error.

This doesn't seem right. Could you please confirm.

(Maybe the server also can avoid conflicting the OPENfrom the same 
client to which it granted the delegation)

Thanks,
Soumya

On 12/6/18 1:52 AM, Anna Schumaker wrote:
> Hi Soumya,
> 
> What conditions are you seeing the writes?  It could be the kernel 
> trying to flush out data to free up memory for other processes.
> 
> I don't think preemptively writing back data is necessarily a bad thing 
> though, even with a write delegation.  If the server decides to recall 
> the delegation, we wouldn't keep it waiting as long while we write back 
> local data.  Additionally, if the client crashes then any updates might 
> have already had a chance to sync to the server.
> 
> I hope this helps!
> Anna
> 
> On Mon, Dec 3, 2018 at 2:55 AM Soumya Koduri <skoduri@redhat.com 
> <mailto:skoduri@redhat.com>> wrote:
> 
>     Hi,
> 
>     In our NFSv4.1 Delegations testing we observed that NFS client sends
>     WRITE requests to the server even when granted WRITE_DELEGATION. I see
>     that only OPEN/CLOSE and LOCK operations are served locally by the
>     client.
> 
>     Please confirm if this behavior is expected and if yes, can it be
>     improved to avoid sending WRITEs to the server until the application
>     does fsync.
> 
>     Thanks,
>     Soumya
> 

Re: NFS client behavior on WRITE_DELEGATION

[Soumya Koduri]

On 12/6/18 8:40 PM, Soumya Koduri wrote:
> Hi Anna,
> 
> Thanks a lot for the response. To understand client behavior under 
> various scenarios, I started with a simple "C" program which does below 
> operations in a loop on a nfsv4.1 mount point-
> 
> while(..)
> {
> OPEN,
> LOCK,
> READ/WRITE,
> UNLCK,
> CLOSE
> }
> 
> While examining the pkt traces, I observed that even when granted 
> WRITE_DELEGATION, there are WRITEs (though asynchronous & followed by 
> COMMIT) sent to the server for each of those iterations. So could it be 
> because of the kernel flushing out the data?
> 
> Also I noticed that the behavior of WRITE_DELEGATION granted for 
> OPEN(O_RDWR) is different when compared to the one granted to 
> OPEN(O_WRONLY)
> 
> 1) When the file is opened with OPEN4_SHARE_ACCESS_WRITE and a 
> OPEN_DELEGATE_WRITE delegation is granted, subsequent OPENs are avoided 
> and not sent to the server as expected, whereas
> If the file is opened with OPEN4_SHARE_ACCESS_BOTH and granted 
> OPEN_DELEGATE_WRITE, the next open is sent to server resulting in server 
> recalling the delegation it granted to the same client.
> 
> 2) And to that recall request, client responded back with EBADHANDLE error.
> 
> This doesn't seem right. Could you please confirm.
> 
> (Maybe the server also can avoid conflicting the OPENfrom the same 
> client to which it granted the delegation)

Apologies. This seem to have resulted from a server-side issue. On 
further debugging, I found that server rejected READ op with 
ERR_BAD_STATEID if there is WRITE_delegation granted to that file. This 
may have led nfs client to discard the delegation granted and send 
subsequent OPEN to server.

I think the server should allow READs if the access is 
OPEN4_SHARE_ACCESS_BOTH and its the same client which holds the write 
delegation. Will send out a patch to fix it.

So the only question I have for now is if/how client can save WRITEs 
from being sent to the server while holding write delegation. Please 
share your thoughts.

Thanks,
Soumya


> 
> Thanks,
> Soumya
> 
> On 12/6/18 1:52 AM, Anna Schumaker wrote:
>> Hi Soumya,
>>
>> What conditions are you seeing the writes?  It could be the kernel 
>> trying to flush out data to free up memory for other processes.
>>
>> I don't think preemptively writing back data is necessarily a bad 
>> thing though, even with a write delegation.  If the server decides to 
>> recall the delegation, we wouldn't keep it waiting as long while we 
>> write back local data.  Additionally, if the client crashes then any 
>> updates might have already had a chance to sync to the server.
>>
>> I hope this helps!
>> Anna
>>
>> On Mon, Dec 3, 2018 at 2:55 AM Soumya Koduri <skoduri@redhat.com 
>> <mailto:skoduri@redhat.com>> wrote:
>>
>>     Hi,
>>
>>     In our NFSv4.1 Delegations testing we observed that NFS client sends
>>     WRITE requests to the server even when granted WRITE_DELEGATION. I 
>> see
>>     that only OPEN/CLOSE and LOCK operations are served locally by the
>>     client.
>>
>>     Please confirm if this behavior is expected and if yes, can it be
>>     improved to avoid sending WRITEs to the server until the application
>>     does fsync.
>>
>>     Thanks,
>>     Soumya
>>