Seemingly inconsistent directory state under NFSv4

Seemingly inconsistent directory state under NFSv4

[Ferenc Wagner]

Hi,

In our somewhat mixed NFSv4 setup (Linux 2.6.32 Debian squeeze server, a
2.6.26 and a 2.6.32 Debian lenny client) we sometimes experience the
following: removing a file on the 2.6.26 client makes that file
disappear from the directory listing of the 2.6.32 client, but the file
itself remains accessible by its old -- now nonexistent -- name there.
Is such a confusing state expected, or is this the manifestation of some
bug?
-- 
Thanks,
Feri.

Re: Seemingly inconsistent directory state under NFSv4

[Ferenc Wagner]

Ferenc Wagner <wferi@niif.hu> writes:

> In our somewhat mixed NFSv4 setup (Linux 2.6.32 Debian squeeze server, a
> 2.6.26 and a 2.6.32 Debian lenny client) we sometimes experience the
> following: removing a file on the 2.6.26 client makes that file
> disappear from the directory listing of the 2.6.32 client, but the file
> itself remains accessible by its old -- now nonexistent -- name there.
> Is such a confusing state expected, or is this the manifestation of some
> bug?

Is this perhaps a manifestation of the following paragraph of
nfs41-server.txt in the Linux kernel documentation?

    Incomplete delegation enforcement: if a file is renamed or
    unlinked, a client holding a delegation may continue to
    indefinitely allow opens of the file under the old name.

If yes, is there a way to list the active delegations (on the server and
on the client) and possibly trigger releasing/recalling them?
-- 
Thanks,
Feri.

Re: Seemingly inconsistent directory state under NFSv4

[J. Bruce Fields]

On Tue, Feb 22, 2011 at 05:42:44PM +0100, Ferenc Wagner wrote:
> Ferenc Wagner <wferi@niif.hu> writes:
> 
> > In our somewhat mixed NFSv4 setup (Linux 2.6.32 Debian squeeze server, a
> > 2.6.26 and a 2.6.32 Debian lenny client) we sometimes experience the
> > following: removing a file on the 2.6.26 client makes that file
> > disappear from the directory listing of the 2.6.32 client, but the file
> > itself remains accessible by its old -- now nonexistent -- name there.
> > Is such a confusing state expected, or is this the manifestation of some
> > bug?
> 
> Is this perhaps a manifestation of the following paragraph of
> nfs41-server.txt in the Linux kernel documentation?
> 
>     Incomplete delegation enforcement: if a file is renamed or
>     unlinked, a client holding a delegation may continue to
>     indefinitely allow opens of the file under the old name.
> 
> If yes, is there a way to list the active delegations (on the server and
> on the client) and possibly trigger releasing/recalling them?

The linux client will return any delegation before it does a rename or
unlink, so if you're just doing everything from one client, I don't
*think* the problem is with delegations--you should hit this delegation
problem only in the case where you are removing the file on one client
while using it on another, or removing on the server while using it on
the client.

(In the case of client-client conflicts, this is fixed in 2.6.38.)

You can turn off delegations completely with

	echo 0 >/proc/sys/fs/leases-enable

before starting the nfs server.

--b.

Re: Seemingly inconsistent directory state under NFSv4

[Ferenc Wagner]

"J. Bruce Fields" <bfields@fieldses.org> writes:

> On Tue, Feb 22, 2011 at 05:42:44PM +0100, Ferenc Wagner wrote:
>> Ferenc Wagner <wferi@niif.hu> writes:
>> 
>>> In our somewhat mixed NFSv4 setup (Linux 2.6.32 Debian squeeze server, a
>>> 2.6.26 and a 2.6.32 Debian lenny client) we sometimes experience the
>>> following: removing a file on the 2.6.26 client makes that file
>>> disappear from the directory listing of the 2.6.32 client, but the file
>>> itself remains accessible by its old -- now nonexistent -- name there.
>>> Is such a confusing state expected, or is this the manifestation of some
>>> bug?
>> 
>> Is this perhaps a manifestation of the following paragraph of
>> nfs41-server.txt in the Linux kernel documentation?
>> 
>>     Incomplete delegation enforcement: if a file is renamed or
>>     unlinked, a client holding a delegation may continue to
>>     indefinitely allow opens of the file under the old name.
>> 
>> If yes, is there a way to list the active delegations (on the server and
>> on the client) and possibly trigger releasing/recalling them?
>
> The linux client will return any delegation before it does a rename or
> unlink, so if you're just doing everything from one client, I don't
> *think* the problem is with delegations--you should hit this delegation
> problem only in the case where you are removing the file on one client
> while using it on another, or removing on the server while using it on
> the client.
>
> (In the case of client-client conflicts, this is fixed in 2.6.38.)

One client removed the file, and another could still access it by name
(although not present in the directory listing).  So it could have been
a client-client conflict, even though we couldn't prove that the removed
file was actually in use on the client.  Is there a way to list the
delegations being hold by a client?

> You can turn off delegations completely with
> 	echo 0 >/proc/sys/fs/leases-enable
> before starting the nfs server.

Wouldn't I lose most of the efficiency advantages of NFSv4 with that
move?
-- 
Thanks,
Feri.

Re: Seemingly inconsistent directory state under NFSv4

[J. Bruce Fields]

By the way, could you configure your mail client not to drop me (or
others) off the cc:?  Those of us that subscribe to a lot of high-volume
kernel lists generally treat mail sent to us diferently from mail sent
to a list.

On Thu, Feb 24, 2011 at 05:48:58PM +0100, Ferenc Wagner wrote:
> "J. Bruce Fields" <bfields@fieldses.org> writes:
> One client removed the file, and another could still access it by name
> (although not present in the directory listing).  So it could have been
> a client-client conflict, even though we couldn't prove that the removed
> file was actually in use on the client.  Is there a way to list the
> delegations being hold by a client?

Not really.  They'll show up as LEASE entries in /proc/locks, with the
pid of an nfsd process, but no way to associate them with a client.

Some way to dump lock (and other state) information might be a nice
thing to have some day for debugging and tuning.

> > You can turn off delegations completely with
> > 	echo 0 >/proc/sys/fs/leases-enable
> > before starting the nfs server.
> 
> Wouldn't I lose most of the efficiency advantages of NFSv4 with that
> move?

Probably so.  It would at least be a way to verify whether delegations
are the source of your problem, if you have a reproduceable test case.

--b.

Re: Seemingly inconsistent directory state under NFSv4

[Ferenc Wagner]

"J. Bruce Fields" <bfields@fieldses.org> writes:

> On Thu, Feb 24, 2011 at 05:48:58PM +0100, Ferenc Wagner wrote:
>
>> "J. Bruce Fields" <bfields@fieldses.org> writes:
>> One client removed the file, and another could still access it by name
>> (although not present in the directory listing).  So it could have been
>> a client-client conflict, even though we couldn't prove that the removed
>> file was actually in use on the client.  Is there a way to list the
>> delegations being hold by a client?
>
> Not really.  They'll show up as LEASE entries in /proc/locks, with the
> pid of an nfsd process, but no way to associate them with a client.

Well, thanks, this is something.  I'll check when it happens again.
There are lots of such lines, so chances are that will be it.

> Some way to dump lock (and other state) information might be a nice
> thing to have some day for debugging and tuning.

Yes, very much!  I guess the info is more or less readily available in
the kernel, just isn't exported to userspace.  Looks like a job for
debugfs.

>>> You can turn off delegations completely with
>>> 	echo 0 >/proc/sys/fs/leases-enable
>>> before starting the nfs server.
>> 
>> Wouldn't I lose most of the efficiency advantages of NFSv4 with that
>> move?
>
> Probably so.  It would at least be a way to verify whether delegations
> are the source of your problem, if you have a reproduceable test case.

Not yet, unfortunately.  But we're trying to work one out.  It's pretty
rare, so only mildly disturbing, but when it hits, it hits hard...
-- 
Thanks,
Feri.