[BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Yuri]

NFS sometimes leaves files like .nfsXXXXXXXXXXX which usually means that 
some process that has this file open is alive.


"git clean -df ." was supposed to remove the folder where such file is 
located, but it encountered the failure, and silently ignored it and 
succeeded.


Desired behavior: "git clean -df ." should notify the user with warnings 
or errors for each such file that it failed to remove.


When "git clean -df ." succeeded I ran the next command and it failed 
because of the still-present folder with the .nfsXXXXXXXXX file in it.


I believe that this is a bug in git.


git-2.38.1



Thanks,

Yuri

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Junio C Hamano]

Yuri <yuri@rawbw.com> writes:

> NFS sometimes leaves files like .nfsXXXXXXXXXXX which usually means
> that some process that has this file open is alive.

Yes, and from everybody's point of view, including "git", a
directory with such a file is not yet empty.

> "git clean -df ." was supposed to remove the folder where such file is
> located, but it encountered the failure, and silently ignored it and
> succeeded.

So "was supposed to remove" above is not quite correct.  Where did
such a piece of misinformation come from?

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Yuri]

On 6/10/24 12:58, Junio C Hamano wrote:
> So "was supposed to remove" above is not quite correct. Where did such 
> a piece of misinformation come from?


"git clean -df ." removes all files that

(1) aren't added to the repo

(2) aren't added as exceptions

(3) aren't repo files themselves.




Is the above definition incorrect?






Yuri

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Junio C Hamano]

Yuri <yuri@rawbw.com> writes:

> On 6/10/24 12:58, Junio C Hamano wrote:
>> So "was supposed to remove" above is not quite correct. Where did
>> such a piece of misinformation come from?
>
>
> "git clean -df ." removes all files that
>
> (1) aren't added to the repo
>
> (2) aren't added as exceptions
>
> (3) aren't repo files themselves.

But .nfs* files are not something you as an application are not
supposed to touch, so a directory that still contains one cannot be
removed, either.

It's a limitation (I wouldn't call it a "bug") of NFS.  You can kill
the process (or wait until they exit) holding the file open and then
run "clean -df" again, perhaps.

A few Google searches tell us more, e.g.

 - https://nfs.sourceforge.net/#faq_d2
 - https://kb.netapp.com/on-prem/ontap/da/NAS/NAS-KBs/What_are_nfsXXXX_files_and_how_do_I_delete_them

They tell us what these files, which are the result of "silly
rename", are, why they exist, and that they will appear again even
if we remove them.  So we don't remove them, which means the
directory that contains them will not become empty, which in turn
means we do not remove them.

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Yuri]

On 6/10/24 14:37, Junio C Hamano wrote:
> But .nfs* files are not something you as an application are not 
> supposed to touch, so a directory that still contains one cannot be 
> removed, either. It's a limitation (I wouldn't call it a "bug") of 
> NFS. You can kill the process (or wait until they exit) holding the 
> file open and then run "clean -df" again, perhaps.


With the '-f' the user tells git to remove all, and if this doesn't work 
git should tell the user that this didn't work for the .nfsNNNNNNN file 
and for the directory as well.


Why is git quiet about leaving the files. It should complain.

Or maybe there should be a verbosity option, like -v 10, that would make 
git complain about such things.







Thanks,

Yuri

RE: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[rsbecker]

On Monday, June 10, 2024 7:28 PM, Yuri wrote:
>On 6/10/24 14:37, Junio C Hamano wrote:
>> But .nfs* files are not something you as an application are not
>> supposed to touch, so a directory that still contains one cannot be
>> removed, either. It's a limitation (I wouldn't call it a "bug") of
>> NFS. You can kill the process (or wait until they exit) holding the
>> file open and then run "clean -df" again, perhaps.
>
>
>With the '-f' the user tells git to remove all, and if this doesn't work git should tell
>the user that this didn't work for the .nfsNNNNNNN file and for the directory as
>well.
>
>
>Why is git quiet about leaving the files. It should complain.
>
>Or maybe there should be a verbosity option, like -v 10, that would make git
>complain about such things.

I have tried to reproduce your situation using git 2.43.0 without success.

$ mkdir test
$ cd test
$ touch .nfs12309
$ git clean -df .
Removing .nfs12309

I have tried this with and without existing commits and files, but outside of an NFS context. Do you have a reproducible set of commands that I can try? What is in your .gitignore file? I saw a very old NFS situation where . prefix files did not get reported on some operating systems - I do not think this is what is happening, however. What do the commands ls -a and ls and find . -exec {} ";" report at the root of your repository? In NFSv4.1, there is a known situation where .nfsXXXXXX and .smbXXXXXX files are retained by NFS until the server is notified (or NFS determines) that the files have been closed by all clients. These files (with those names) may be automatically created by NFS (at its whim) and are managed by that subsystem - As I understand it, NFS manages removal of those files independently of the client or programs running on the client, so git may think the file is actually removed but the file may not actually be removed because the unlink() can be deferred by NFS. My suspicion is that this NFS itself might be contributing to the situation.

Can you try creating your repository, then restarting your server and client to isolate the "is open" tests that NFS could be doing, and see whether git clean continues to experience this?

--Randall

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Junio C Hamano]

<rsbecker@nexbridge.com> writes:

> I have tried to reproduce your situation using git 2.43.0 without success.
>
> $ mkdir test
> $ cd test
> $ touch .nfs12309
> $ git clean -df .
> Removing .nfs12309

I suspec that this is different from a real NFSv3 client in that
removal of such "removed while still open" files would result in
another one automatically resurrected by the filesystem.

In any case, if we cannot remove a file (due to filesystem
limitation), we should report the fact, just like in a case
where we cannot remove a regular file, e.g.

    $ cd git.git/
    $ mkdir -p junk/ttt
    $ >junk/ttt/sss
    $ chmod a-w junk/ttt
    $ cd junk
    $ git clean -f -d -x ttt; echo $?
    warning: failed to remove ttt/sss: Permission denied
    1

Figuring out why it is not happening is left as an exercise to
readers ;-), as I no longer have an NFSv3 environment handy.

Thanks.

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

['Yuri']

On 6/10/24 16:55, rsbecker@nexbridge.com wrote:
> I have tried to reproduce your situation using git 2.43.0 without 
> success. $ mkdir test $ cd test $ touch .nfs12309 $ git clean -df . 
> Removing .nfs12309 


"touch .nfs12309" isn't enough.


Here is a reliable way to reproduce the problem:
1. Have a git repository on an NFS disk.
2. mkdir xx
3. touch xx/x
4. tail -f xx/x &
5. rm xx/x
6. git clean -df .




The last operation reproduces the problem. The xx directory and the 
.nfsNNNN file in it stay without warnings.
The .nfsNNNN file is created by the NFS client when the xx/x file is 
removed.


Anybody with an NFS disk should be able to reproduce it.

FYI: Git generally warns about files that it can't remove because of 
permissions and special flags reasons.
But Git doesn't warn users about this situation with an NFS directory 
lock file.


Thanks,
Yuri

RE: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[rsbecker]

On Monday, June 10, 2024 9:10 PM, Yuri wrote:
>On 6/10/24 16:55, rsbecker@nexbridge.com wrote:
>> I have tried to reproduce your situation using git 2.43.0 without
>> success. $ mkdir test $ cd test $ touch .nfs12309 $ git clean -df .
>> Removing .nfs12309
>
>
>"touch .nfs12309" isn't enough.
>
>
>Here is a reliable way to reproduce the problem:
>1. Have a git repository on an NFS disk.
>2. mkdir xx
>3. touch xx/x
>4. tail -f xx/x &
>5. rm xx/x
>6. git clean -df .
>
>
>
>
>The last operation reproduces the problem. The xx directory and the .nfsNNNN file
>in it stay without warnings.
>The .nfsNNNN file is created by the NFS client when the xx/x file is removed.
>
>
>Anybody with an NFS disk should be able to reproduce it.
>
>FYI: Git generally warns about files that it can't remove because of permissions and
>special flags reasons.
>But Git doesn't warn users about this situation with an NFS directory lock file.

That is what I suspected. I am suspecting that git does not see the .nfsNNNN file when it is performing the clean. I think NFS creates the file after git does the scan, so as far as git is concerned, there is no .nfsNNNN file until after the operation completes. NFS puts the file there independent of git, so git does not even know about it. Does a second git clean -df . remove the .nfsNNNN file and put a new one, with a different name, in place?

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

['Yuri']

On 6/10/24 18:19, rsbecker@nexbridge.com wrote:
> That is what I suspected. I am suspecting that git does not see the 
> .nfsNNNN file when it is performing the clean. I think NFS creates the 
> file after git does the scan, so as far as git is concerned, there is 
> no .nfsNNNN file until after the operation completes. NFS puts the 
> file there independent of git, so git does not even know about it. 
> Does a second git clean -df . remove the .nfsNNNN file and put a new 
> one, with a different name, in place?


No, *only* the .nfsXXXX file exists in the xx directory when git runs.




Yuri

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Chris Torek]

On Mon, Jun 10, 2024 at 6:22 PM Yuri <yuri@rawbw.com> wrote:
> No, *only* the .nfsXXXX file exists in the xx directory when git runs.

In the case you showed, yes. However, there are *at least* two
*different* problem cases, one of which gives Git no warning that
something weird is going on.

The way "NFS silly renames" work is this:

 * an NFS client tells an NFS server to do operations, and this
   is (nominally) stateless;

 * if an NFS client tells the server to remove a file, the server
   attempts to remove the file;

 * but for POSIX-style operation, if a client knows that some file
   is *open* and that *same client* intends to remove the file,
   the client *must not* send a removal request.  Instead, the
   client sends a "rename" request, renaming the original file
   to ".nfs<unique-id>".  When the client's last user of the file
   closes its last file descriptor, the client *then* sends the
   final remove for the renamed file.

So, the scenarios we must consider are these:

 1. Nobody has the file open anywhere.  The client will send a
    removal request and the server will obey or not depending on
    permissions.

 2. The server itself has the file open, but no client does.  The
    client will send a removal request and the server (assuming it
    is itself a POSIX system) will unlink the open file, really
    deleting the file later on final close.

 3. The server does not have the file open, but some client does.
    This could be a *different* client, in which case your Git
    process will result in a removal request, which will operate
    as in cases 1 and 2 above.  Or this could be *your* client, in
    which case your Git process will cause your client OS to send
    a "silly rename" if it has not already done so, *or* will cause
    your client *not* to send a removal request.

So, the main problem cases are case 3, where your own client is
the one that has the file open.  This causes your OS to *convert*
an unlink() call into a rename() call for a "silly rename", in
effect, *if* your OS has not yet done a silly rename.  But if your
OS has *already done* a "silly rename", your OS knows that the
".nfs<unique-id>" name is its own preserved name.

If the silly-rename operation has already occurred, your OS has
the option of making the unlink(".nfs<unique-id>") operation fail,
or ignoring it entirely.  If not, however, your OS has only one
option: to convert the unlink() to a silly rename and *report
success*.

That last case will definitely fool Git, which sees a successful
result of its unlink() call.  The other case -- the one where the
silly rename has already occurred -- will either report a failure
to unlink the silly-name ".nfs<unique-id>" file, which Git could
detect at that point, or will report success, lying to Git.

In both cases, of course, the directory will be non-empty at
the end of the series of unlink() calls, and the attempt to remove
the directory will fail with ENOTEMPTY.  Presumably Git should
detect this and warn, but there's nothing else Git can do here.

Anyway, that's the OS view of this mess. I leave the work on
Git itself to others. :-)

Chris

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Jeff King]

On Mon, Jun 10, 2024 at 06:09:52PM -0700, 'Yuri' wrote:

> "touch .nfs12309" isn't enough.
> 
> Here is a reliable way to reproduce the problem:
> 1. Have a git repository on an NFS disk.
> 2. mkdir xx
> 3. touch xx/x
> 4. tail -f xx/x &
> 5. rm xx/x
> 6. git clean -df .
> 
> The last operation reproduces the problem. The xx directory and the .nfsNNNN
> file in it stay without warnings.
> The .nfsNNNN file is created by the NFS client when the xx/x file is
> removed.

That is not the behavior I get. I see:

  $ git clean -df .
  warning: failed to remove xx/.nfs0000000002c8197f00000002: Device or resource busy

Which makes sense, since the kernel fails our unlink() call. Maybe your
system behaves differently at the syscall level?

This is a pretty standard Debian system with kernel 6.8.12. I set up the
NFS mount with:

   mkdir /mnt/{server,client}
   exportfs -o rw,sync 127.0.0.1:/mnt/server
   mount -t nfs 127.0.0.1:/mnt/server /mnt/client

and then made the repository in /mnt/client. "mount" tells me it's using
nfs4.

Running "git clean" on the server side does remove the files (no
warning, but the directories are actually removed).

-peff

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

['Yuri']

On 6/10/24 23:48, Jeff King wrote:
> $ git clean -df . warning: failed to remove 
> xx/.nfs0000000002c8197f00000002: Device or resource busy Which makes 
> sense, since the kernel fails our unlink() call. Maybe your system 
> behaves differently at the syscall level?


The system I observed the problem is Centos with the kernel 
3.10.0-1160.76.1.el7.x86_64 (10 years old).


"rm -rf xx" command also says Device or resource busy
But git-2.43.0 doesn't say anything.




Yuri

RE: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[rsbecker]

On Tuesday, June 11, 2024 2:49 AM, Jeff King wrote:
>On Mon, Jun 10, 2024 at 06:09:52PM -0700, 'Yuri' wrote:
>
>> "touch .nfs12309" isn't enough.
>>
>> Here is a reliable way to reproduce the problem:
>> 1. Have a git repository on an NFS disk.
>> 2. mkdir xx
>> 3. touch xx/x
>> 4. tail -f xx/x &
>> 5. rm xx/x
>> 6. git clean -df .
>>
>> The last operation reproduces the problem. The xx directory and the
>> .nfsNNNN file in it stay without warnings.
>> The .nfsNNNN file is created by the NFS client when the xx/x file is
>> removed.
>
>That is not the behavior I get. I see:
>
>  $ git clean -df .
>  warning: failed to remove xx/.nfs0000000002c8197f00000002: Device or
>resource busy
>
>Which makes sense, since the kernel fails our unlink() call. Maybe your system
>behaves differently at the syscall level?
>
>This is a pretty standard Debian system with kernel 6.8.12. I set up the NFS mount
>with:
>
>   mkdir /mnt/{server,client}
>   exportfs -o rw,sync 127.0.0.1:/mnt/server
>   mount -t nfs 127.0.0.1:/mnt/server /mnt/client
>
>and then made the repository in /mnt/client. "mount" tells me it's using nfs4.
>
>Running "git clean" on the server side does remove the files (no warning, but the
>directories are actually removed).

It has been a while since I did a self-mount in NFS, but I do not think that will reproduce the issue. The mounts have to be on different servers from the client to experience this silly rename situation. On self-mount, IIRC, the client is aware that it is on its own machine and will not try to detect the situation. 

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

['Yuri']

On 6/11/24 06:48, rsbecker@nexbridge.com wrote:
> It has been a while since I did a self-mount in NFS, but I do not 
> think that will reproduce the issue. The mounts have to be on 
> different servers from the client to experience this silly rename 
> situation. On self-mount, IIRC, the client is aware that it is on its 
> own machine and will not try to detect the situation. 


You can use a VirtualBox VM.



Yuri

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

[Gabor Gombas]

On Tue, Jun 11, 2024 at 12:43:51AM -0700, 'Yuri' wrote:
> The system I observed the problem is Centos with the kernel
> 3.10.0-1160.76.1.el7.x86_64 (10 years old).
> 
> 
> "rm -rf xx" command also says Device or resource busy
> But git-2.43.0 doesn't say anything.

On RHEL7:

$ mkdir foo
$ cd foo
$ git --version
git version 2.39.2
$ git init
$ mkdir xx
$ touch xx/x
$ tail -f xx/x
$ git clean -dfx
warning: failed to remove xx/: Directory not empty
Removig xx/x
$ git clean -dfx
warning: failed to remove xx/.nfs[... long hex number...]: Device or resource busy

Well, try something:

$ echo xx > .gitignore
$ git add .gitignore
$ git commit -m foo
$ git clean -df
[there is no output - bingo]
$ git clean -dfx
warning: failed to remove xx/.nfs[... long hex number...]: Device or resource busy

Regards,
Gabor

Re: [BUG] "git clean -df ." silently doesn't delete folders with stale .nfs* files

['Yuri']

On 6/13/24 01:09, Gabor Gombas wrote:
> $ git clean -df [there is no output - bingo] $ git clean -dfx warning: 
> failed to remove xx/.nfs[... long hex number...]: Device or resource busy


'git -dfx' does warn about the .nfsNNN files.

It turns out that my git repository has .nfs* files in the ignore list.

So it silently keeps these files with the -df flags.




Yuri