Re: knfsd server bug when GETATTR follows READDIR

[Chuck Lever <chuck.lever@oracle.com>]

On 12/24/24 1:51 AM, Cedric Blancher wrote:
> On Sun, 22 Dec 2024 at 21:19, Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>> On 12/21/24 6:53 PM, Rick Macklem wrote:
>>> On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>>>>
>>>> On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>
>>>>> On 12/20/24 9:16 PM, J David wrote:
>>>>>> Hello,
>>>>>>
>>>>>> On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>>> If they can reproduce
>>>>>>> this issue with an "in tree" file system contained in a recent upstream
>>>>>>> Linux kernel, then we can take a look. (Or you and J. David can give it
>>>>>>> a try).
>>>>>>
>>>>>> Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
>>>>>> Debian backports on completely different hardware.
>>>>>>
>>>>>> Then I set up another NFS server on Arch (running kernel 6.12.4), and
>>>>>> reproduced the issue there as well.
>>>>>>
>>>>>> Then, just to be sure, I went and found the instructions for building
>>>>>> the Linux kernel from source, built and tested both 6.12.6 and
>>>>>> 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
>>>>>> occurs with those as well.
>>>>>
>>>>> Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
>>>>>
>>>>>
>>>>>> Additionally, I have tested every combination of FreeBSD, Linux and
>>>>>> OpenIndiana as client and server to confirm that FreeBSD client with
>>>>>> Linux server is the only case where this problem occurs.
>>>>>
>>>>> Interesting.
>>>>>
>>>>>
>>>>>> Does this count as reproducing the issue with an "in tree" file system
>>>>>> contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
>>>>>> so far out of my depth that I'm pretty sure there are sea monsters
>>>>>> swimming around down there. So I can't rule out the possibility that
>>>>>> I've done something wrong either in setup or testing.
>>>>>>
>>>>>> During the course of this, I've gotten the reproduction down to
>>>>>> extracting a 2k tar file and then running "du" on the resulting
>>>>>> directory from the client. Doesn't matter if the file is untarred on
>>>>>> the FreeBSD client, the server, or another client. The tar file
>>>>>> contains a directory with a handful of random Javascript files from
>>>>>> Drupal. As far as I can tell, it has something to do with the number,
>>>>>> size, or names of the files. The Drupal project has three separate
>>>>>> directories all structured like this with the same filenames, but the
>>>>>> file contents vary. The issue occurs with all of them.
>>>>>>
>>>>>> The Linux /etc/exports file is just:
>>>>>>
>>>>>> /data 192.168.201.0/24(rw,sync)
>>>>>>
>>>>>> (The production case also uses crossmnt and no_subtree_check, anonuid,
>>>>>> and anongid, but I eliminated those one by one to make sure they
>>>>>> weren't responsible.)
>>>>>>
>>>>>> The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
>>>>>>
>>>>>> 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
>>>>>
>>>>> Out of curiosity, do you see the problem recur with nfsv3 or the other
>>>>> NFSv4 minor versions?
>>>>>
>>>>>
>>>>>> One additional thing I noticed that really blew my mind is that I can
>>>>>> shutdown both the client and the server, wait, power them back on, and
>>>>>> the issue is still there. So it's not something in RAM.  That prompted
>>>>>> me to try "touch x" in the directory to create a new 0-length file.
>>>>>> The issue then goes away. Then I can "rm x" and the issue comes back.
>>>>>> By contrast, I can write megabytes from /dev/random into one of the
>>>>>> files without affecting anything; the issue stays the same.
>>>>>>
>>>>>> I then tried it with all empty files using the same filenames. The
>>>>>> issue still occurred. Add or remove one file and the issue goes away.
>>>>>> I then renamed one of the files to zz.js. Issue still occurs. Renamed
>>>>>> it to zzz.js. Problem still occurs. Kept going until I got to
>>>>>> zzzzzz.js and it worked.
>>>>>>
>>>>>> Finally, I got it to the point where running this in an empty mounted
>>>>>> directory will create the issue:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxxxxx.xx
>>>>>>
>>>>>> and this will not:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxxxxxx.xx
>>>>>>
>>>>>> (The difference being one extra x in the last filename.)
>>>>>>
>>>>>> It works in the other direction as well. This causes the issue:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxx.xx
>>>>>>
>>>>>> This does not:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xx.xx
>>>>>>
>>>>>> There's a four-character window involving the length of the filenames
>>>>>> where 62 files in a directory causes this issue. There's a little more
>>>>>> to it than that; it doesn't look like you can just create 61
>>>>>> two-letter filenames and then one really long one and get the issue.
>>>>>>
>>>>>> So I haven't found the specifics yet, but perhaps due to pure chance
>>>>>> this directory structure is exactly right to provoke an incredibly
>>>>>> obscure edge case?
>>>>>
>>>>> Well it's likely that this is a problem with READDIR, so file content
>>>>> is not going to be an issue. The file name lengths are the problem.
>>>>>
>>>>> Also, I'm wondering what the FreeBSD client's directory readdir
>>>>> arguments are (how much does it request, what are the maximum limits it
>>>>> negotiates, and so on). Rick?
>>>> As you'll see in the packet trace:
>>>> Sequence: cache this: No
>>>> Putfh: directory fh
>>>> Readdir:
>>>>       cookie: 0
>>>>       cookie_verf: 0
>>>>       dircount: 8706
>>>>       maxcount: 8706
>>>>       attr: type, RDattr_error, fileid, mounted_on_fleid
>>>> Getattr: same attributes as requested for a previous GETATTR, mainly
>>>>                 to keep the directory's attribute cache up to date.
>>>>
>>>> The session negotiates a max request/reply size of just over 1Mbyte and a
>>>> maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
>>>>
>>>> If you are wondering where the 8706 comes from, it was an estimate of how
>>>> much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
>>>> by adding 512 to 8K.
>>>>
>>>> I have not yet had a chance to see if I can reproduce the problem with
>>>> J. David's
>>>> reproducer. I will try that soon, and if I can reproduce it, I will
>>>> poke at it to try and
>>>> figure out what is going on.
>>> Just fyi, I have reproduced it. Once you use J. David's little shell script to
>>> create the files in the directory, the Readdir RPC gets the junk reply
>>> to GETATTR
>>> (the count of words for the attribute bitmap in the reply is 0 instead of 2).
>>> You can unmount/remount it and still get the failure, assuming you do not
>>> mess with the directory contents.
>>>
>>> Good work finding the reproducer, J. David!
>>>
>>> I will start to poke around to see if I can figure out what the knfsd server is
>>> doing.
>>>
>>> Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
>>> reproduce this, just in case you are inspired to cross over to the dark
>>> side and install FreeBSD somewhere.
>>
>> I see the same malformed GETATTR result in the attachments.
>>
>> Linux doesn't trip on this issue because it's NFS client doesn't ever
>> append a GETATTR operation after a READDIR.
> 
> Windows ms-nfs41-client driver also does GETATTR after READDIR, and
> trips over bogus return values on a regular basis. Solaris 11.4 nfsd
> and nfs4j do not exhibit such problems.
> 
> Another garbage value that client gets from Linux nfsd is
> FATTR4_WORD0_CHANGE, which sometimes returns absurdly high values.
> Maybe add some WARN_ONCE() to Linux nfsd if unexpected crazy values
> are sent over the wire?

I prefer that we address these bugs forthwith. If you have a reproducer
for these behaviors, please file a bug on bugzilla.kernel.org under the
filesystem/NFSD component.

We can't fix bugs that we don't know about...

-- 
Chuck Lever