knfsd server bug when GETATTR follows READDIR

knfsd server bug when GETATTR follows READDIR

[Rick Macklem]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

Hi,

The attached pcap file shows that the knfsd server generates
bogus XDR for the reply to a GETATTR that follows a READDIR
operation.

More specifically, if you look at the pcap file in wireshark
and go to packet#22 and then click on the operations and
then "Opcode: GETATTR (9)", the start of
the XDR for the GETATTR will be highlighted in the hexadecimal
window.
Now, if you look at what follows (in the hexadeciaml window),
you'll see that the GETATTR reply looks like:
- GETATTR (9)
- NFS4_OK (0)
- Length of bitmap (0) <-- Not (2)
- 2 words of attribute bitmap
- 98 (length of attributes in hex)
- attribute values

Everything looks ok, except the number of bitmap words is
0 and not 2.

Since the knfsd does not do this normally, I'd guess it is
some sort of runaway pointer or use after free type bug that
causes this, maybe?

Sofar, it only appears to happen when the GETATTR follows a
READDIR operation.

This was reported to me for a FreeBSD client mounting the following:
Debian 12 w/kernel:
$ uname -r
6.1.0-25-amd64

> - what type of file system it exports

ZFS:

$ dpkg -l | fgrep libzfs4linux
ii  libzfs4linux                    2.1.11-1
               amd64
I suspect that ZFS exports are not common for the Linux knfsd?

Anyhow, I am not sure if you have seen such a problem before,
but I thought I would at least report it.
(I have cc'd the reporter, in case you have questions for him.)

rick
ps: If the pcap file does not make it through the mailing list,
       email me and I'll send you a copy.

[-- Attachment #2: rpc-struct-is-bad.pcap --]
[-- Type: application/octet-stream, Size: 10666 bytes --]

Re: knfsd server bug when GETATTR follows READDIR

[Chuck Lever]

On 12/17/24 5:10 PM, Rick Macklem wrote:
> Hi,
> 
> The attached pcap file shows that the knfsd server generates
> bogus XDR for the reply to a GETATTR that follows a READDIR
> operation.
> 
> More specifically, if you look at the pcap file in wireshark
> and go to packet#22 and then click on the operations and
> then "Opcode: GETATTR (9)", the start of
> the XDR for the GETATTR will be highlighted in the hexadecimal
> window.
> Now, if you look at what follows (in the hexadeciaml window),
> you'll see that the GETATTR reply looks like:
> - GETATTR (9)
> - NFS4_OK (0)
> - Length of bitmap (0) <-- Not (2)
> - 2 words of attribute bitmap
> - 98 (length of attributes in hex)
> - attribute values
> 
> Everything looks ok, except the number of bitmap words is
> 0 and not 2.
> 
> Since the knfsd does not do this normally, I'd guess it is
> some sort of runaway pointer or use after free type bug that
> causes this, maybe?
> 
> Sofar, it only appears to happen when the GETATTR follows a
> READDIR operation.
> 
> This was reported to me for a FreeBSD client mounting the following:
> Debian 12 w/kernel:
> $ uname -r
> 6.1.0-25-amd64
> 
>> - what type of file system it exports
> 
> ZFS:
> 
> $ dpkg -l | fgrep libzfs4linux
> ii  libzfs4linux                    2.1.11-1
>                 amd64
> I suspect that ZFS exports are not common for the Linux knfsd?
> 
> Anyhow, I am not sure if you have seen such a problem before,
> but I thought I would at least report it.
> (I have cc'd the reporter, in case you have questions for him.)
> 
> rick
> ps: If the pcap file does not make it through the mailing list,
>         email me and I'll send you a copy.

Hi Rick,

ZFS is an "out of tree" filesystem, so the upstream Linux community does
not support it. I'm guessing libzfs4linux has its own upstream
community. Even so, it is a not unpopular choice among Linux NAS
enthusiasts.

Also, 6.1.0-25-amd64 is a Debian-built kernel, not sure how it relates 
to the upstream kernel (distros add their own special sauce, and this
code base looks a couple of years old to begin with).

It might seem unfriendly, but we usually ask, in such cases, for the
reporter to work with the Linux distributor first. If they can reproduce
this issue with an "in tree" file system contained in a recent upstream
Linux kernel, then we can take a look. (Or you and J. David can give it
a try).

If it can be reproduced, we have to fix the tip of the upstream branch
first, and then take that patch, as cleanly as possible, back to the LTS
6.1 kernel code base that 6.1.0-25-amd64 is based on. Debian would then
be responsible for taking the LTS fix into their kernel.

If it cannot be reproduced, then it's likely there is already a fix
somewhere between 6.1 and the tip of upstream. Generally a "git bisect"
can identify the commit, and it can then be backported as described
above.

HTH


-- 
Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[J David]

Hello,

On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> If they can reproduce
> this issue with an "in tree" file system contained in a recent upstream
> Linux kernel, then we can take a look. (Or you and J. David can give it
> a try).

Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
Debian backports on completely different hardware.

Then I set up another NFS server on Arch (running kernel 6.12.4), and
reproduced the issue there as well.

Then, just to be sure, I went and found the instructions for building
the Linux kernel from source, built and tested both 6.12.6 and
6.13-rc3 as downloaded directly from www.kernel.org, and the issue
occurs with those as well.

Additionally, I have tested every combination of FreeBSD, Linux and
OpenIndiana as client and server to confirm that FreeBSD client with
Linux server is the only case where this problem occurs.

Does this count as reproducing the issue with an "in tree" file system
contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
so far out of my depth that I'm pretty sure there are sea monsters
swimming around down there. So I can't rule out the possibility that
I've done something wrong either in setup or testing.

During the course of this, I've gotten the reproduction down to
extracting a 2k tar file and then running "du" on the resulting
directory from the client. Doesn't matter if the file is untarred on
the FreeBSD client, the server, or another client. The tar file
contains a directory with a handful of random Javascript files from
Drupal. As far as I can tell, it has something to do with the number,
size, or names of the files. The Drupal project has three separate
directories all structured like this with the same filenames, but the
file contents vary. The issue occurs with all of them.

The Linux /etc/exports file is just:

/data 192.168.201.0/24(rw,sync)

(The production case also uses crossmnt and no_subtree_check, anonuid,
and anongid, but I eliminated those one by one to make sure they
weren't responsible.)

The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:

192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0

One additional thing I noticed that really blew my mind is that I can
shutdown both the client and the server, wait, power them back on, and
the issue is still there. So it's not something in RAM.  That prompted
me to try "touch x" in the directory to create a new 0-length file.
The issue then goes away. Then I can "rm x" and the issue comes back.
By contrast, I can write megabytes from /dev/random into one of the
files without affecting anything; the issue stays the same.

I then tried it with all empty files using the same filenames. The
issue still occurred. Add or remove one file and the issue goes away.
I then renamed one of the files to zz.js. Issue still occurs. Renamed
it to zzz.js. Problem still occurs. Kept going until I got to
zzzzzz.js and it worked.

Finally, I got it to the point where running this in an empty mounted
directory will create the issue:

rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
done; touch y0-xxxxxx.xx

and this will not:

rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
done; touch y0-xxxxxxx.xx

(The difference being one extra x in the last filename.)

It works in the other direction as well. This causes the issue:

rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
done; touch y0-xxx.xx

This does not:

rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
done; touch y0-xx.xx

There's a four-character window involving the length of the filenames
where 62 files in a directory causes this issue. There's a little more
to it than that; it doesn't look like you can just create 61
two-letter filenames and then one really long one and get the issue.

So I haven't found the specifics yet, but perhaps due to pure chance
this directory structure is exactly right to provoke an incredibly
obscure edge case?

Thanks!

Re: knfsd server bug when GETATTR follows READDIR

[Chuck Lever]

On 12/20/24 9:16 PM, J David wrote:
> Hello,
> 
> On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>> If they can reproduce
>> this issue with an "in tree" file system contained in a recent upstream
>> Linux kernel, then we can take a look. (Or you and J. David can give it
>> a try).
> 
> Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
> Debian backports on completely different hardware.
> 
> Then I set up another NFS server on Arch (running kernel 6.12.4), and
> reproduced the issue there as well.
> 
> Then, just to be sure, I went and found the instructions for building
> the Linux kernel from source, built and tested both 6.12.6 and
> 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
> occurs with those as well.

Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!


> Additionally, I have tested every combination of FreeBSD, Linux and
> OpenIndiana as client and server to confirm that FreeBSD client with
> Linux server is the only case where this problem occurs.

Interesting.


> Does this count as reproducing the issue with an "in tree" file system
> contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
> so far out of my depth that I'm pretty sure there are sea monsters
> swimming around down there. So I can't rule out the possibility that
> I've done something wrong either in setup or testing.
> 
> During the course of this, I've gotten the reproduction down to
> extracting a 2k tar file and then running "du" on the resulting
> directory from the client. Doesn't matter if the file is untarred on
> the FreeBSD client, the server, or another client. The tar file
> contains a directory with a handful of random Javascript files from
> Drupal. As far as I can tell, it has something to do with the number,
> size, or names of the files. The Drupal project has three separate
> directories all structured like this with the same filenames, but the
> file contents vary. The issue occurs with all of them.
> 
> The Linux /etc/exports file is just:
> 
> /data 192.168.201.0/24(rw,sync)
> 
> (The production case also uses crossmnt and no_subtree_check, anonuid,
> and anongid, but I eliminated those one by one to make sure they
> weren't responsible.)
> 
> The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
> 
> 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0

Out of curiosity, do you see the problem recur with nfsv3 or the other
NFSv4 minor versions?


> One additional thing I noticed that really blew my mind is that I can
> shutdown both the client and the server, wait, power them back on, and
> the issue is still there. So it's not something in RAM.  That prompted
> me to try "touch x" in the directory to create a new 0-length file.
> The issue then goes away. Then I can "rm x" and the issue comes back.
> By contrast, I can write megabytes from /dev/random into one of the
> files without affecting anything; the issue stays the same.
> 
> I then tried it with all empty files using the same filenames. The
> issue still occurred. Add or remove one file and the issue goes away.
> I then renamed one of the files to zz.js. Issue still occurs. Renamed
> it to zzz.js. Problem still occurs. Kept going until I got to
> zzzzzz.js and it worked.
> 
> Finally, I got it to the point where running this in an empty mounted
> directory will create the issue:
> 
> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> done; touch y0-xxxxxx.xx
> 
> and this will not:
> 
> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> done; touch y0-xxxxxxx.xx
> 
> (The difference being one extra x in the last filename.)
> 
> It works in the other direction as well. This causes the issue:
> 
> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> done; touch y0-xxx.xx
> 
> This does not:
> 
> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> done; touch y0-xx.xx
> 
> There's a four-character window involving the length of the filenames
> where 62 files in a directory causes this issue. There's a little more
> to it than that; it doesn't look like you can just create 61
> two-letter filenames and then one really long one and get the issue.
> 
> So I haven't found the specifics yet, but perhaps due to pure chance
> this directory structure is exactly right to provoke an incredibly
> obscure edge case?

Well it's likely that this is a problem with READDIR, so file content
is not going to be an issue. The file name lengths are the problem.

Also, I'm wondering what the FreeBSD client's directory readdir
arguments are (how much does it request, what are the maximum limits it
negotiates, and so on). Rick?

Since this isn't reproducible (yet) with a Linux client, let's try
another set of network captures, and you can send these to me
privately.

Start the capture
Mount
Run one of the reproducers above
Unmount
Stop the capture

I'd like to see one with v6.13-rc3 and ext4 that works as expected, and
one with the same configuration that fails.

-- 
Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[J David]

On Sat, Dec 21, 2024 at 12:34 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> Out of curiosity, do you see the problem recur with nfsv3 or the other
> NFSv4 minor versions?

Great question. I tested this today. The issue also occurs with
NFSv4.1, but *not* with NFSv4.0. (And, not surprisingly, does not
occur with v3.)

Since I believe OpenIndiana only implements up to v4.0 that might
eliminate it as a useful third point of comparison. I'm not sure if
there are any other open implementations of NFS v4.1/v4.2.

> Since this isn't reproducible (yet) with a Linux client, let's try
> another set of network captures, and you can send these to me
> privately.

Will follow up on this as soon as possible.

Thanks!

Re: knfsd server bug when GETATTR follows READDIR

[Rick Macklem]

On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> On 12/20/24 9:16 PM, J David wrote:
> > Hello,
> >
> > On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> >> If they can reproduce
> >> this issue with an "in tree" file system contained in a recent upstream
> >> Linux kernel, then we can take a look. (Or you and J. David can give it
> >> a try).
> >
> > Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
> > Debian backports on completely different hardware.
> >
> > Then I set up another NFS server on Arch (running kernel 6.12.4), and
> > reproduced the issue there as well.
> >
> > Then, just to be sure, I went and found the instructions for building
> > the Linux kernel from source, built and tested both 6.12.6 and
> > 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
> > occurs with those as well.
>
> Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
>
>
> > Additionally, I have tested every combination of FreeBSD, Linux and
> > OpenIndiana as client and server to confirm that FreeBSD client with
> > Linux server is the only case where this problem occurs.
>
> Interesting.
>
>
> > Does this count as reproducing the issue with an "in tree" file system
> > contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
> > so far out of my depth that I'm pretty sure there are sea monsters
> > swimming around down there. So I can't rule out the possibility that
> > I've done something wrong either in setup or testing.
> >
> > During the course of this, I've gotten the reproduction down to
> > extracting a 2k tar file and then running "du" on the resulting
> > directory from the client. Doesn't matter if the file is untarred on
> > the FreeBSD client, the server, or another client. The tar file
> > contains a directory with a handful of random Javascript files from
> > Drupal. As far as I can tell, it has something to do with the number,
> > size, or names of the files. The Drupal project has three separate
> > directories all structured like this with the same filenames, but the
> > file contents vary. The issue occurs with all of them.
> >
> > The Linux /etc/exports file is just:
> >
> > /data 192.168.201.0/24(rw,sync)
> >
> > (The production case also uses crossmnt and no_subtree_check, anonuid,
> > and anongid, but I eliminated those one by one to make sure they
> > weren't responsible.)
> >
> > The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
> >
> > 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
>
> Out of curiosity, do you see the problem recur with nfsv3 or the other
> NFSv4 minor versions?
>
>
> > One additional thing I noticed that really blew my mind is that I can
> > shutdown both the client and the server, wait, power them back on, and
> > the issue is still there. So it's not something in RAM.  That prompted
> > me to try "touch x" in the directory to create a new 0-length file.
> > The issue then goes away. Then I can "rm x" and the issue comes back.
> > By contrast, I can write megabytes from /dev/random into one of the
> > files without affecting anything; the issue stays the same.
> >
> > I then tried it with all empty files using the same filenames. The
> > issue still occurred. Add or remove one file and the issue goes away.
> > I then renamed one of the files to zz.js. Issue still occurs. Renamed
> > it to zzz.js. Problem still occurs. Kept going until I got to
> > zzzzzz.js and it worked.
> >
> > Finally, I got it to the point where running this in an empty mounted
> > directory will create the issue:
> >
> > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > done; touch y0-xxxxxx.xx
> >
> > and this will not:
> >
> > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > done; touch y0-xxxxxxx.xx
> >
> > (The difference being one extra x in the last filename.)
> >
> > It works in the other direction as well. This causes the issue:
> >
> > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > done; touch y0-xxx.xx
> >
> > This does not:
> >
> > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > done; touch y0-xx.xx
> >
> > There's a four-character window involving the length of the filenames
> > where 62 files in a directory causes this issue. There's a little more
> > to it than that; it doesn't look like you can just create 61
> > two-letter filenames and then one really long one and get the issue.
> >
> > So I haven't found the specifics yet, but perhaps due to pure chance
> > this directory structure is exactly right to provoke an incredibly
> > obscure edge case?
>
> Well it's likely that this is a problem with READDIR, so file content
> is not going to be an issue. The file name lengths are the problem.
>
> Also, I'm wondering what the FreeBSD client's directory readdir
> arguments are (how much does it request, what are the maximum limits it
> negotiates, and so on). Rick?
As you'll see in the packet trace:
Sequence: cache this: No
Putfh: directory fh
Readdir:
    cookie: 0
    cookie_verf: 0
    dircount: 8706
    maxcount: 8706
    attr: type, RDattr_error, fileid, mounted_on_fleid
Getattr: same attributes as requested for a previous GETATTR, mainly
              to keep the directory's attribute cache up to date.

The session negotiates a max request/reply size of just over 1Mbyte and a
maximum of something like 20 ops. (Can't recall, but definitely more than 4.)

If you are wondering where the 8706 comes from, it was an estimate of how
much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
by adding 512 to 8K.

I have not yet had a chance to see if I can reproduce the problem with
J. David's
reproducer. I will try that soon, and if I can reproduce it, I will
poke at it to try and
figure out what is going on.

rick

>
> Since this isn't reproducible (yet) with a Linux client, let's try
> another set of network captures, and you can send these to me
> privately.
>
> Start the capture
> Mount
> Run one of the reproducers above
> Unmount
> Stop the capture
>
> I'd like to see one with v6.13-rc3 and ext4 that works as expected, and
> one with the same configuration that fails.
>
> --
> Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[Rick Macklem]

On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
> >
> > On 12/20/24 9:16 PM, J David wrote:
> > > Hello,
> > >
> > > On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> > >> If they can reproduce
> > >> this issue with an "in tree" file system contained in a recent upstream
> > >> Linux kernel, then we can take a look. (Or you and J. David can give it
> > >> a try).
> > >
> > > Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
> > > Debian backports on completely different hardware.
> > >
> > > Then I set up another NFS server on Arch (running kernel 6.12.4), and
> > > reproduced the issue there as well.
> > >
> > > Then, just to be sure, I went and found the instructions for building
> > > the Linux kernel from source, built and tested both 6.12.6 and
> > > 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
> > > occurs with those as well.
> >
> > Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
> >
> >
> > > Additionally, I have tested every combination of FreeBSD, Linux and
> > > OpenIndiana as client and server to confirm that FreeBSD client with
> > > Linux server is the only case where this problem occurs.
> >
> > Interesting.
> >
> >
> > > Does this count as reproducing the issue with an "in tree" file system
> > > contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
> > > so far out of my depth that I'm pretty sure there are sea monsters
> > > swimming around down there. So I can't rule out the possibility that
> > > I've done something wrong either in setup or testing.
> > >
> > > During the course of this, I've gotten the reproduction down to
> > > extracting a 2k tar file and then running "du" on the resulting
> > > directory from the client. Doesn't matter if the file is untarred on
> > > the FreeBSD client, the server, or another client. The tar file
> > > contains a directory with a handful of random Javascript files from
> > > Drupal. As far as I can tell, it has something to do with the number,
> > > size, or names of the files. The Drupal project has three separate
> > > directories all structured like this with the same filenames, but the
> > > file contents vary. The issue occurs with all of them.
> > >
> > > The Linux /etc/exports file is just:
> > >
> > > /data 192.168.201.0/24(rw,sync)
> > >
> > > (The production case also uses crossmnt and no_subtree_check, anonuid,
> > > and anongid, but I eliminated those one by one to make sure they
> > > weren't responsible.)
> > >
> > > The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
> > >
> > > 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
> >
> > Out of curiosity, do you see the problem recur with nfsv3 or the other
> > NFSv4 minor versions?
> >
> >
> > > One additional thing I noticed that really blew my mind is that I can
> > > shutdown both the client and the server, wait, power them back on, and
> > > the issue is still there. So it's not something in RAM.  That prompted
> > > me to try "touch x" in the directory to create a new 0-length file.
> > > The issue then goes away. Then I can "rm x" and the issue comes back.
> > > By contrast, I can write megabytes from /dev/random into one of the
> > > files without affecting anything; the issue stays the same.
> > >
> > > I then tried it with all empty files using the same filenames. The
> > > issue still occurred. Add or remove one file and the issue goes away.
> > > I then renamed one of the files to zz.js. Issue still occurs. Renamed
> > > it to zzz.js. Problem still occurs. Kept going until I got to
> > > zzzzzz.js and it worked.
> > >
> > > Finally, I got it to the point where running this in an empty mounted
> > > directory will create the issue:
> > >
> > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > done; touch y0-xxxxxx.xx
> > >
> > > and this will not:
> > >
> > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > done; touch y0-xxxxxxx.xx
> > >
> > > (The difference being one extra x in the last filename.)
> > >
> > > It works in the other direction as well. This causes the issue:
> > >
> > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > done; touch y0-xxx.xx
> > >
> > > This does not:
> > >
> > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > done; touch y0-xx.xx
> > >
> > > There's a four-character window involving the length of the filenames
> > > where 62 files in a directory causes this issue. There's a little more
> > > to it than that; it doesn't look like you can just create 61
> > > two-letter filenames and then one really long one and get the issue.
> > >
> > > So I haven't found the specifics yet, but perhaps due to pure chance
> > > this directory structure is exactly right to provoke an incredibly
> > > obscure edge case?
> >
> > Well it's likely that this is a problem with READDIR, so file content
> > is not going to be an issue. The file name lengths are the problem.
> >
> > Also, I'm wondering what the FreeBSD client's directory readdir
> > arguments are (how much does it request, what are the maximum limits it
> > negotiates, and so on). Rick?
> As you'll see in the packet trace:
> Sequence: cache this: No
> Putfh: directory fh
> Readdir:
>     cookie: 0
>     cookie_verf: 0
>     dircount: 8706
>     maxcount: 8706
>     attr: type, RDattr_error, fileid, mounted_on_fleid
> Getattr: same attributes as requested for a previous GETATTR, mainly
>               to keep the directory's attribute cache up to date.
>
> The session negotiates a max request/reply size of just over 1Mbyte and a
> maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
>
> If you are wondering where the 8706 comes from, it was an estimate of how
> much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
> by adding 512 to 8K.
>
> I have not yet had a chance to see if I can reproduce the problem with
> J. David's
> reproducer. I will try that soon, and if I can reproduce it, I will
> poke at it to try and
> figure out what is going on.
Just fyi, I have reproduced it. Once you use J. David's little shell script to
create the files in the directory, the Readdir RPC gets the junk reply
to GETATTR
(the count of words for the attribute bitmap in the reply is 0 instead of 2).
You can unmount/remount it and still get the failure, assuming you do not
mess with the directory contents.

Good work finding the reproducer, J. David!

I will start to poke around to see if I can figure out what the knfsd server is
doing.

Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
reproduce this, just in case you are inspired to cross over to the dark
side and install FreeBSD somewhere.

I'll post when I have more info, rick

>
> rick
>
> >
> > Since this isn't reproducible (yet) with a Linux client, let's try
> > another set of network captures, and you can send these to me
> > privately.
> >
> > Start the capture
> > Mount
> > Run one of the reproducers above
> > Unmount
> > Stop the capture
> >
> > I'd like to see one with v6.13-rc3 and ext4 that works as expected, and
> > one with the same configuration that fails.
> >
> > --
> > Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[Rick Macklem]

On Sat, Dec 21, 2024 at 3:53 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
> >
> > On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
> > >
> > > On 12/20/24 9:16 PM, J David wrote:
> > > > Hello,
> > > >
> > > > On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> > > >> If they can reproduce
> > > >> this issue with an "in tree" file system contained in a recent upstream
> > > >> Linux kernel, then we can take a look. (Or you and J. David can give it
> > > >> a try).
> > > >
> > > > Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
> > > > Debian backports on completely different hardware.
> > > >
> > > > Then I set up another NFS server on Arch (running kernel 6.12.4), and
> > > > reproduced the issue there as well.
> > > >
> > > > Then, just to be sure, I went and found the instructions for building
> > > > the Linux kernel from source, built and tested both 6.12.6 and
> > > > 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
> > > > occurs with those as well.
> > >
> > > Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
> > >
> > >
> > > > Additionally, I have tested every combination of FreeBSD, Linux and
> > > > OpenIndiana as client and server to confirm that FreeBSD client with
> > > > Linux server is the only case where this problem occurs.
> > >
> > > Interesting.
> > >
> > >
> > > > Does this count as reproducing the issue with an "in tree" file system
> > > > contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
> > > > so far out of my depth that I'm pretty sure there are sea monsters
> > > > swimming around down there. So I can't rule out the possibility that
> > > > I've done something wrong either in setup or testing.
> > > >
> > > > During the course of this, I've gotten the reproduction down to
> > > > extracting a 2k tar file and then running "du" on the resulting
> > > > directory from the client. Doesn't matter if the file is untarred on
> > > > the FreeBSD client, the server, or another client. The tar file
> > > > contains a directory with a handful of random Javascript files from
> > > > Drupal. As far as I can tell, it has something to do with the number,
> > > > size, or names of the files. The Drupal project has three separate
> > > > directories all structured like this with the same filenames, but the
> > > > file contents vary. The issue occurs with all of them.
> > > >
> > > > The Linux /etc/exports file is just:
> > > >
> > > > /data 192.168.201.0/24(rw,sync)
> > > >
> > > > (The production case also uses crossmnt and no_subtree_check, anonuid,
> > > > and anongid, but I eliminated those one by one to make sure they
> > > > weren't responsible.)
> > > >
> > > > The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
> > > >
> > > > 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
> > >
> > > Out of curiosity, do you see the problem recur with nfsv3 or the other
> > > NFSv4 minor versions?
> > >
> > >
> > > > One additional thing I noticed that really blew my mind is that I can
> > > > shutdown both the client and the server, wait, power them back on, and
> > > > the issue is still there. So it's not something in RAM.  That prompted
> > > > me to try "touch x" in the directory to create a new 0-length file.
> > > > The issue then goes away. Then I can "rm x" and the issue comes back.
> > > > By contrast, I can write megabytes from /dev/random into one of the
> > > > files without affecting anything; the issue stays the same.
> > > >
> > > > I then tried it with all empty files using the same filenames. The
> > > > issue still occurred. Add or remove one file and the issue goes away.
> > > > I then renamed one of the files to zz.js. Issue still occurs. Renamed
> > > > it to zzz.js. Problem still occurs. Kept going until I got to
> > > > zzzzzz.js and it worked.
> > > >
> > > > Finally, I got it to the point where running this in an empty mounted
> > > > directory will create the issue:
> > > >
> > > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > > done; touch y0-xxxxxx.xx
> > > >
> > > > and this will not:
> > > >
> > > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > > done; touch y0-xxxxxxx.xx
> > > >
> > > > (The difference being one extra x in the last filename.)
> > > >
> > > > It works in the other direction as well. This causes the issue:
> > > >
> > > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > > done; touch y0-xxx.xx
> > > >
> > > > This does not:
> > > >
> > > > rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> > > > touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> > > > done; touch y0-xx.xx
> > > >
> > > > There's a four-character window involving the length of the filenames
> > > > where 62 files in a directory causes this issue. There's a little more
> > > > to it than that; it doesn't look like you can just create 61
> > > > two-letter filenames and then one really long one and get the issue.
> > > >
> > > > So I haven't found the specifics yet, but perhaps due to pure chance
> > > > this directory structure is exactly right to provoke an incredibly
> > > > obscure edge case?
> > >
> > > Well it's likely that this is a problem with READDIR, so file content
> > > is not going to be an issue. The file name lengths are the problem.
> > >
> > > Also, I'm wondering what the FreeBSD client's directory readdir
> > > arguments are (how much does it request, what are the maximum limits it
> > > negotiates, and so on). Rick?
> > As you'll see in the packet trace:
> > Sequence: cache this: No
> > Putfh: directory fh
> > Readdir:
> >     cookie: 0
> >     cookie_verf: 0
> >     dircount: 8706
> >     maxcount: 8706
> >     attr: type, RDattr_error, fileid, mounted_on_fleid
> > Getattr: same attributes as requested for a previous GETATTR, mainly
> >               to keep the directory's attribute cache up to date.
> >
> > The session negotiates a max request/reply size of just over 1Mbyte and a
> > maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
> >
> > If you are wondering where the 8706 comes from, it was an estimate of how
> > much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
> > by adding 512 to 8K.
> >
> > I have not yet had a chance to see if I can reproduce the problem with
> > J. David's
> > reproducer. I will try that soon, and if I can reproduce it, I will
> > poke at it to try and
> > figure out what is going on.
> Just fyi, I have reproduced it. Once you use J. David's little shell script to
> create the files in the directory, the Readdir RPC gets the junk reply
> to GETATTR
> (the count of words for the attribute bitmap in the reply is 0 instead of 2).
> You can unmount/remount it and still get the failure, assuming you do not
> mess with the directory contents.
>
> Good work finding the reproducer, J. David!
>
> I will start to poke around to see if I can figure out what the knfsd server is
> doing.
>
> Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
> reproduce this, just in case you are inspired to cross over to the dark
> side and install FreeBSD somewhere.
>
> I'll post when I have more info, rick
Here's a little more info...
(A) NFSv4.0 with the following ops in a compound works (as J. David noted)
PUTFH
READDIR
GETATTR

(B) NFSv4.1/4.2 with the following ops in a compound works
SEQUENCE
PUTFH
READDIR

(C) NFSv4.1/4.2 with the following ops in a compound does not work
SEQUENCE
PUTFH
READDIR
GETATTR

Note that all I did for (B) was remove the GETATTR from the compound
and that (A) uses the same PUTFH, READDIR and GETATTR as (C).
Btw J. David, the patch I sent you that removes GETATTR from the RPC
does seem to be a workaround.

rick


>
> >
> > rick
> >
> > >
> > > Since this isn't reproducible (yet) with a Linux client, let's try
> > > another set of network captures, and you can send these to me
> > > privately.
> > >
> > > Start the capture
> > > Mount
> > > Run one of the reproducers above
> > > Unmount
> > > Stop the capture
> > >
> > > I'd like to see one with v6.13-rc3 and ext4 that works as expected, and
> > > one with the same configuration that fails.
> > >
> > > --
> > > Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[J David]

On Sat, Dec 21, 2024 at 7:30 PM Rick Macklem <rick.macklem@gmail.com> wrote:
> On Sat, Dec 21, 2024 at 3:53 PM Rick Macklem <rick.macklem@gmail.com> wrote:
> > I have reproduced it.

Let me take a second to appreciate how great it feels anytime someone
else says that.

> Btw J. David, the patch I sent you that removes GETATTR from the RPC
> does seem to be a workaround.

Great! I have the source tree patched and ready to build, but haven't
wanted to mess up my ability to reproduce this in the test
environment. Now that it's out there, I'll give that a go.

Thanks!

Re: knfsd server bug when GETATTR follows READDIR

[Chuck Lever]

On 12/21/24 6:53 PM, Rick Macklem wrote:
> On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>>
>> On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>
>>> On 12/20/24 9:16 PM, J David wrote:
>>>> Hello,
>>>>
>>>> On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>> If they can reproduce
>>>>> this issue with an "in tree" file system contained in a recent upstream
>>>>> Linux kernel, then we can take a look. (Or you and J. David can give it
>>>>> a try).
>>>>
>>>> Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
>>>> Debian backports on completely different hardware.
>>>>
>>>> Then I set up another NFS server on Arch (running kernel 6.12.4), and
>>>> reproduced the issue there as well.
>>>>
>>>> Then, just to be sure, I went and found the instructions for building
>>>> the Linux kernel from source, built and tested both 6.12.6 and
>>>> 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
>>>> occurs with those as well.
>>>
>>> Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
>>>
>>>
>>>> Additionally, I have tested every combination of FreeBSD, Linux and
>>>> OpenIndiana as client and server to confirm that FreeBSD client with
>>>> Linux server is the only case where this problem occurs.
>>>
>>> Interesting.
>>>
>>>
>>>> Does this count as reproducing the issue with an "in tree" file system
>>>> contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
>>>> so far out of my depth that I'm pretty sure there are sea monsters
>>>> swimming around down there. So I can't rule out the possibility that
>>>> I've done something wrong either in setup or testing.
>>>>
>>>> During the course of this, I've gotten the reproduction down to
>>>> extracting a 2k tar file and then running "du" on the resulting
>>>> directory from the client. Doesn't matter if the file is untarred on
>>>> the FreeBSD client, the server, or another client. The tar file
>>>> contains a directory with a handful of random Javascript files from
>>>> Drupal. As far as I can tell, it has something to do with the number,
>>>> size, or names of the files. The Drupal project has three separate
>>>> directories all structured like this with the same filenames, but the
>>>> file contents vary. The issue occurs with all of them.
>>>>
>>>> The Linux /etc/exports file is just:
>>>>
>>>> /data 192.168.201.0/24(rw,sync)
>>>>
>>>> (The production case also uses crossmnt and no_subtree_check, anonuid,
>>>> and anongid, but I eliminated those one by one to make sure they
>>>> weren't responsible.)
>>>>
>>>> The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
>>>>
>>>> 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
>>>
>>> Out of curiosity, do you see the problem recur with nfsv3 or the other
>>> NFSv4 minor versions?
>>>
>>>
>>>> One additional thing I noticed that really blew my mind is that I can
>>>> shutdown both the client and the server, wait, power them back on, and
>>>> the issue is still there. So it's not something in RAM.  That prompted
>>>> me to try "touch x" in the directory to create a new 0-length file.
>>>> The issue then goes away. Then I can "rm x" and the issue comes back.
>>>> By contrast, I can write megabytes from /dev/random into one of the
>>>> files without affecting anything; the issue stays the same.
>>>>
>>>> I then tried it with all empty files using the same filenames. The
>>>> issue still occurred. Add or remove one file and the issue goes away.
>>>> I then renamed one of the files to zz.js. Issue still occurs. Renamed
>>>> it to zzz.js. Problem still occurs. Kept going until I got to
>>>> zzzzzz.js and it worked.
>>>>
>>>> Finally, I got it to the point where running this in an empty mounted
>>>> directory will create the issue:
>>>>
>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>> done; touch y0-xxxxxx.xx
>>>>
>>>> and this will not:
>>>>
>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>> done; touch y0-xxxxxxx.xx
>>>>
>>>> (The difference being one extra x in the last filename.)
>>>>
>>>> It works in the other direction as well. This causes the issue:
>>>>
>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>> done; touch y0-xxx.xx
>>>>
>>>> This does not:
>>>>
>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>> done; touch y0-xx.xx
>>>>
>>>> There's a four-character window involving the length of the filenames
>>>> where 62 files in a directory causes this issue. There's a little more
>>>> to it than that; it doesn't look like you can just create 61
>>>> two-letter filenames and then one really long one and get the issue.
>>>>
>>>> So I haven't found the specifics yet, but perhaps due to pure chance
>>>> this directory structure is exactly right to provoke an incredibly
>>>> obscure edge case?
>>>
>>> Well it's likely that this is a problem with READDIR, so file content
>>> is not going to be an issue. The file name lengths are the problem.
>>>
>>> Also, I'm wondering what the FreeBSD client's directory readdir
>>> arguments are (how much does it request, what are the maximum limits it
>>> negotiates, and so on). Rick?
>> As you'll see in the packet trace:
>> Sequence: cache this: No
>> Putfh: directory fh
>> Readdir:
>>      cookie: 0
>>      cookie_verf: 0
>>      dircount: 8706
>>      maxcount: 8706
>>      attr: type, RDattr_error, fileid, mounted_on_fleid
>> Getattr: same attributes as requested for a previous GETATTR, mainly
>>                to keep the directory's attribute cache up to date.
>>
>> The session negotiates a max request/reply size of just over 1Mbyte and a
>> maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
>>
>> If you are wondering where the 8706 comes from, it was an estimate of how
>> much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
>> by adding 512 to 8K.
>>
>> I have not yet had a chance to see if I can reproduce the problem with
>> J. David's
>> reproducer. I will try that soon, and if I can reproduce it, I will
>> poke at it to try and
>> figure out what is going on.
> Just fyi, I have reproduced it. Once you use J. David's little shell script to
> create the files in the directory, the Readdir RPC gets the junk reply
> to GETATTR
> (the count of words for the attribute bitmap in the reply is 0 instead of 2).
> You can unmount/remount it and still get the failure, assuming you do not
> mess with the directory contents.
> 
> Good work finding the reproducer, J. David!
> 
> I will start to poke around to see if I can figure out what the knfsd server is
> doing.
> 
> Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
> reproduce this, just in case you are inspired to cross over to the dark
> side and install FreeBSD somewhere.

I see the same malformed GETATTR result in the attachments.

Linux doesn't trip on this issue because it's NFS client doesn't ever
append a GETATTR operation after a READDIR.

So I've installed a small FreeBSD 14.2 guest, and copied the reproducer
script over to it. I see the extra GETATTR now, and I'm trying to
figure out what is causing the corrupted reply. At first glance, I
can see the problem involves a particularly placed page boundary in
the XDR encoding buffer, but it isn't a problem with GETATTR encoding
per se.


> I'll post when I have more info, rick
> 
>>
>> rick
>>
>>>
>>> Since this isn't reproducible (yet) with a Linux client, let's try
>>> another set of network captures, and you can send these to me
>>> privately.
>>>
>>> Start the capture
>>> Mount
>>> Run one of the reproducers above
>>> Unmount
>>> Stop the capture
>>>
>>> I'd like to see one with v6.13-rc3 and ext4 that works as expected, and
>>> one with the same configuration that fails.
>>>
>>> --
>>> Chuck Lever


-- 
Chuck Lever

Re: knfsd server bug when GETATTR follows READDIR

[Cedric Blancher]

On Sun, 22 Dec 2024 at 21:19, Chuck Lever <chuck.lever@oracle.com> wrote:
>
> On 12/21/24 6:53 PM, Rick Macklem wrote:
> > On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
> >>
> >> On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
> >>>
> >>> On 12/20/24 9:16 PM, J David wrote:
> >>>> Hello,
> >>>>
> >>>> On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> >>>>> If they can reproduce
> >>>>> this issue with an "in tree" file system contained in a recent upstream
> >>>>> Linux kernel, then we can take a look. (Or you and J. David can give it
> >>>>> a try).
> >>>>
> >>>> Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
> >>>> Debian backports on completely different hardware.
> >>>>
> >>>> Then I set up another NFS server on Arch (running kernel 6.12.4), and
> >>>> reproduced the issue there as well.
> >>>>
> >>>> Then, just to be sure, I went and found the instructions for building
> >>>> the Linux kernel from source, built and tested both 6.12.6 and
> >>>> 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
> >>>> occurs with those as well.
> >>>
> >>> Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
> >>>
> >>>
> >>>> Additionally, I have tested every combination of FreeBSD, Linux and
> >>>> OpenIndiana as client and server to confirm that FreeBSD client with
> >>>> Linux server is the only case where this problem occurs.
> >>>
> >>> Interesting.
> >>>
> >>>
> >>>> Does this count as reproducing the issue with an "in tree" file system
> >>>> contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
> >>>> so far out of my depth that I'm pretty sure there are sea monsters
> >>>> swimming around down there. So I can't rule out the possibility that
> >>>> I've done something wrong either in setup or testing.
> >>>>
> >>>> During the course of this, I've gotten the reproduction down to
> >>>> extracting a 2k tar file and then running "du" on the resulting
> >>>> directory from the client. Doesn't matter if the file is untarred on
> >>>> the FreeBSD client, the server, or another client. The tar file
> >>>> contains a directory with a handful of random Javascript files from
> >>>> Drupal. As far as I can tell, it has something to do with the number,
> >>>> size, or names of the files. The Drupal project has three separate
> >>>> directories all structured like this with the same filenames, but the
> >>>> file contents vary. The issue occurs with all of them.
> >>>>
> >>>> The Linux /etc/exports file is just:
> >>>>
> >>>> /data 192.168.201.0/24(rw,sync)
> >>>>
> >>>> (The production case also uses crossmnt and no_subtree_check, anonuid,
> >>>> and anongid, but I eliminated those one by one to make sure they
> >>>> weren't responsible.)
> >>>>
> >>>> The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
> >>>>
> >>>> 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
> >>>
> >>> Out of curiosity, do you see the problem recur with nfsv3 or the other
> >>> NFSv4 minor versions?
> >>>
> >>>
> >>>> One additional thing I noticed that really blew my mind is that I can
> >>>> shutdown both the client and the server, wait, power them back on, and
> >>>> the issue is still there. So it's not something in RAM.  That prompted
> >>>> me to try "touch x" in the directory to create a new 0-length file.
> >>>> The issue then goes away. Then I can "rm x" and the issue comes back.
> >>>> By contrast, I can write megabytes from /dev/random into one of the
> >>>> files without affecting anything; the issue stays the same.
> >>>>
> >>>> I then tried it with all empty files using the same filenames. The
> >>>> issue still occurred. Add or remove one file and the issue goes away.
> >>>> I then renamed one of the files to zz.js. Issue still occurs. Renamed
> >>>> it to zzz.js. Problem still occurs. Kept going until I got to
> >>>> zzzzzz.js and it worked.
> >>>>
> >>>> Finally, I got it to the point where running this in an empty mounted
> >>>> directory will create the issue:
> >>>>
> >>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> >>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> >>>> done; touch y0-xxxxxx.xx
> >>>>
> >>>> and this will not:
> >>>>
> >>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> >>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> >>>> done; touch y0-xxxxxxx.xx
> >>>>
> >>>> (The difference being one extra x in the last filename.)
> >>>>
> >>>> It works in the other direction as well. This causes the issue:
> >>>>
> >>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> >>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> >>>> done; touch y0-xxx.xx
> >>>>
> >>>> This does not:
> >>>>
> >>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
> >>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
> >>>> done; touch y0-xx.xx
> >>>>
> >>>> There's a four-character window involving the length of the filenames
> >>>> where 62 files in a directory causes this issue. There's a little more
> >>>> to it than that; it doesn't look like you can just create 61
> >>>> two-letter filenames and then one really long one and get the issue.
> >>>>
> >>>> So I haven't found the specifics yet, but perhaps due to pure chance
> >>>> this directory structure is exactly right to provoke an incredibly
> >>>> obscure edge case?
> >>>
> >>> Well it's likely that this is a problem with READDIR, so file content
> >>> is not going to be an issue. The file name lengths are the problem.
> >>>
> >>> Also, I'm wondering what the FreeBSD client's directory readdir
> >>> arguments are (how much does it request, what are the maximum limits it
> >>> negotiates, and so on). Rick?
> >> As you'll see in the packet trace:
> >> Sequence: cache this: No
> >> Putfh: directory fh
> >> Readdir:
> >>      cookie: 0
> >>      cookie_verf: 0
> >>      dircount: 8706
> >>      maxcount: 8706
> >>      attr: type, RDattr_error, fileid, mounted_on_fleid
> >> Getattr: same attributes as requested for a previous GETATTR, mainly
> >>                to keep the directory's attribute cache up to date.
> >>
> >> The session negotiates a max request/reply size of just over 1Mbyte and a
> >> maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
> >>
> >> If you are wondering where the 8706 comes from, it was an estimate of how
> >> much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
> >> by adding 512 to 8K.
> >>
> >> I have not yet had a chance to see if I can reproduce the problem with
> >> J. David's
> >> reproducer. I will try that soon, and if I can reproduce it, I will
> >> poke at it to try and
> >> figure out what is going on.
> > Just fyi, I have reproduced it. Once you use J. David's little shell script to
> > create the files in the directory, the Readdir RPC gets the junk reply
> > to GETATTR
> > (the count of words for the attribute bitmap in the reply is 0 instead of 2).
> > You can unmount/remount it and still get the failure, assuming you do not
> > mess with the directory contents.
> >
> > Good work finding the reproducer, J. David!
> >
> > I will start to poke around to see if I can figure out what the knfsd server is
> > doing.
> >
> > Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
> > reproduce this, just in case you are inspired to cross over to the dark
> > side and install FreeBSD somewhere.
>
> I see the same malformed GETATTR result in the attachments.
>
> Linux doesn't trip on this issue because it's NFS client doesn't ever
> append a GETATTR operation after a READDIR.

Windows ms-nfs41-client driver also does GETATTR after READDIR, and
trips over bogus return values on a regular basis. Solaris 11.4 nfsd
and nfs4j do not exhibit such problems.

Another garbage value that client gets from Linux nfsd is
FATTR4_WORD0_CHANGE, which sometimes returns absurdly high values.
Maybe add some WARN_ONCE() to Linux nfsd if unexpected crazy values
are sent over the wire?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: knfsd server bug when GETATTR follows READDIR

[Chuck Lever]

On 12/24/24 1:51 AM, Cedric Blancher wrote:
> On Sun, 22 Dec 2024 at 21:19, Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>> On 12/21/24 6:53 PM, Rick Macklem wrote:
>>> On Sat, Dec 21, 2024 at 3:27 PM Rick Macklem <rick.macklem@gmail.com> wrote:
>>>>
>>>> On Sat, Dec 21, 2024 at 9:34 AM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>
>>>>> On 12/20/24 9:16 PM, J David wrote:
>>>>>> Hello,
>>>>>>
>>>>>> On Tue, Dec 17, 2024 at 8:51 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>>> If they can reproduce
>>>>>>> this issue with an "in tree" file system contained in a recent upstream
>>>>>>> Linux kernel, then we can take a look. (Or you and J. David can give it
>>>>>>> a try).
>>>>>>
>>>>>> Yes, I reproduced this behavior on ext4 with 6.11.5+bpo-amd64 from
>>>>>> Debian backports on completely different hardware.
>>>>>>
>>>>>> Then I set up another NFS server on Arch (running kernel 6.12.4), and
>>>>>> reproduced the issue there as well.
>>>>>>
>>>>>> Then, just to be sure, I went and found the instructions for building
>>>>>> the Linux kernel from source, built and tested both 6.12.6 and
>>>>>> 6.13-rc3 as downloaded directly from www.kernel.org, and the issue
>>>>>> occurs with those as well.
>>>>>
>>>>> Reproducing on v6.13-rc with ext4 is all that was necessary, thank you!
>>>>>
>>>>>
>>>>>> Additionally, I have tested every combination of FreeBSD, Linux and
>>>>>> OpenIndiana as client and server to confirm that FreeBSD client with
>>>>>> Linux server is the only case where this problem occurs.
>>>>>
>>>>> Interesting.
>>>>>
>>>>>
>>>>>> Does this count as reproducing the issue with an "in tree" file system
>>>>>> contained in a recent upstream Linux kernel? I'm asking sincerely; I'm
>>>>>> so far out of my depth that I'm pretty sure there are sea monsters
>>>>>> swimming around down there. So I can't rule out the possibility that
>>>>>> I've done something wrong either in setup or testing.
>>>>>>
>>>>>> During the course of this, I've gotten the reproduction down to
>>>>>> extracting a 2k tar file and then running "du" on the resulting
>>>>>> directory from the client. Doesn't matter if the file is untarred on
>>>>>> the FreeBSD client, the server, or another client. The tar file
>>>>>> contains a directory with a handful of random Javascript files from
>>>>>> Drupal. As far as I can tell, it has something to do with the number,
>>>>>> size, or names of the files. The Drupal project has three separate
>>>>>> directories all structured like this with the same filenames, but the
>>>>>> file contents vary. The issue occurs with all of them.
>>>>>>
>>>>>> The Linux /etc/exports file is just:
>>>>>>
>>>>>> /data 192.168.201.0/24(rw,sync)
>>>>>>
>>>>>> (The production case also uses crossmnt and no_subtree_check, anonuid,
>>>>>> and anongid, but I eliminated those one by one to make sure they
>>>>>> weren't responsible.)
>>>>>>
>>>>>> The corresponding fstab entry on the FreeBSD 14.2-RELEASE client is:
>>>>>>
>>>>>> 192.168.201.200:/data /data nfs rw,tcp,nfsv4,minorversion=2 0 0
>>>>>
>>>>> Out of curiosity, do you see the problem recur with nfsv3 or the other
>>>>> NFSv4 minor versions?
>>>>>
>>>>>
>>>>>> One additional thing I noticed that really blew my mind is that I can
>>>>>> shutdown both the client and the server, wait, power them back on, and
>>>>>> the issue is still there. So it's not something in RAM.  That prompted
>>>>>> me to try "touch x" in the directory to create a new 0-length file.
>>>>>> The issue then goes away. Then I can "rm x" and the issue comes back.
>>>>>> By contrast, I can write megabytes from /dev/random into one of the
>>>>>> files without affecting anything; the issue stays the same.
>>>>>>
>>>>>> I then tried it with all empty files using the same filenames. The
>>>>>> issue still occurred. Add or remove one file and the issue goes away.
>>>>>> I then renamed one of the files to zz.js. Issue still occurs. Renamed
>>>>>> it to zzz.js. Problem still occurs. Kept going until I got to
>>>>>> zzzzzz.js and it worked.
>>>>>>
>>>>>> Finally, I got it to the point where running this in an empty mounted
>>>>>> directory will create the issue:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxxxxx.xx
>>>>>>
>>>>>> and this will not:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxxxxxx.xx
>>>>>>
>>>>>> (The difference being one extra x in the last filename.)
>>>>>>
>>>>>> It works in the other direction as well. This causes the issue:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xxx.xx
>>>>>>
>>>>>> This does not:
>>>>>>
>>>>>> rm *.xx; for a in a b c d e f g h ; do for b in 1 2 3 4 5 6 7 ; do
>>>>>> touch $a$b.xx ; done; done; for a in 1 2 3 4 5; do touch x$a-xx.xx;
>>>>>> done; touch y0-xx.xx
>>>>>>
>>>>>> There's a four-character window involving the length of the filenames
>>>>>> where 62 files in a directory causes this issue. There's a little more
>>>>>> to it than that; it doesn't look like you can just create 61
>>>>>> two-letter filenames and then one really long one and get the issue.
>>>>>>
>>>>>> So I haven't found the specifics yet, but perhaps due to pure chance
>>>>>> this directory structure is exactly right to provoke an incredibly
>>>>>> obscure edge case?
>>>>>
>>>>> Well it's likely that this is a problem with READDIR, so file content
>>>>> is not going to be an issue. The file name lengths are the problem.
>>>>>
>>>>> Also, I'm wondering what the FreeBSD client's directory readdir
>>>>> arguments are (how much does it request, what are the maximum limits it
>>>>> negotiates, and so on). Rick?
>>>> As you'll see in the packet trace:
>>>> Sequence: cache this: No
>>>> Putfh: directory fh
>>>> Readdir:
>>>>       cookie: 0
>>>>       cookie_verf: 0
>>>>       dircount: 8706
>>>>       maxcount: 8706
>>>>       attr: type, RDattr_error, fileid, mounted_on_fleid
>>>> Getattr: same attributes as requested for a previous GETATTR, mainly
>>>>                 to keep the directory's attribute cache up to date.
>>>>
>>>> The session negotiates a max request/reply size of just over 1Mbyte and a
>>>> maximum of something like 20 ops. (Can't recall, but definitely more than 4.)
>>>>
>>>> If you are wondering where the 8706 comes from, it was an estimate of how
>>>> much would be needed to fill an 8K buffer with the XDR translated to UFS dirents
>>>> by adding 512 to 8K.
>>>>
>>>> I have not yet had a chance to see if I can reproduce the problem with
>>>> J. David's
>>>> reproducer. I will try that soon, and if I can reproduce it, I will
>>>> poke at it to try and
>>>> figure out what is going on.
>>> Just fyi, I have reproduced it. Once you use J. David's little shell script to
>>> create the files in the directory, the Readdir RPC gets the junk reply
>>> to GETATTR
>>> (the count of words for the attribute bitmap in the reply is 0 instead of 2).
>>> You can unmount/remount it and still get the failure, assuming you do not
>>> mess with the directory contents.
>>>
>>> Good work finding the reproducer, J. David!
>>>
>>> I will start to poke around to see if I can figure out what the knfsd server is
>>> doing.
>>>
>>> Chuck, I suspect any fairly recent FreeBSD client will be sufficient to
>>> reproduce this, just in case you are inspired to cross over to the dark
>>> side and install FreeBSD somewhere.
>>
>> I see the same malformed GETATTR result in the attachments.
>>
>> Linux doesn't trip on this issue because it's NFS client doesn't ever
>> append a GETATTR operation after a READDIR.
> 
> Windows ms-nfs41-client driver also does GETATTR after READDIR, and
> trips over bogus return values on a regular basis. Solaris 11.4 nfsd
> and nfs4j do not exhibit such problems.
> 
> Another garbage value that client gets from Linux nfsd is
> FATTR4_WORD0_CHANGE, which sometimes returns absurdly high values.
> Maybe add some WARN_ONCE() to Linux nfsd if unexpected crazy values
> are sent over the wire?

I prefer that we address these bugs forthwith. If you have a reproducer
for these behaviors, please file a bug on bugzilla.kernel.org under the
filesystem/NFSD component.

We can't fix bugs that we don't know about...

-- 
Chuck Lever