[REGRESSION] reads and writes hang if NFS server has v3 disabled

[REGRESSION] reads and writes hang if NFS server has v3 disabled

[Benji Wiebe]

I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a 
directory with rw and mount it from my personal computer running Fedora 
45 as well.
If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state, 
and there's a continuous loop of V3 NULL calls, with the server replying 
that it only supports v4.
If nfs.conf has vers3=y, then the reads and writes succeed.
Introduced in 70b7e3526c53d9dd, still present in the latest 
mainline/master (841e384b841a).

The hung state can be rescued by changing the setting and restarting 
nfs-server, at which point the read/write will complete.

I can do more testing / get more information as needed.

Here's links to pcap files with it working/bugged:
http://grepper.net/lkml-files/nfs-working.pcap
http://grepper.net/lkml-files/nfs-bugged.pcap
http://grepper.net/lkml-files/nfs-bug-kernel.config

#regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1

--
Benji Wiebe

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Chuck Lever]

[ Adding correct addresses from MAINTAINERS ]

On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
> I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
> directory with rw and mount it from my personal computer running Fedora
> 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
> D+ state, and there's a continuous
>
>
> I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a 
> directory with rw and mount it from my personal computer running Fedora 
> 45 as well.
> If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state, 
> and there's a continuous loop of V3 NULL calls, with the server replying 
> that it only supports v4.
> If nfs.conf has vers3=y, then the reads and writes succeed.
> Introduced in 70b7e3526c53d9dd, still present in the latest 
> mainline/master (841e384b841a).
>
> The hung state can be rescued by changing the setting and restarting 
> nfs-server, at which point the read/write will complete.
>
> I can do more testing / get more information as needed.
>
> Here's links to pcap files with it working/bugged:
> http://grepper.net/lkml-files/nfs-working.pcap
> http://grepper.net/lkml-files/nfs-bugged.pcap
> http://grepper.net/lkml-files/nfs-bug-kernel.config
>
> #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1

It's not immediately obvious to me why 70b7e3526c53d9dd should
cause this kind of behavior regression.

Jeff, it's a netlink change. Can you triage?


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Jeff Layton]

On Fri, 2026-09-04 at 09:09 -0400, Chuck Lever wrote:
> [ Adding correct addresses from MAINTAINERS ]
> 
> On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
> > I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
> > directory with rw and mount it from my personal computer running Fedora
> > 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
> > D+ state, and there's a continuous
> > 
> > 
> > I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a 
> > directory with rw and mount it from my personal computer running Fedora 
> > 45 as well.
> > If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state, 
> > and there's a continuous loop of V3 NULL calls, with the server replying 
> > that it only supports v4.
> > If nfs.conf has vers3=y, then the reads and writes succeed.
> > Introduced in 70b7e3526c53d9dd, still present in the latest 
> > mainline/master (841e384b841a).
> > 
> > The hung state can be rescued by changing the setting and restarting 
> > nfs-server, at which point the read/write will complete.
> > 
> > I can do more testing / get more information as needed.
> > 
> > Here's links to pcap files with it working/bugged:
> > http://grepper.net/lkml-files/nfs-working.pcap
> > http://grepper.net/lkml-files/nfs-bugged.pcap
> > http://grepper.net/lkml-files/nfs-bug-kernel.config
> > 
> > #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1
> 
> It's not immediately obvious to me why 70b7e3526c53d9dd should
> cause this kind of behavior regression.
> 
> Jeff, it's a netlink change. Can you triage?
> 

Sure.

Looking at the captures, it looks like the server is handing out a
flexfiles v3 pNFS layout for the trivial server, and that's (of course)
failing in this situation.

So we have at least one server bug there -- it shouldn't advertise a
flexfiles v3 layout when it's not serving v3! I'll plan to fix that
soon.

That doesn't seem to have anything to do with commit 70b7e3526c53d9dd
though. I'm not sure why a bisect would have landed there.

What does the /etc/exports entry for this export look like? Disabling
pnfs should work around this for now too if you have it explicitly
enabled here.
-- 
Jeff Layton <jlayton@kernel.org>

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Chuck Lever]

On Fri, Sep 4, 2026, at 9:45 AM, Jeff Layton wrote:
> On Fri, 2026-09-04 at 09:09 -0400, Chuck Lever wrote:
>> [ Adding correct addresses from MAINTAINERS ]
>> 
>> On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
>> > I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
>> > directory with rw and mount it from my personal computer running Fedora
>> > 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
>> > D+ state, and there's a continuous
>> > 
>> > 
>> > I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a 
>> > directory with rw and mount it from my personal computer running Fedora 
>> > 45 as well.
>> > If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state, 
>> > and there's a continuous loop of V3 NULL calls, with the server replying 
>> > that it only supports v4.
>> > If nfs.conf has vers3=y, then the reads and writes succeed.
>> > Introduced in 70b7e3526c53d9dd, still present in the latest 
>> > mainline/master (841e384b841a).
>> > 
>> > The hung state can be rescued by changing the setting and restarting 
>> > nfs-server, at which point the read/write will complete.
>> > 
>> > I can do more testing / get more information as needed.
>> > 
>> > Here's links to pcap files with it working/bugged:
>> > http://grepper.net/lkml-files/nfs-working.pcap
>> > http://grepper.net/lkml-files/nfs-bugged.pcap
>> > http://grepper.net/lkml-files/nfs-bug-kernel.config
>> > 
>> > #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1
>> 
>> It's not immediately obvious to me why 70b7e3526c53d9dd should
>> cause this kind of behavior regression.
>> 
>> Jeff, it's a netlink change. Can you triage?
>> 
>
> Sure.
>
> Looking at the captures, it looks like the server is handing out a
> flexfiles v3 pNFS layout for the trivial server, and that's (of course)
> failing in this situation.
>
> So we have at least one server bug there -- it shouldn't advertise a
> flexfiles v3 layout when it's not serving v3! I'll plan to fix that
> soon.
>
> That doesn't seem to have anything to do with commit 70b7e3526c53d9dd
> though. I'm not sure why a bisect would have landed there.
>
> What does the /etc/exports entry for this export look like? Disabling
> pnfs should work around this for now too if you have it explicitly
> enabled here.

Could be:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=nfsd-testing&id=89225ab5f186ed80b7044cdf3b50902f4c1ee024

-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Jeff Layton]

On Fri, 2026-09-04 at 10:15 -0400, Chuck Lever wrote:
> 
> On Fri, Sep 4, 2026, at 9:45 AM, Jeff Layton wrote:
> > On Fri, 2026-09-04 at 09:09 -0400, Chuck Lever wrote:
> > > [ Adding correct addresses from MAINTAINERS ]
> > > 
> > > On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
> > > > I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
> > > > directory with rw and mount it from my personal computer running Fedora
> > > > 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
> > > > D+ state, and there's a continuous
> > > > 
> > > > 
> > > > I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a 
> > > > directory with rw and mount it from my personal computer running Fedora 
> > > > 45 as well.
> > > > If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state, 
> > > > and there's a continuous loop of V3 NULL calls, with the server replying 
> > > > that it only supports v4.
> > > > If nfs.conf has vers3=y, then the reads and writes succeed.
> > > > Introduced in 70b7e3526c53d9dd, still present in the latest 
> > > > mainline/master (841e384b841a).
> > > > 
> > > > The hung state can be rescued by changing the setting and restarting 
> > > > nfs-server, at which point the read/write will complete.
> > > > 
> > > > I can do more testing / get more information as needed.
> > > > 
> > > > Here's links to pcap files with it working/bugged:
> > > > http://grepper.net/lkml-files/nfs-working.pcap
> > > > http://grepper.net/lkml-files/nfs-bugged.pcap
> > > > http://grepper.net/lkml-files/nfs-bug-kernel.config
> > > > 
> > > > #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1
> > > 
> > > It's not immediately obvious to me why 70b7e3526c53d9dd should
> > > cause this kind of behavior regression.
> > > 
> > > Jeff, it's a netlink change. Can you triage?
> > > 
> > 
> > Sure.
> > 
> > Looking at the captures, it looks like the server is handing out a
> > flexfiles v3 pNFS layout for the trivial server, and that's (of course)
> > failing in this situation.
> > 
> > So we have at least one server bug there -- it shouldn't advertise a
> > flexfiles v3 layout when it's not serving v3! I'll plan to fix that
> > soon.
> > 
> > That doesn't seem to have anything to do with commit 70b7e3526c53d9dd
> > though. I'm not sure why a bisect would have landed there.
> > 
> > What does the /etc/exports entry for this export look like? Disabling
> > pnfs should work around this for now too if you have it explicitly
> > enabled here.
> 
> Could be:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=nfsd-testing&id=89225ab5f186ed80b7044cdf3b50902f4c1ee024

Yep, that's it. That patch didn't make it in for some reason, so pnfs
is always enabled without it. We'll need that to go to stable soon.

I'll work on how to go about disabling flexfiles layouts when v3 isn't
enabled.
-- 
Jeff Layton <jlayton@kernel.org>

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Benji Wiebe]

On 9/4/26 8:45 AM, Jeff Layton wrote:
> On Fri, 2026-09-04 at 09:09 -0400, Chuck Lever wrote:
>> [ Adding correct addresses from MAINTAINERS ]
>>
>> On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
>>> I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
>>> directory with rw and mount it from my personal computer running Fedora
>>> 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
>>> D+ state, and there's a continuous
>>>
>>>
>>> I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a
>>> directory with rw and mount it from my personal computer running Fedora
>>> 45 as well.
>>> If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state,
>>> and there's a continuous loop of V3 NULL calls, with the server replying
>>> that it only supports v4.
>>> If nfs.conf has vers3=y, then the reads and writes succeed.
>>> Introduced in 70b7e3526c53d9dd, still present in the latest
>>> mainline/master (841e384b841a).
>>>
>>> The hung state can be rescued by changing the setting and restarting
>>> nfs-server, at which point the read/write will complete.
>>>
>>> I can do more testing / get more information as needed.
>>>
>>> Here's links to pcap files with it working/bugged:
>>> http://grepper.net/lkml-files/nfs-working.pcap
>>> http://grepper.net/lkml-files/nfs-bugged.pcap
>>> http://grepper.net/lkml-files/nfs-bug-kernel.config
>>>
>>> #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1
>>
>> It's not immediately obvious to me why 70b7e3526c53d9dd should
>> cause this kind of behavior regression.
>>
>> Jeff, it's a netlink change. Can you triage?
>>
> 
> Sure.
> 
> Looking at the captures, it looks like the server is handing out a
> flexfiles v3 pNFS layout for the trivial server, and that's (of course)
> failing in this situation.
> 
> So we have at least one server bug there -- it shouldn't advertise a
> flexfiles v3 layout when it's not serving v3! I'll plan to fix that
> soon.
> 
> That doesn't seem to have anything to do with commit 70b7e3526c53d9dd
> though. I'm not sure why a bisect would have landed there.
I checked and rechecked several times (and again now!) and 
70b7e3526c53d9dd always had the bug present and 565ee23ea2374 did not 
have the bug present.>
> What does the /etc/exports entry for this export look like? Disabling
> pnfs should work around this for now too if you have it explicitly
> enabled here.

The /etc/exports entry I'm using:
/home/export *(rw,no_root_squash)

I'm new to the kernel development process; is there a patch I can apply 
now to see if it fixes the bug?

Re: [REGRESSION] reads and writes hang if NFS server has v3 disabled

[Jeff Layton]

On Mon, 2026-09-07 at 13:57 -0500, Benji Wiebe wrote:
> On 9/4/26 8:45 AM, Jeff Layton wrote:
> > On Fri, 2026-09-04 at 09:09 -0400, Chuck Lever wrote:
> > > [ Adding correct addresses from MAINTAINERS ]
> > > 
> > > On Fri, Sep 4, 2026, at 9:05 AM, Chuck Lever wrote:
> > > > I'm using a minimal Fedora 45 VM (kvm) for testing.   I export a
> > > > directory with rw and mount it from my personal computer running Fedora
> > > > 45 as well. If /etc/nfs. conf has vers3=n, then reads and writes hang in
> > > > D+ state, and there's a continuous
> > > > 
> > > > 
> > > > I'm using a minimal Fedora 45 VM (kvm) for testing.  I export a
> > > > directory with rw and mount it from my personal computer running Fedora
> > > > 45 as well.
> > > > If /etc/nfs.conf has vers3=n, then reads and writes hang in D+ state,
> > > > and there's a continuous loop of V3 NULL calls, with the server replying
> > > > that it only supports v4.
> > > > If nfs.conf has vers3=y, then the reads and writes succeed.
> > > > Introduced in 70b7e3526c53d9dd, still present in the latest
> > > > mainline/master (841e384b841a).
> > > > 
> > > > The hung state can be rescued by changing the setting and restarting
> > > > nfs-server, at which point the read/write will complete.
> > > > 
> > > > I can do more testing / get more information as needed.
> > > > 
> > > > Here's links to pcap files with it working/bugged:
> > > > http://grepper.net/lkml-files/nfs-working.pcap
> > > > http://grepper.net/lkml-files/nfs-bugged.pcap
> > > > http://grepper.net/lkml-files/nfs-bug-kernel.config
> > > > 
> > > > #regzbot introduced:  70b7e3526c53d9dd7caccdbeff5b0485640d8cf1
> > > 
> > > It's not immediately obvious to me why 70b7e3526c53d9dd should
> > > cause this kind of behavior regression.
> > > 
> > > Jeff, it's a netlink change. Can you triage?
> > > 
> > 
> > Sure.
> > 
> > Looking at the captures, it looks like the server is handing out a
> > flexfiles v3 pNFS layout for the trivial server, and that's (of course)
> > failing in this situation.
> > 
> > So we have at least one server bug there -- it shouldn't advertise a
> > flexfiles v3 layout when it's not serving v3! I'll plan to fix that
> > soon.
> > 
> > That doesn't seem to have anything to do with commit 70b7e3526c53d9dd
> > though. I'm not sure why a bisect would have landed there.
> I checked and rechecked several times (and again now!) and 
> 70b7e3526c53d9dd always had the bug present and 565ee23ea2374 did not 
> have the bug present.>

The bug is in 565ee23ea2374, but until 70b7e3526c53d9dd is applied, the
code it's in is not used, so that explains why you're seeing it there.
 
> > What does the /etc/exports entry for this export look like? Disabling
> > pnfs should work around this for now too if you have it explicitly
> > enabled here.
> 
> The /etc/exports entry I'm using:
> /home/export *(rw,no_root_squash)
> 
> I'm new to the kernel development process; is there a patch I can apply 
> now to see if it fixes the bug?

You want commit f76017a7663c4ce5e379f8a8d39f032bdb1fd865 from Chuck's
nfsd-fixes branch. Basically, what's happening is that pnfs is being
enabled even though you haven't requested it:

https://lore.kernel.org/linux-nfs/20260414-pnfs-exp-fix-v1-1-9face14c16c2@kernel.org/

-- 
Jeff Layton <jlayton@kernel.org>