Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Jon Hunter]

Hi Greg,

On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
> This is the start of the stable review cycle for the 5.15.160 release.
> There are 23 patches in this series, all will be posted as a response
> to this one.  If anyone has any issues with these being applied, please
> let me know.
> 
> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
> Anything received after that time might be too late.
> 
> The whole patch series can be found in one patch at:
> 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
> or in the git tree and branch at:
> 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
> and the diffstat can be found below.
> 
> thanks,
> 
> greg k-h
> 
> -------------
> Pseudo-Shortlog of commits:

...

> NeilBrown <neilb@suse.de>
>      nfsd: don't allow nfsd threads to be signalled.


I am seeing a suspend regression on a couple boards and bisect is 
pointing to the above commit. Reverting this commit does fix the issue.

Test results for stable-v5.15:
     10 builds:	10 pass, 0 fail
     26 boots:	26 pass, 0 fail
     102 tests:	100 pass, 2 fail

Linux version:	5.15.160-rc1-g7c7f29d3b2af
Boards tested:	tegra124-jetson-tk1, tegra186-p2771-0000,
                 tegra194-p2972-0000, tegra194-p3509-0000+p3668-0000,
                 tegra20-ventana, tegra210-p2371-2180,
                 tegra210-p3450-0000, tegra30-cardhu-a04

Test failures:	tegra186-p2771-0000: pm-system-suspend.sh
                 tegra194-p2972-0000: pm-system-suspend.sh

Jon

-- 
nvpublic

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Greg Kroah-Hartman]

On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
> Hi Greg,
> 
> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
> > This is the start of the stable review cycle for the 5.15.160 release.
> > There are 23 patches in this series, all will be posted as a response
> > to this one.  If anyone has any issues with these being applied, please
> > let me know.
> > 
> > Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
> > Anything received after that time might be too late.
> > 
> > The whole patch series can be found in one patch at:
> > 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
> > or in the git tree and branch at:
> > 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
> > and the diffstat can be found below.
> > 
> > thanks,
> > 
> > greg k-h
> > 
> > -------------
> > Pseudo-Shortlog of commits:
> 
> ...
> 
> > NeilBrown <neilb@suse.de>
> >      nfsd: don't allow nfsd threads to be signalled.
> 
> 
> I am seeing a suspend regression on a couple boards and bisect is pointing
> to the above commit. Reverting this commit does fix the issue.

Ugh, that fixes the report from others.  Can you cc: everyone on that
and figure out what is going on, as this keeps going back and forth...

thanks,

greg k-h

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Jon Hunter]

On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>> Hi Greg,
>>
>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>> This is the start of the stable review cycle for the 5.15.160 release.
>>> There are 23 patches in this series, all will be posted as a response
>>> to this one.  If anyone has any issues with these being applied, please
>>> let me know.
>>>
>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>> Anything received after that time might be too late.
>>>
>>> The whole patch series can be found in one patch at:
>>> 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>> or in the git tree and branch at:
>>> 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>> and the diffstat can be found below.
>>>
>>> thanks,
>>>
>>> greg k-h
>>>
>>> -------------
>>> Pseudo-Shortlog of commits:
>>
>> ...
>>
>>> NeilBrown <neilb@suse.de>
>>>       nfsd: don't allow nfsd threads to be signalled.
>>
>>
>> I am seeing a suspend regression on a couple boards and bisect is pointing
>> to the above commit. Reverting this commit does fix the issue.
> 
> Ugh, that fixes the report from others.  Can you cc: everyone on that
> and figure out what is going on, as this keeps going back and forth...


Adding Chuck, Neil and Chris from the bug report here [0].

With the above applied to v5.15.y, I am seeing suspend on 2 of our 
boards fail. These boards are using NFS and on entry to suspend I am now 
seeing ...

  Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
  freeze, wq_busy=0):

The boards appear to hang at that point. So may be something else missing?

Jon

[0] 
https://lore.kernel.org/lkml/b363e394-7549-4b9e-b71b-d97cd13f9607@alliedtelesis.co.nz/

-- 
nvpublic

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever III]

> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> 
> 
> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>> Hi Greg,
>>> 
>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>> This is the start of the stable review cycle for the 5.15.160 release.
>>>> There are 23 patches in this series, all will be posted as a response
>>>> to this one.  If anyone has any issues with these being applied, please
>>>> let me know.
>>>> 
>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>> Anything received after that time might be too late.
>>>> 
>>>> The whole patch series can be found in one patch at:
>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>>> or in the git tree and branch at:
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>> and the diffstat can be found below.
>>>> 
>>>> thanks,
>>>> 
>>>> greg k-h
>>>> 
>>>> -------------
>>>> Pseudo-Shortlog of commits:
>>> 
>>> ...
>>> 
>>>> NeilBrown <neilb@suse.de>
>>>>      nfsd: don't allow nfsd threads to be signalled.
>>> 
>>> 
>>> I am seeing a suspend regression on a couple boards and bisect is pointing
>>> to the above commit. Reverting this commit does fix the issue.
>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>> and figure out what is going on, as this keeps going back and forth...
> 
> 
> Adding Chuck, Neil and Chris from the bug report here [0].
> 
> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
> 
> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
> freeze, wq_busy=0):
> 
> The boards appear to hang at that point. So may be something else missing?

Note that we don't have access to hardware like this, so
we haven't tested that patch (even the upstream version)
with suspend on that hardware.

So, it could be something missing, or it could be that
patch has a problem.

It would help us to know if you observe the same issue
with an upstream kernel, if that is possible.


> Jon
> 
> [0] https://lore.kernel.org/lkml/b363e394-7549-4b9e-b71b-d97cd13f9607@alliedtelesis.co.nz/
> 
> -- 
> nvpublic

--
Chuck Lever

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Jon Hunter]

On 28/05/2024 14:14, Chuck Lever III wrote:
> 
> 
>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>
>>
>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>>> Hi Greg,
>>>>
>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>>> This is the start of the stable review cycle for the 5.15.160 release.
>>>>> There are 23 patches in this series, all will be posted as a response
>>>>> to this one.  If anyone has any issues with these being applied, please
>>>>> let me know.
>>>>>
>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>>> Anything received after that time might be too late.
>>>>>
>>>>> The whole patch series can be found in one patch at:
>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>>>> or in the git tree and branch at:
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>>> and the diffstat can be found below.
>>>>>
>>>>> thanks,
>>>>>
>>>>> greg k-h
>>>>>
>>>>> -------------
>>>>> Pseudo-Shortlog of commits:
>>>>
>>>> ...
>>>>
>>>>> NeilBrown <neilb@suse.de>
>>>>>       nfsd: don't allow nfsd threads to be signalled.
>>>>
>>>>
>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
>>>> to the above commit. Reverting this commit does fix the issue.
>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>>> and figure out what is going on, as this keeps going back and forth...
>>
>>
>> Adding Chuck, Neil and Chris from the bug report here [0].
>>
>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
>>
>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
>> freeze, wq_busy=0):
>>
>> The boards appear to hang at that point. So may be something else missing?
> 
> Note that we don't have access to hardware like this, so
> we haven't tested that patch (even the upstream version)
> with suspend on that hardware.


No problem, I would not expect you to have this particular hardware :-)

> So, it could be something missing, or it could be that
> patch has a problem.
> 
> It would help us to know if you observe the same issue
> with an upstream kernel, if that is possible.


I don't observe this with either mainline, -next or any other stable 
branch. So that would suggest that something else is missing from 
linux-5.15.y.

Jon

-- 
nvpublic

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chris Packham]

On 29/05/24 02:18, Jon Hunter wrote:
>
> On 28/05/2024 14:14, Chuck Lever III wrote:
>>
>>
>>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>>
>>>
>>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>>>> Hi Greg,
>>>>>
>>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>>>> This is the start of the stable review cycle for the 5.15.160 
>>>>>> release.
>>>>>> There are 23 patches in this series, all will be posted as a 
>>>>>> response
>>>>>> to this one.  If anyone has any issues with these being applied, 
>>>>>> please
>>>>>> let me know.
>>>>>>
>>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>>>> Anything received after that time might be too late.
>>>>>>
>>>>>> The whole patch series can be found in one patch at:
>>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz 
>>>>>>
>>>>>> or in the git tree and branch at:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git 
>>>>>> linux-5.15.y
>>>>>> and the diffstat can be found below.
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> greg k-h
>>>>>>
>>>>>> -------------
>>>>>> Pseudo-Shortlog of commits:
>>>>>
>>>>> ...
>>>>>
>>>>>> NeilBrown <neilb@suse.de>
>>>>>>       nfsd: don't allow nfsd threads to be signalled.
>>>>>
>>>>>
>>>>> I am seeing a suspend regression on a couple boards and bisect is 
>>>>> pointing
>>>>> to the above commit. Reverting this commit does fix the issue.
>>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>>>> and figure out what is going on, as this keeps going back and forth...
>>>
>>>
>>> Adding Chuck, Neil and Chris from the bug report here [0].
>>>
>>> With the above applied to v5.15.y, I am seeing suspend on 2 of our 
>>> boards fail. These boards are using NFS and on entry to suspend I am 
>>> now seeing ...
>>>
>>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
>>> freeze, wq_busy=0):
>>>
>>> The boards appear to hang at that point. So may be something else 
>>> missing?
>>
>> Note that we don't have access to hardware like this, so
>> we haven't tested that patch (even the upstream version)
>> with suspend on that hardware.
>
>
> No problem, I would not expect you to have this particular hardware :-)
>
It's probably not much help but for the oops I originally reported we've 
been carrying "nfsd: don't allow nfsd threads to be signalled" locally 
and that resolved it. Now that we've updated to 5.15.160 and dropped the 
local patch it's still resolved for us. Our systems don't make use of 
suspend so they won't hit any issue related to that.
>> So, it could be something missing, or it could be that
>> patch has a problem.
>>
>> It would help us to know if you observe the same issue
>> with an upstream kernel, if that is possible.
>
>
> I don't observe this with either mainline, -next or any other stable 
> branch. So that would suggest that something else is missing from 
> linux-5.15.y.
>
> Jon
>

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever III]

> On May 28, 2024, at 10:18 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> 
> 
> On 28/05/2024 14:14, Chuck Lever III wrote:
>>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>> 
>>> 
>>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>>>> Hi Greg,
>>>>> 
>>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>>>> This is the start of the stable review cycle for the 5.15.160 release.
>>>>>> There are 23 patches in this series, all will be posted as a response
>>>>>> to this one.  If anyone has any issues with these being applied, please
>>>>>> let me know.
>>>>>> 
>>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>>>> Anything received after that time might be too late.
>>>>>> 
>>>>>> The whole patch series can be found in one patch at:
>>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>>>>> or in the git tree and branch at:
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>>>> and the diffstat can be found below.
>>>>>> 
>>>>>> thanks,
>>>>>> 
>>>>>> greg k-h
>>>>>> 
>>>>>> -------------
>>>>>> Pseudo-Shortlog of commits:
>>>>> 
>>>>> ...
>>>>> 
>>>>>> NeilBrown <neilb@suse.de>
>>>>>>      nfsd: don't allow nfsd threads to be signalled.
>>>>> 
>>>>> 
>>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
>>>>> to the above commit. Reverting this commit does fix the issue.
>>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>>>> and figure out what is going on, as this keeps going back and forth...
>>> 
>>> 
>>> Adding Chuck, Neil and Chris from the bug report here [0].
>>> 
>>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
>>> 
>>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
>>> freeze, wq_busy=0):
>>> 
>>> The boards appear to hang at that point. So may be something else missing?
>> Note that we don't have access to hardware like this, so
>> we haven't tested that patch (even the upstream version)
>> with suspend on that hardware.
> 
> 
> No problem, I would not expect you to have this particular hardware :-)
> 
>> So, it could be something missing, or it could be that
>> patch has a problem.
>> It would help us to know if you observe the same issue
>> with an upstream kernel, if that is possible.
> 
> 
> I don't observe this with either mainline, -next or any other stable branch. So that would suggest that something else is missing from linux-5.15.y.

That helps. It would be very helpful to have a reproducer I can
use to confirm we have a fix. I'm sure this will be a process
that involves a non-trivial number of iterations.


--
Chuck Lever

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[NeilBrown]

On Wed, 29 May 2024, Chuck Lever III wrote:
> 
> 
> > On May 28, 2024, at 10:18 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> > 
> > 
> > On 28/05/2024 14:14, Chuck Lever III wrote:
> >>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> >>> 
> >>> 
> >>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
> >>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
> >>>>> Hi Greg,
> >>>>> 
> >>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
> >>>>>> This is the start of the stable review cycle for the 5.15.160 release.
> >>>>>> There are 23 patches in this series, all will be posted as a response
> >>>>>> to this one.  If anyone has any issues with these being applied, please
> >>>>>> let me know.
> >>>>>> 
> >>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
> >>>>>> Anything received after that time might be too late.
> >>>>>> 
> >>>>>> The whole patch series can be found in one patch at:
> >>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
> >>>>>> or in the git tree and branch at:
> >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
> >>>>>> and the diffstat can be found below.
> >>>>>> 
> >>>>>> thanks,
> >>>>>> 
> >>>>>> greg k-h
> >>>>>> 
> >>>>>> -------------
> >>>>>> Pseudo-Shortlog of commits:
> >>>>> 
> >>>>> ...
> >>>>> 
> >>>>>> NeilBrown <neilb@suse.de>
> >>>>>>      nfsd: don't allow nfsd threads to be signalled.
> >>>>> 
> >>>>> 
> >>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
> >>>>> to the above commit. Reverting this commit does fix the issue.
> >>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
> >>>> and figure out what is going on, as this keeps going back and forth...
> >>> 
> >>> 
> >>> Adding Chuck, Neil and Chris from the bug report here [0].
> >>> 
> >>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
> >>> 
> >>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
> >>> freeze, wq_busy=0):
> >>> 
> >>> The boards appear to hang at that point. So may be something else missing?
> >> Note that we don't have access to hardware like this, so
> >> we haven't tested that patch (even the upstream version)
> >> with suspend on that hardware.
> > 
> > 
> > No problem, I would not expect you to have this particular hardware :-)
> > 
> >> So, it could be something missing, or it could be that
> >> patch has a problem.
> >> It would help us to know if you observe the same issue
> >> with an upstream kernel, if that is possible.
> > 
> > 
> > I don't observe this with either mainline, -next or any other stable branch. So that would suggest that something else is missing from linux-5.15.y.
> 
> That helps. It would be very helpful to have a reproducer I can
> use to confirm we have a fix. I'm sure this will be a process
> that involves a non-trivial number of iterations.

Missing upstream patch is

Commit 9bd4161c5917 ("SUNRPC: change service idle list to be an llist")

This contains some freezer-related changes which probably should
have been a separate patch.

We probably just need to add "| TASK_FREEZABLE" in one or two places.
I'll post a patch for testing in a little while.

NeilBrown

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever III]

> On May 28, 2024, at 6:01 PM, NeilBrown <neilb@suse.de> wrote:
> 
> On Wed, 29 May 2024, Chuck Lever III wrote:
>> 
>> 
>>> On May 28, 2024, at 10:18 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>> 
>>> 
>>> On 28/05/2024 14:14, Chuck Lever III wrote:
>>>>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>>>> 
>>>>> 
>>>>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>>>>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>>>>>> Hi Greg,
>>>>>>> 
>>>>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>>>>>> This is the start of the stable review cycle for the 5.15.160 release.
>>>>>>>> There are 23 patches in this series, all will be posted as a response
>>>>>>>> to this one.  If anyone has any issues with these being applied, please
>>>>>>>> let me know.
>>>>>>>> 
>>>>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>>>>>> Anything received after that time might be too late.
>>>>>>>> 
>>>>>>>> The whole patch series can be found in one patch at:
>>>>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>>>>>>> or in the git tree and branch at:
>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>>>>>> and the diffstat can be found below.
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> 
>>>>>>>> greg k-h
>>>>>>>> 
>>>>>>>> -------------
>>>>>>>> Pseudo-Shortlog of commits:
>>>>>>> 
>>>>>>> ...
>>>>>>> 
>>>>>>>> NeilBrown <neilb@suse.de>
>>>>>>>>     nfsd: don't allow nfsd threads to be signalled.
>>>>>>> 
>>>>>>> 
>>>>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
>>>>>>> to the above commit. Reverting this commit does fix the issue.
>>>>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>>>>>> and figure out what is going on, as this keeps going back and forth...
>>>>> 
>>>>> 
>>>>> Adding Chuck, Neil and Chris from the bug report here [0].
>>>>> 
>>>>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
>>>>> 
>>>>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
>>>>> freeze, wq_busy=0):
>>>>> 
>>>>> The boards appear to hang at that point. So may be something else missing?
>>>> Note that we don't have access to hardware like this, so
>>>> we haven't tested that patch (even the upstream version)
>>>> with suspend on that hardware.
>>> 
>>> 
>>> No problem, I would not expect you to have this particular hardware :-)
>>> 
>>>> So, it could be something missing, or it could be that
>>>> patch has a problem.
>>>> It would help us to know if you observe the same issue
>>>> with an upstream kernel, if that is possible.
>>> 
>>> 
>>> I don't observe this with either mainline, -next or any other stable branch. So that would suggest that something else is missing from linux-5.15.y.
>> 
>> That helps. It would be very helpful to have a reproducer I can
>> use to confirm we have a fix. I'm sure this will be a process
>> that involves a non-trivial number of iterations.
> 
> Missing upstream patch is
> 
> Commit 9bd4161c5917 ("SUNRPC: change service idle list to be an llist")
> 
> This contains some freezer-related changes which probably should
> have been a separate patch.

Thanks for tracking that down.


> We probably just need to add "| TASK_FREEZABLE" in one or two places.
> I'll post a patch for testing in a little while.

My understanding is that the stable maintainers prefer a backport
of a patch (or patches) that are already applied to Linus' tree.


--
Chuck Lever

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[NeilBrown]

On Wed, 29 May 2024, Chuck Lever III wrote:
> 
> 
> > On May 28, 2024, at 6:01 PM, NeilBrown <neilb@suse.de> wrote:
> > 
> > On Wed, 29 May 2024, Chuck Lever III wrote:
> >> 
> >> 
> >>> On May 28, 2024, at 10:18 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> >>> 
> >>> 
> >>> On 28/05/2024 14:14, Chuck Lever III wrote:
> >>>>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
> >>>>> 
> >>>>> 
> >>>>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
> >>>>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
> >>>>>>> Hi Greg,
> >>>>>>> 
> >>>>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
> >>>>>>>> This is the start of the stable review cycle for the 5.15.160 release.
> >>>>>>>> There are 23 patches in this series, all will be posted as a response
> >>>>>>>> to this one.  If anyone has any issues with these being applied, please
> >>>>>>>> let me know.
> >>>>>>>> 
> >>>>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
> >>>>>>>> Anything received after that time might be too late.
> >>>>>>>> 
> >>>>>>>> The whole patch series can be found in one patch at:
> >>>>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
> >>>>>>>> or in the git tree and branch at:
> >>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
> >>>>>>>> and the diffstat can be found below.
> >>>>>>>> 
> >>>>>>>> thanks,
> >>>>>>>> 
> >>>>>>>> greg k-h
> >>>>>>>> 
> >>>>>>>> -------------
> >>>>>>>> Pseudo-Shortlog of commits:
> >>>>>>> 
> >>>>>>> ...
> >>>>>>> 
> >>>>>>>> NeilBrown <neilb@suse.de>
> >>>>>>>>     nfsd: don't allow nfsd threads to be signalled.
> >>>>>>> 
> >>>>>>> 
> >>>>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
> >>>>>>> to the above commit. Reverting this commit does fix the issue.
> >>>>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
> >>>>>> and figure out what is going on, as this keeps going back and forth...
> >>>>> 
> >>>>> 
> >>>>> Adding Chuck, Neil and Chris from the bug report here [0].
> >>>>> 
> >>>>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
> >>>>> 
> >>>>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
> >>>>> freeze, wq_busy=0):
> >>>>> 
> >>>>> The boards appear to hang at that point. So may be something else missing?
> >>>> Note that we don't have access to hardware like this, so
> >>>> we haven't tested that patch (even the upstream version)
> >>>> with suspend on that hardware.
> >>> 
> >>> 
> >>> No problem, I would not expect you to have this particular hardware :-)
> >>> 
> >>>> So, it could be something missing, or it could be that
> >>>> patch has a problem.
> >>>> It would help us to know if you observe the same issue
> >>>> with an upstream kernel, if that is possible.
> >>> 
> >>> 
> >>> I don't observe this with either mainline, -next or any other stable branch. So that would suggest that something else is missing from linux-5.15.y.
> >> 
> >> That helps. It would be very helpful to have a reproducer I can
> >> use to confirm we have a fix. I'm sure this will be a process
> >> that involves a non-trivial number of iterations.
> > 
> > Missing upstream patch is
> > 
> > Commit 9bd4161c5917 ("SUNRPC: change service idle list to be an llist")
> > 
> > This contains some freezer-related changes which probably should
> > have been a separate patch.
> 
> Thanks for tracking that down.
> 
> 
> > We probably just need to add "| TASK_FREEZABLE" in one or two places.
> > I'll post a patch for testing in a little while.
> 
> My understanding is that the stable maintainers prefer a backport
> of a patch (or patches) that are already applied to Linus' tree.

They also preferred a full backport of fs/nfsd/..  That hasn't gone so
well :-)

In this case we would need 

Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")

to get TASK_FREEZABLE.
I doubt that would be a good choice.

NeilBrown


> 
> 
> --
> Chuck Lever
> 
> 
> 

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever III]

> On May 28, 2024, at 7:44 PM, NeilBrown <neilb@suse.de> wrote:
> 
> On Wed, 29 May 2024, Chuck Lever III wrote:
>> 
>> 
>>> On May 28, 2024, at 6:01 PM, NeilBrown <neilb@suse.de> wrote:
>>> 
>>> On Wed, 29 May 2024, Chuck Lever III wrote:
>>>> 
>>>> 
>>>>> On May 28, 2024, at 10:18 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>>>> 
>>>>> 
>>>>> On 28/05/2024 14:14, Chuck Lever III wrote:
>>>>>>> On May 28, 2024, at 5:04 AM, Jon Hunter <jonathanh@nvidia.com> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On 25/05/2024 15:20, Greg Kroah-Hartman wrote:
>>>>>>>> On Sat, May 25, 2024 at 12:13:28AM +0100, Jon Hunter wrote:
>>>>>>>>> Hi Greg,
>>>>>>>>> 
>>>>>>>>> On 23/05/2024 14:12, Greg Kroah-Hartman wrote:
>>>>>>>>>> This is the start of the stable review cycle for the 5.15.160 release.
>>>>>>>>>> There are 23 patches in this series, all will be posted as a response
>>>>>>>>>> to this one.  If anyone has any issues with these being applied, please
>>>>>>>>>> let me know.
>>>>>>>>>> 
>>>>>>>>>> Responses should be made by Sat, 25 May 2024 13:03:15 +0000.
>>>>>>>>>> Anything received after that time might be too late.
>>>>>>>>>> 
>>>>>>>>>> The whole patch series can be found in one patch at:
>>>>>>>>>> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.160-rc1.gz
>>>>>>>>>> or in the git tree and branch at:
>>>>>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>>>>>>>> and the diffstat can be found below.
>>>>>>>>>> 
>>>>>>>>>> thanks,
>>>>>>>>>> 
>>>>>>>>>> greg k-h
>>>>>>>>>> 
>>>>>>>>>> -------------
>>>>>>>>>> Pseudo-Shortlog of commits:
>>>>>>>>> 
>>>>>>>>> ...
>>>>>>>>> 
>>>>>>>>>> NeilBrown <neilb@suse.de>
>>>>>>>>>>    nfsd: don't allow nfsd threads to be signalled.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I am seeing a suspend regression on a couple boards and bisect is pointing
>>>>>>>>> to the above commit. Reverting this commit does fix the issue.
>>>>>>>> Ugh, that fixes the report from others.  Can you cc: everyone on that
>>>>>>>> and figure out what is going on, as this keeps going back and forth...
>>>>>>> 
>>>>>>> 
>>>>>>> Adding Chuck, Neil and Chris from the bug report here [0].
>>>>>>> 
>>>>>>> With the above applied to v5.15.y, I am seeing suspend on 2 of our boards fail. These boards are using NFS and on entry to suspend I am now seeing ...
>>>>>>> 
>>>>>>> Freezing of tasks failed after 20.002 seconds (1 tasks refusing to
>>>>>>> freeze, wq_busy=0):
>>>>>>> 
>>>>>>> The boards appear to hang at that point. So may be something else missing?
>>>>>> Note that we don't have access to hardware like this, so
>>>>>> we haven't tested that patch (even the upstream version)
>>>>>> with suspend on that hardware.
>>>>> 
>>>>> 
>>>>> No problem, I would not expect you to have this particular hardware :-)
>>>>> 
>>>>>> So, it could be something missing, or it could be that
>>>>>> patch has a problem.
>>>>>> It would help us to know if you observe the same issue
>>>>>> with an upstream kernel, if that is possible.
>>>>> 
>>>>> 
>>>>> I don't observe this with either mainline, -next or any other stable branch. So that would suggest that something else is missing from linux-5.15.y.
>>>> 
>>>> That helps. It would be very helpful to have a reproducer I can
>>>> use to confirm we have a fix. I'm sure this will be a process
>>>> that involves a non-trivial number of iterations.
>>> 
>>> Missing upstream patch is
>>> 
>>> Commit 9bd4161c5917 ("SUNRPC: change service idle list to be an llist")
>>> 
>>> This contains some freezer-related changes which probably should
>>> have been a separate patch.
>> 
>> Thanks for tracking that down.
>> 
>> 
>>> We probably just need to add "| TASK_FREEZABLE" in one or two places.
>>> I'll post a patch for testing in a little while.
>> 
>> My understanding is that the stable maintainers prefer a backport
>> of a patch (or patches) that are already applied to Linus' tree.
> 
> They also preferred a full backport of fs/nfsd/..  That hasn't gone so
> well :-)

Really? I count about 350 patches in the initial backport. Those patches
include nearly every NFSD patch from v5.16 up to the end of v6.2. We
agreed to stop once the filecache fixes had been applied; no-one asked
for a "full backport" from torvalds/HEAD.

Only two more patches have been applied since then. Three if you count
this one. All of these issues have been very narrow corner cases or
obscure test failures.

That is quite good, if you ask me. I don't see a problem, given the
monumental task and lack of NFSD stable testing infrastructure before
I began.


> In this case we would need 
> 
> Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")
> 
> to get TASK_FREEZABLE.
> I doubt that would be a good choice.

I will let Greg and Sasha decide how they want to proceed, but it
would be wise to include this detail in your patch description.


--
Chuck Lever

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[NeilBrown]

On Wed, 29 May 2024, NeilBrown wrote:
> 
> We probably just need to add "| TASK_FREEZABLE" in one or two places.
> I'll post a patch for testing in a little while.

There is no TASK_FREEZABLE before v6.1.
This isn't due to a missed backport. It is simply because of differences
in the freezer in older kernels.

Please test this patch.

Thanks,
NeilBrown

From 416bd6ae9a598e64931d34b76aa58f39b11841cd Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.de>
Date: Wed, 29 May 2024 09:38:22 +1000
Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:

Prior to v6.1, the freezer will only wake a kernel thread from an
uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
ignore it instead.

Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
Signed-off-by: NeilBrown <neilb@suse.de>
---
 net/sunrpc/svc_xprt.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index b19592673eef..12e9293bd12b 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -764,10 +764,12 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
 	clear_bit(RQ_BUSY, &rqstp->rq_flags);
 	smp_mb__after_atomic();
 
+	freezer_do_not_count();
 	if (likely(rqst_should_sleep(rqstp)))
 		time_left = schedule_timeout(timeout);
 	else
 		__set_current_state(TASK_RUNNING);
+	freezer_count();
 
 	try_to_freeze();
 
-- 
2.44.0

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Jon Hunter]

On 29/05/2024 00:42, NeilBrown wrote:
> On Wed, 29 May 2024, NeilBrown wrote:
>>
>> We probably just need to add "| TASK_FREEZABLE" in one or two places.
>> I'll post a patch for testing in a little while.
> 
> There is no TASK_FREEZABLE before v6.1.
> This isn't due to a missed backport. It is simply because of differences
> in the freezer in older kernels.
> 
> Please test this patch.
> 
> Thanks,
> NeilBrown
> 
>  From 416bd6ae9a598e64931d34b76aa58f39b11841cd Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Wed, 29 May 2024 09:38:22 +1000
> Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
> 
> Prior to v6.1, the freezer will only wake a kernel thread from an
> uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
> IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
> ignore it instead.
> 
> Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>   net/sunrpc/svc_xprt.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> index b19592673eef..12e9293bd12b 100644
> --- a/net/sunrpc/svc_xprt.c
> +++ b/net/sunrpc/svc_xprt.c
> @@ -764,10 +764,12 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
>   	clear_bit(RQ_BUSY, &rqstp->rq_flags);
>   	smp_mb__after_atomic();
>   
> +	freezer_do_not_count();
>   	if (likely(rqst_should_sleep(rqstp)))
>   		time_left = schedule_timeout(timeout);
>   	else
>   		__set_current_state(TASK_RUNNING);
> +	freezer_count();
>   
>   	try_to_freeze();
>   


Thanks. I gave this a try on top of v5.15.160-rc1, but I am still seeing
the following and the board hangs ...

Freezing of tasks failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):

So unfortunately this does not fix it :-(

Jon

-- 
nvpublic

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[NeilBrown]

On Wed, 29 May 2024, Jon Hunter wrote:
> On 29/05/2024 00:42, NeilBrown wrote:
> > On Wed, 29 May 2024, NeilBrown wrote:
> >>
> >> We probably just need to add "| TASK_FREEZABLE" in one or two places.
> >> I'll post a patch for testing in a little while.
> > 
> > There is no TASK_FREEZABLE before v6.1.
> > This isn't due to a missed backport. It is simply because of differences
> > in the freezer in older kernels.
> > 
> > Please test this patch.
> > 
> > Thanks,
> > NeilBrown
> > 
> >  From 416bd6ae9a598e64931d34b76aa58f39b11841cd Mon Sep 17 00:00:00 2001
> > From: NeilBrown <neilb@suse.de>
> > Date: Wed, 29 May 2024 09:38:22 +1000
> > Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
> > 
> > Prior to v6.1, the freezer will only wake a kernel thread from an
> > uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
> > IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
> > ignore it instead.
> > 
> > Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
> > Signed-off-by: NeilBrown <neilb@suse.de>
> > ---
> >   net/sunrpc/svc_xprt.c | 2 ++
> >   1 file changed, 2 insertions(+)
> > 
> > diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> > index b19592673eef..12e9293bd12b 100644
> > --- a/net/sunrpc/svc_xprt.c
> > +++ b/net/sunrpc/svc_xprt.c
> > @@ -764,10 +764,12 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
> >   	clear_bit(RQ_BUSY, &rqstp->rq_flags);
> >   	smp_mb__after_atomic();
> >   
> > +	freezer_do_not_count();
> >   	if (likely(rqst_should_sleep(rqstp)))
> >   		time_left = schedule_timeout(timeout);
> >   	else
> >   		__set_current_state(TASK_RUNNING);
> > +	freezer_count();
> >   
> >   	try_to_freeze();
> >   
> 
> 
> Thanks. I gave this a try on top of v5.15.160-rc1, but I am still seeing
> the following and the board hangs ...
> 
> Freezing of tasks failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
> 
> So unfortunately this does not fix it :-(

Thanks for testing.
I can only guess that you had an active NFSv4.1 mount and that the
callback thread was causing problems.  Please try this.  I also changed
to use freezable_schedule* which seems like a better interface to do the
same thing.

If this doesn't fix it, we'll probably need to ask someone who remembers
the old freezer code.

Thanks,
NeilBrown

From 518f0c1150f988b3fe8e5e0d053a25c3aa6c7d44 Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.de>
Date: Wed, 29 May 2024 09:38:22 +1000
Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:

Prior to v6.1, the freezer will only wake a kernel thread from an
uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
ignore it instead.

To make this work with only upstream requests we would need
  Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")
which allows non-interruptible sleeps to be woken by the freezer.

Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
Signed-off-by: NeilBrown <neilb@suse.de>
---
 fs/nfs/callback.c     | 2 +-
 fs/nfsd/nfs4proc.c    | 3 ++-
 net/sunrpc/svc_xprt.c | 4 ++--
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
index 46a0a2d6962e..8fe143cad4a2 100644
--- a/fs/nfs/callback.c
+++ b/fs/nfs/callback.c
@@ -124,7 +124,7 @@ nfs41_callback_svc(void *vrqstp)
 		} else {
 			spin_unlock_bh(&serv->sv_cb_lock);
 			if (!kthread_should_stop())
-				schedule();
+				freezable_schedule();
 			finish_wait(&serv->sv_cb_waitq, &wq);
 		}
 	}
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 6779291efca9..e0ff2212866a 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -38,6 +38,7 @@
 #include <linux/slab.h>
 #include <linux/kthread.h>
 #include <linux/namei.h>
+#include <linux/freezer.h>
 
 #include <linux/sunrpc/addr.h>
 #include <linux/nfs_ssc.h>
@@ -1322,7 +1323,7 @@ static __be32 nfsd4_ssc_setup_dul(struct nfsd_net *nn, char *ipaddr,
 
 			/* allow 20secs for mount/unmount for now - revisit */
 			if (kthread_should_stop() ||
-					(schedule_timeout(20*HZ) == 0)) {
+					(freezable_schedule_timeout(20*HZ) == 0)) {
 				finish_wait(&nn->nfsd_ssc_waitq, &wait);
 				kfree(work);
 				return nfserr_eagain;
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index b19592673eef..3cf53e3140a5 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -705,7 +705,7 @@ static int svc_alloc_arg(struct svc_rqst *rqstp)
 			set_current_state(TASK_RUNNING);
 			return -EINTR;
 		}
-		schedule_timeout(msecs_to_jiffies(500));
+		freezable_schedule_timeout(msecs_to_jiffies(500));
 	}
 	rqstp->rq_page_end = &rqstp->rq_pages[pages];
 	rqstp->rq_pages[pages] = NULL; /* this might be seen in nfsd_splice_actor() */
@@ -765,7 +765,7 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
 	smp_mb__after_atomic();
 
 	if (likely(rqst_should_sleep(rqstp)))
-		time_left = schedule_timeout(timeout);
+		time_left = freezable_schedule_timeout(timeout);
 	else
 		__set_current_state(TASK_RUNNING);
 
-- 
2.44.0

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Jon Hunter]

On 29/05/2024 21:59, NeilBrown wrote:

...

> Thanks for testing.
> I can only guess that you had an active NFSv4.1 mount and that the
> callback thread was causing problems.  Please try this.  I also changed
> to use freezable_schedule* which seems like a better interface to do the
> same thing.
> 
> If this doesn't fix it, we'll probably need to ask someone who remembers
> the old freezer code.
> 
> Thanks,
> NeilBrown
> 
>  From 518f0c1150f988b3fe8e5e0d053a25c3aa6c7d44 Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Wed, 29 May 2024 09:38:22 +1000
> Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
> 
> Prior to v6.1, the freezer will only wake a kernel thread from an
> uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
> IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
> ignore it instead.
> 
> To make this work with only upstream requests we would need
>    Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")
> which allows non-interruptible sleeps to be woken by the freezer.
> 
> Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>   fs/nfs/callback.c     | 2 +-
>   fs/nfsd/nfs4proc.c    | 3 ++-
>   net/sunrpc/svc_xprt.c | 4 ++--
>   3 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
> index 46a0a2d6962e..8fe143cad4a2 100644
> --- a/fs/nfs/callback.c
> +++ b/fs/nfs/callback.c
> @@ -124,7 +124,7 @@ nfs41_callback_svc(void *vrqstp)
>   		} else {
>   			spin_unlock_bh(&serv->sv_cb_lock);
>   			if (!kthread_should_stop())
> -				schedule();
> +				freezable_schedule();
>   			finish_wait(&serv->sv_cb_waitq, &wq);
>   		}
>   	}
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 6779291efca9..e0ff2212866a 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -38,6 +38,7 @@
>   #include <linux/slab.h>
>   #include <linux/kthread.h>
>   #include <linux/namei.h>
> +#include <linux/freezer.h>
>   
>   #include <linux/sunrpc/addr.h>
>   #include <linux/nfs_ssc.h>
> @@ -1322,7 +1323,7 @@ static __be32 nfsd4_ssc_setup_dul(struct nfsd_net *nn, char *ipaddr,
>   
>   			/* allow 20secs for mount/unmount for now - revisit */
>   			if (kthread_should_stop() ||
> -					(schedule_timeout(20*HZ) == 0)) {
> +					(freezable_schedule_timeout(20*HZ) == 0)) {
>   				finish_wait(&nn->nfsd_ssc_waitq, &wait);
>   				kfree(work);
>   				return nfserr_eagain;
> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> index b19592673eef..3cf53e3140a5 100644
> --- a/net/sunrpc/svc_xprt.c
> +++ b/net/sunrpc/svc_xprt.c
> @@ -705,7 +705,7 @@ static int svc_alloc_arg(struct svc_rqst *rqstp)
>   			set_current_state(TASK_RUNNING);
>   			return -EINTR;
>   		}
> -		schedule_timeout(msecs_to_jiffies(500));
> +		freezable_schedule_timeout(msecs_to_jiffies(500));
>   	}
>   	rqstp->rq_page_end = &rqstp->rq_pages[pages];
>   	rqstp->rq_pages[pages] = NULL; /* this might be seen in nfsd_splice_actor() */
> @@ -765,7 +765,7 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
>   	smp_mb__after_atomic();
>   
>   	if (likely(rqst_should_sleep(rqstp)))
> -		time_left = schedule_timeout(timeout);
> +		time_left = freezable_schedule_timeout(timeout);
>   	else
>   		__set_current_state(TASK_RUNNING);
>   


That did the trick! Suspend is now working again on top of v5.15.160-rc1 
with this change.

Feel free to add my ...

Tested-by: Jon Hunter <jonathanh@nvidia.com>

Thanks
Jon

-- 
nvpublic

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever]

On Thu, May 30, 2024 at 01:11:47PM +0100, Jon Hunter wrote:
> 
> On 29/05/2024 21:59, NeilBrown wrote:
> 
> ...
> 
> > Thanks for testing.
> > I can only guess that you had an active NFSv4.1 mount and that the
> > callback thread was causing problems.  Please try this.  I also changed
> > to use freezable_schedule* which seems like a better interface to do the
> > same thing.
> > 
> > If this doesn't fix it, we'll probably need to ask someone who remembers
> > the old freezer code.
> > 
> > Thanks,
> > NeilBrown
> > 
> >  From 518f0c1150f988b3fe8e5e0d053a25c3aa6c7d44 Mon Sep 17 00:00:00 2001
> > From: NeilBrown <neilb@suse.de>
> > Date: Wed, 29 May 2024 09:38:22 +1000
> > Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
> > 
> > Prior to v6.1, the freezer will only wake a kernel thread from an
> > uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
> > IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
> > ignore it instead.
> > 
> > To make this work with only upstream requests we would need
> >    Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")
> > which allows non-interruptible sleeps to be woken by the freezer.
> > 
> > Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
> > Signed-off-by: NeilBrown <neilb@suse.de>
> > ---
> >   fs/nfs/callback.c     | 2 +-
> >   fs/nfsd/nfs4proc.c    | 3 ++-
> >   net/sunrpc/svc_xprt.c | 4 ++--
> >   3 files changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
> > index 46a0a2d6962e..8fe143cad4a2 100644
> > --- a/fs/nfs/callback.c
> > +++ b/fs/nfs/callback.c
> > @@ -124,7 +124,7 @@ nfs41_callback_svc(void *vrqstp)
> >   		} else {
> >   			spin_unlock_bh(&serv->sv_cb_lock);
> >   			if (!kthread_should_stop())
> > -				schedule();
> > +				freezable_schedule();
> >   			finish_wait(&serv->sv_cb_waitq, &wq);
> >   		}
> >   	}
> > diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> > index 6779291efca9..e0ff2212866a 100644
> > --- a/fs/nfsd/nfs4proc.c
> > +++ b/fs/nfsd/nfs4proc.c
> > @@ -38,6 +38,7 @@
> >   #include <linux/slab.h>
> >   #include <linux/kthread.h>
> >   #include <linux/namei.h>
> > +#include <linux/freezer.h>
> >   #include <linux/sunrpc/addr.h>
> >   #include <linux/nfs_ssc.h>
> > @@ -1322,7 +1323,7 @@ static __be32 nfsd4_ssc_setup_dul(struct nfsd_net *nn, char *ipaddr,
> >   			/* allow 20secs for mount/unmount for now - revisit */
> >   			if (kthread_should_stop() ||
> > -					(schedule_timeout(20*HZ) == 0)) {
> > +					(freezable_schedule_timeout(20*HZ) == 0)) {
> >   				finish_wait(&nn->nfsd_ssc_waitq, &wait);
> >   				kfree(work);
> >   				return nfserr_eagain;
> > diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> > index b19592673eef..3cf53e3140a5 100644
> > --- a/net/sunrpc/svc_xprt.c
> > +++ b/net/sunrpc/svc_xprt.c
> > @@ -705,7 +705,7 @@ static int svc_alloc_arg(struct svc_rqst *rqstp)
> >   			set_current_state(TASK_RUNNING);
> >   			return -EINTR;
> >   		}
> > -		schedule_timeout(msecs_to_jiffies(500));
> > +		freezable_schedule_timeout(msecs_to_jiffies(500));
> >   	}
> >   	rqstp->rq_page_end = &rqstp->rq_pages[pages];
> >   	rqstp->rq_pages[pages] = NULL; /* this might be seen in nfsd_splice_actor() */
> > @@ -765,7 +765,7 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
> >   	smp_mb__after_atomic();
> >   	if (likely(rqst_should_sleep(rqstp)))
> > -		time_left = schedule_timeout(timeout);
> > +		time_left = freezable_schedule_timeout(timeout);
> >   	else
> >   		__set_current_state(TASK_RUNNING);
> 
> 
> That did the trick! Suspend is now working again on top of v5.15.160-rc1
> with this change.
> 
> Feel free to add my ...
> 
> Tested-by: Jon Hunter <jonathanh@nvidia.com>

Since I've heard no objections, I've added this to nfsd-5.15.y for
testing. I plan to send it along to stable@ when testing completes.

-- 
Chuck Lever

Re: [PATCH 5.15 00/23] 5.15.160-rc1 review

[Chuck Lever III]

> On May 29, 2024, at 4:59 PM, NeilBrown <neilb@suse.de> wrote:
> 
> On Wed, 29 May 2024, Jon Hunter wrote:
>> On 29/05/2024 00:42, NeilBrown wrote:
>>> On Wed, 29 May 2024, NeilBrown wrote:
>>>> 
>>>> We probably just need to add "| TASK_FREEZABLE" in one or two places.
>>>> I'll post a patch for testing in a little while.
>>> 
>>> There is no TASK_FREEZABLE before v6.1.
>>> This isn't due to a missed backport. It is simply because of differences
>>> in the freezer in older kernels.
>>> 
>>> Please test this patch.
>>> 
>>> Thanks,
>>> NeilBrown
>>> 
>>> From 416bd6ae9a598e64931d34b76aa58f39b11841cd Mon Sep 17 00:00:00 2001
>>> From: NeilBrown <neilb@suse.de>
>>> Date: Wed, 29 May 2024 09:38:22 +1000
>>> Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
>>> 
>>> Prior to v6.1, the freezer will only wake a kernel thread from an
>>> uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
>>> IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
>>> ignore it instead.
>>> 
>>> Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
>>> Signed-off-by: NeilBrown <neilb@suse.de>
>>> ---
>>>  net/sunrpc/svc_xprt.c | 2 ++
>>>  1 file changed, 2 insertions(+)
>>> 
>>> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
>>> index b19592673eef..12e9293bd12b 100644
>>> --- a/net/sunrpc/svc_xprt.c
>>> +++ b/net/sunrpc/svc_xprt.c
>>> @@ -764,10 +764,12 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
>>>   clear_bit(RQ_BUSY, &rqstp->rq_flags);
>>>   smp_mb__after_atomic();
>>> 
>>> + freezer_do_not_count();
>>>   if (likely(rqst_should_sleep(rqstp)))
>>>   time_left = schedule_timeout(timeout);
>>>   else
>>>   __set_current_state(TASK_RUNNING);
>>> + freezer_count();
>>> 
>>>   try_to_freeze();
>>> 
>> 
>> 
>> Thanks. I gave this a try on top of v5.15.160-rc1, but I am still seeing
>> the following and the board hangs ...
>> 
>> Freezing of tasks failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
>> 
>> So unfortunately this does not fix it :-(
> 
> Thanks for testing.
> I can only guess that you had an active NFSv4.1 mount and that the
> callback thread was causing problems.  Please try this.  I also changed
> to use freezable_schedule* which seems like a better interface to do the
> same thing.
> 
> If this doesn't fix it, we'll probably need to ask someone who remembers
> the old freezer code.
> 
> Thanks,
> NeilBrown
> 
> From 518f0c1150f988b3fe8e5e0d053a25c3aa6c7d44 Mon Sep 17 00:00:00 2001
> From: NeilBrown <neilb@suse.de>
> Date: Wed, 29 May 2024 09:38:22 +1000
> Subject: [PATCH] sunrpc: exclude from freezer when waiting for requests:
> 
> Prior to v6.1, the freezer will only wake a kernel thread from an
> uninterruptible sleep.  Since we changed svc_get_next_xprt() to use and
> IDLE sleep the freezer cannot wake it.  we need to tell the freezer to
> ignore it instead.
> 
> To make this work with only upstream requests we would need
>  Commit f5d39b020809 ("freezer,sched: Rewrite core freezer logic")
> which allows non-interruptible sleeps to be woken by the freezer.
> 
> Fixes: 9b8a8e5e8129 ("nfsd: don't allow nfsd threads to be signalled.")
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
> fs/nfs/callback.c     | 2 +-
> fs/nfsd/nfs4proc.c    | 3 ++-
> net/sunrpc/svc_xprt.c | 4 ++--
> 3 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfs/callback.c b/fs/nfs/callback.c
> index 46a0a2d6962e..8fe143cad4a2 100644
> --- a/fs/nfs/callback.c
> +++ b/fs/nfs/callback.c
> @@ -124,7 +124,7 @@ nfs41_callback_svc(void *vrqstp)
> } else {
> spin_unlock_bh(&serv->sv_cb_lock);
> if (!kthread_should_stop())
> - schedule();
> + freezable_schedule();
> finish_wait(&serv->sv_cb_waitq, &wq);
> }
> }
> diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> index 6779291efca9..e0ff2212866a 100644
> --- a/fs/nfsd/nfs4proc.c
> +++ b/fs/nfsd/nfs4proc.c
> @@ -38,6 +38,7 @@
> #include <linux/slab.h>
> #include <linux/kthread.h>
> #include <linux/namei.h>
> +#include <linux/freezer.h>
> 
> #include <linux/sunrpc/addr.h>
> #include <linux/nfs_ssc.h>
> @@ -1322,7 +1323,7 @@ static __be32 nfsd4_ssc_setup_dul(struct nfsd_net *nn, char *ipaddr,
> 
> /* allow 20secs for mount/unmount for now - revisit */
> if (kthread_should_stop() ||
> - (schedule_timeout(20*HZ) == 0)) {
> + (freezable_schedule_timeout(20*HZ) == 0)) {
> finish_wait(&nn->nfsd_ssc_waitq, &wait);
> kfree(work);
> return nfserr_eagain;
> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> index b19592673eef..3cf53e3140a5 100644
> --- a/net/sunrpc/svc_xprt.c
> +++ b/net/sunrpc/svc_xprt.c
> @@ -705,7 +705,7 @@ static int svc_alloc_arg(struct svc_rqst *rqstp)
> set_current_state(TASK_RUNNING);
> return -EINTR;
> }
> - schedule_timeout(msecs_to_jiffies(500));
> + freezable_schedule_timeout(msecs_to_jiffies(500));
> }
> rqstp->rq_page_end = &rqstp->rq_pages[pages];
> rqstp->rq_pages[pages] = NULL; /* this might be seen in nfsd_splice_actor() */
> @@ -765,7 +765,7 @@ static struct svc_xprt *svc_get_next_xprt(struct svc_rqst *rqstp, long timeout)
> smp_mb__after_atomic();
> 
> if (likely(rqst_should_sleep(rqstp)))
> - time_left = schedule_timeout(timeout);
> + time_left = freezable_schedule_timeout(timeout);
> else
> __set_current_state(TASK_RUNNING);
> 
> -- 
> 2.44.0

I will merge this into nfsd-5.10.

I can also run this past NFSD's upstream CI and send it along to
stable if you would like.


--
Chuck Lever