[REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[email200202]

#regzbot introduced: v6.1.70..v6.1.71


After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown.

The problem is related to NFS service. Stopping NFS service hangs:

# /etc/init.d/nfs  stop
  * Caching service dependencies ... [ ok ]
  * Stopping NFS mountd ... [ ok ]
  * Stopping NFS daemon ... [ ok ]

then it hangs


Shutdown does not hang when NFS service is removed.

# rc-update  del  nfs

I had this kernel error in the log:

Jan 10 17:32:25 [rpc.mountd] Caught signal 15, un-registering and exiting.
Jan 10 17:32:25 [kernel] [ 2005.560991] ------------[ cut here ]------------
Jan 10 17:32:25 [kernel] [ 2005.560996] kernel BUG at net/sunrpc/svc.c:576!
Jan 10 17:32:25 [kernel] [ 2005.561004] invalid opcode: 0000 [#1] 
PREEMPT SMP PTI
Jan 10 17:32:25 [kernel] [ 2005.561012] CPU: 0 PID: 8079 Comm: nfsd 
Tainted: P           O       6.1.71-gentoo #1
Jan 10 17:32:25 [kernel] [ 2005.561017] Hardware name: Gigabyte 
Technology Co., Ltd. X58A-UD3R/X58A-UD3R, BIOS FB 08/24/2010
Jan 10 17:32:25 [kernel] [ 2005.561020] RIP: 0010:svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561033] Code: 5b 5d 41 5c 41 5d c3 cc cc 
cc cc 55 48 8d 6f ec 53 48 89 fb 48 83 c7 44 e8 b6 6c 5f ff 48 8b 53 1c 
48 8d 43 1c 48 39 c2 74 02 <0f> 0b 48 8b 53 2c 48 8d 43 2c 48 39 c2 74 
02 0f 0b 48 89 ef e8 6b
Jan 10 17:32:25 [kernel] [ 2005.561038] RSP: 0018:ffffc90001edbee8 
EFLAGS: 00010287
Jan 10 17:32:25 [kernel] [ 2005.561043] RAX: ffff88816c1a1c30 RBX: 
ffff88816c1a1c14 RCX: 0000000000000000
Jan 10 17:32:25 [kernel] [ 2005.561047] RDX: ffff88813dfcc018 RSI: 
0000000000000286 RDI: ffff88890bc9b9c0
Jan 10 17:32:25 [kernel] [ 2005.561051] RBP: ffff88816c1a1c00 R08: 
ffff88810310b600 R09: 0000000000000000
Jan 10 17:32:25 [kernel] [ 2005.561055] R10: ffff8881bc3b4000 R11: 
ffff8881bc3b4000 R12: ffffffff82e061c0
Jan 10 17:32:25 [kernel] [ 2005.561058] R13: ffff8881bc0e8000 R14: 
ffff88810310b600 R15: ffffc90002323c80
Jan 10 17:32:25 [kernel] [ 2005.561062] FS:  0000000000000000(0000) 
GS:ffff88890bc00000(0000) knlGS:0000000000000000
Jan 10 17:32:25 [kernel] [ 2005.561067] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
Jan 10 17:32:25 [kernel] [ 2005.561071] CR2: 000055e971e376f8 CR3: 
000000000260a000 CR4: 00000000000006f0
Jan 10 17:32:25 [kernel] [ 2005.561075] Call Trace:
Jan 10 17:32:25 [kernel] [ 2005.561079]  <TASK>
Jan 10 17:32:25 [kernel] [ 2005.561082]  ? __die_body+0x15/0x57
Jan 10 17:32:25 [kernel] [ 2005.561091]  ? die+0x2b/0x44
Jan 10 17:32:25 [kernel] [ 2005.561097]  ? do_trap+0x76/0xf9
Jan 10 17:32:25 [kernel] [ 2005.561102]  ? svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561108]  ? svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561114]  ? do_error_trap+0x69/0x93
Jan 10 17:32:25 [kernel] [ 2005.561119]  ? svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561126]  ? exc_invalid_op+0x49/0x5d
Jan 10 17:32:25 [kernel] [ 2005.561133]  ? svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561139]  ? asm_exc_invalid_op+0x16/0x20
Jan 10 17:32:25 [kernel] [ 2005.561148]  ? svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561155]  ? svc_destroy+0x12/0x56
Jan 10 17:32:25 [kernel] [ 2005.561161]  nfsd+0x13d/0x162
Jan 10 17:32:25 [kernel] [ 2005.561170]  ? svc_put+0x2f/0x2f
Jan 10 17:32:25 [kernel] [ 2005.561176]  kthread+0xd0/0xd8
Jan 10 17:32:25 [kernel] [ 2005.561183]  ? 
kthread_complete_and_exit+0x16/0x16
Jan 10 17:32:25 [kernel] [ 2005.561189]  ret_from_fork+0x22/0x30
Jan 10 17:32:25 [kernel] [ 2005.561196]  </TASK>
Jan 10 17:32:25 [kernel] [ 2005.561198] Modules linked in: snd_seq_dummy 
snd_seq snd_seq_device nvidia_uvm(PO) bluetooth ecdh_generic ecc bridge 
stp llc ipv6 crc_ccitt ch341 usbserial nvidia_drm(PO) nvidia_modeset(PO) 
nvidia(PO) tda10048 tda8290 iTCO_wdt iTCO_vendor_support tda18271 it87 
hwmon_vid dm_crypt coretemp dm_multipath dm_mod kvm_intel dax video 
snd_hda_codec_realtek snd_hda_codec_generic kvm drm_kms_helper irqbypass 
ledtrig_audio i2c_i801 pcspkr serio_raw drm i2c_smbus snd_hda_intel 
i2c_core snd_intel_dspcfg fb_sys_fops syscopyarea snd_hda_codec 
sysfillrect lpc_ich snd_hda_core sysimgblt mfd_core snd_hwdep uhci_hcd 
rtc_cmos wmi
Jan 10 17:32:25 [kernel] [ 2005.561274] ---[ end trace 0000000000000000 ]---
Jan 10 17:32:25 [kernel] [ 2005.561277] RIP: 0010:svc_destroy+0x1f/0x56
Jan 10 17:32:25 [kernel] [ 2005.561284] Code: 5b 5d 41 5c 41 5d c3 cc cc 
cc cc 55 48 8d 6f ec 53 48 89 fb 48 83 c7 44 e8 b6 6c 5f ff 48 8b 53 1c 
48 8d 43 1c 48 39 c2 74 02 <0f> 0b 48 8b 53 2c 48 8d 43 2c 48 39 c2 74 
02 0f 0b 48 89 ef e8 6b

The problem is always reproducible. Steps to reproduce:
1. Update to kernel 6.1.71
2. Start nfs service
3. Try to shutdown
4. The computer hangs

Reverting the following 3 commits fixed the problem in kernel 6.1.71:

f9a01938e07910224d4a2fd00583725d686c3f38
bb4f791cb2de1140d0fbcedfe9e791ff364021d7
03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Greg KH]

On Thu, Jan 11, 2024 at 07:20:02PM +1100, email200202 wrote:
> Reverting the following 3 commits fixed the problem in kernel 6.1.71:
> 
> f9a01938e07910224d4a2fd00583725d686c3f38
> bb4f791cb2de1140d0fbcedfe9e791ff364021d7
> 03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a

When sending us git ids, please show the full context so we have a hint
as to what they are.  For this, it should be:

f9a01938e079 ("NFSD: fix possible oops when nfsd/pool_stats is closed.")
bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")

Do you also have these issues in the latest 6.6.y release?  6.7?

thanks,

greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[email200202]

Hi Greg

I'm sorry. This my first kernel report.

I didn't test 6.6.x and 6.7.x.  I use only 6.1.x.

Best Regards

John G

On 11/1/24 20:30, Greg KH wrote:
> On Thu, Jan 11, 2024 at 07:20:02PM +1100, email200202 wrote:
>> Reverting the following 3 commits fixed the problem in kernel 6.1.71:
>>
>> f9a01938e07910224d4a2fd00583725d686c3f38
>> bb4f791cb2de1140d0fbcedfe9e791ff364021d7
>> 03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a
> When sending us git ids, please show the full context so we have a hint
> as to what they are.  For this, it should be:
>
> f9a01938e079 ("NFSD: fix possible oops when nfsd/pool_stats is closed.")
> bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
>
> Do you also have these issues in the latest 6.6.y release?  6.7?
>
> thanks,
>
> greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Greg KH]

On Thu, Jan 11, 2024 at 09:10:39PM +1100, email200202 wrote:
> Hi Greg
> 
> I'm sorry. This my first kernel report.
> 
> I didn't test 6.6.x and 6.7.x.  I use only 6.1.x.

Can you do so?

thanks,

greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[email200202]

Hi Greg

Here is the test results for the latest versions of 6.6.x and 6.7.x 
available in Gentoo portage

1- Stopping NFS service failed but, unlike 6.1.x, it did NOT hang.

# uname -r
6.6.11-gentoo
# /etc/init.d/nfs stop
  * Stopping NFS mountd ... [ ok ]
  * Stopping NFS daemon ...
  * start-stop-daemon: 8 process(es) refused to stop [ !! ]
  * Unexporting NFS directories ... [ ok ]
  * ERROR: nfs failed to stop
# /etc/init.d/nfs start
  * WARNING: nfs has already been started


# uname -r
6.7.0-gentoo
# /etc/init.d/nfs stop
  * Stopping NFS mountd ... [ ok ]
  * Stopping NFS daemon ...
  * start-stop-daemon: 8 process(es) refused to stop [ !! ]
  * Unexporting NFS directories ... [ ok ]
  * ERROR: nfs failed to stop
# /etc/init.d/nfs start
  * WARNING: nfs has already been started

2- Shutdown didn't hang in both of them

Best regards
John G

On 11/1/24 21:21, Greg KH wrote:
> On Thu, Jan 11, 2024 at 09:10:39PM +1100, email200202 wrote:
>> Hi Greg
>>
>> I'm sorry. This my first kernel report.
>>
>> I didn't test 6.6.x and 6.7.x.  I use only 6.1.x.
> Can you do so?
>
> thanks,
>
> greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Chuck Lever III]

> On Jan 11, 2024, at 8:58 AM, email200202 <email200202@yahoo.com> wrote:
> 
> Hi Greg
> 
> Here is the test results for the latest versions of 6.6.x and 6.7.x available in Gentoo portage
> 
> 1- Stopping NFS service failed but, unlike 6.1.x, it did NOT hang.
> 
> # uname -r
> 6.6.11-gentoo
> # /etc/init.d/nfs stop
>  * Stopping NFS mountd ... [ ok ]
>  * Stopping NFS daemon ...
>  * start-stop-daemon: 8 process(es) refused to stop [ !! ]
>  * Unexporting NFS directories ... [ ok ]
>  * ERROR: nfs failed to stop
> # /etc/init.d/nfs start
>  * WARNING: nfs has already been started
> 
> 
> # uname -r
> 6.7.0-gentoo
> # /etc/init.d/nfs stop
>  * Stopping NFS mountd ... [ ok ]
>  * Stopping NFS daemon ...
>  * start-stop-daemon: 8 process(es) refused to stop [ !! ]
>  * Unexporting NFS directories ... [ ok ]
>  * ERROR: nfs failed to stop
> # /etc/init.d/nfs start
>  * WARNING: nfs has already been started
> 
> 2- Shutdown didn't hang in both of them
> 
> Best regards
> John G
> 
> On 11/1/24 21:21, Greg KH wrote:
>> On Thu, Jan 11, 2024 at 09:10:39PM +1100, email200202 wrote:
>>> Hi Greg
>>> 
>>> I'm sorry. This my first kernel report.
>>> 
>>> I didn't test 6.6.x and 6.7.x.  I use only 6.1.x.
>> Can you do so?
>> 
>> thanks,
>> 
>> greg k-h

I haven't seen emails before this one. But here are the only
three new NFSD commits in v6.1.71:

f9a01938e079 NFSD: fix possible oops when nfsd/pool_stats is closed.
bb4f791cb2de nfsd: call nfsd_last_thread() before final nfsd_put()
03d68ffc48b9 nfsd: separate nfsd_last_thread() from nfsd_put()

And the latter two are directly related to NFSD shut down.

--
Chuck Lever

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Jeff Layton]

On Thu, 2024-01-11 at 10:30 +0100, Greg KH wrote:
> On Thu, Jan 11, 2024 at 07:20:02PM +1100, email200202 wrote:
> > Reverting the following 3 commits fixed the problem in kernel 6.1.71:
> > 
> > f9a01938e07910224d4a2fd00583725d686c3f38
> > bb4f791cb2de1140d0fbcedfe9e791ff364021d7
> > 03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a
> 
> When sending us git ids, please show the full context so we have a hint
> as to what they are.  For this, it should be:
> 
> f9a01938e079 ("NFSD: fix possible oops when nfsd/pool_stats is closed.")
> bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> 
> Do you also have these issues in the latest 6.6.y release?  6.7?
> 
> 

I wasn't able to reproduce the exact same bug as the reporter, but I did
see a different panic due to the above patches. Some of those fixes
aren't appropriate without pulling in earlier commits, but I think it's
probably best to just drop these two:

bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")

Greg, does that sound OK?

Thanks,
-- 
Jeff Layton <jlayton@kernel.org>

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Greg KH]

On Thu, Jan 11, 2024 at 12:26:39PM -0500, Jeff Layton wrote:
> On Thu, 2024-01-11 at 10:30 +0100, Greg KH wrote:
> > On Thu, Jan 11, 2024 at 07:20:02PM +1100, email200202 wrote:
> > > Reverting the following 3 commits fixed the problem in kernel 6.1.71:
> > > 
> > > f9a01938e07910224d4a2fd00583725d686c3f38
> > > bb4f791cb2de1140d0fbcedfe9e791ff364021d7
> > > 03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a
> > 
> > When sending us git ids, please show the full context so we have a hint
> > as to what they are.  For this, it should be:
> > 
> > f9a01938e079 ("NFSD: fix possible oops when nfsd/pool_stats is closed.")
> > bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> > 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> > 
> > Do you also have these issues in the latest 6.6.y release?  6.7?
> > 
> > 
> 
> I wasn't able to reproduce the exact same bug as the reporter, but I did
> see a different panic due to the above patches. Some of those fixes
> aren't appropriate without pulling in earlier commits, but I think it's
> probably best to just drop these two:
> 
> bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> 
> Greg, does that sound OK?

Sure, can you send reverts for them, or do you need me to make them?

thanks,

greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Greg KH]

On Sat, Jan 13, 2024 at 10:35:09AM +0100, Greg KH wrote:
> On Thu, Jan 11, 2024 at 12:26:39PM -0500, Jeff Layton wrote:
> > On Thu, 2024-01-11 at 10:30 +0100, Greg KH wrote:
> > > On Thu, Jan 11, 2024 at 07:20:02PM +1100, email200202 wrote:
> > > > Reverting the following 3 commits fixed the problem in kernel 6.1.71:
> > > > 
> > > > f9a01938e07910224d4a2fd00583725d686c3f38
> > > > bb4f791cb2de1140d0fbcedfe9e791ff364021d7
> > > > 03d68ffc48b94cc1e15bbf3b4f16f1e1e4fa286a
> > > 
> > > When sending us git ids, please show the full context so we have a hint
> > > as to what they are.  For this, it should be:
> > > 
> > > f9a01938e079 ("NFSD: fix possible oops when nfsd/pool_stats is closed.")
> > > bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> > > 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> > > 
> > > Do you also have these issues in the latest 6.6.y release?  6.7?
> > > 
> > > 
> > 
> > I wasn't able to reproduce the exact same bug as the reporter, but I did
> > see a different panic due to the above patches. Some of those fixes
> > aren't appropriate without pulling in earlier commits, but I think it's
> > probably best to just drop these two:
> > 
> > bb4f791cb2de ("nfsd: call nfsd_last_thread() before final nfsd_put()")
> > 03d68ffc48b9 ("nfsd: separate nfsd_last_thread() from nfsd_put()")
> > 
> > Greg, does that sound OK?
> 
> Sure, can you send reverts for them, or do you need me to make them?

Nevermind, I made them, thanks.

greg k-h

Re: [REGRESSION] After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown

[Linux regression tracking #update (Thorsten Leemhuis)]

On 11.01.24 09:20, email200202 wrote:
> 
> #regzbot introduced: v6.1.70..v6.1.71
> 
> 
> After kernel upgrade 6.1.70 to 6.1.71, the computer hangs during shutdown.
> 
> The problem is related to NFS service. Stopping NFS service hangs:
> 
> # /etc/init.d/nfs  stop
>  * Caching service dependencies ... [ ok ]
>  * Stopping NFS mountd ... [ ok ]
>  * Stopping NFS daemon ... [ ok ]
> [...]

#regzbot fix: b2c545c39877408a2fe2
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.