* [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
@ 2026-03-03 18:31 Mikhail Zaslonko
2026-03-04 16:13 ` Mikhail Zaslonko
0 siblings, 1 reply; 11+ messages in thread
From: Mikhail Zaslonko @ 2026-03-03 18:31 UTC (permalink / raw)
To: linux-next, linux-s390, Ingo Molnar; +Cc: Heiko Carstens, Alexander Egorenkov
Hello,
we have kernel-next boot hang on s390 starting next-20260302.
I bisected it in linux-next to:
c50f05bd3c4e ("Merge branch into tip/master: 'sched/hrtick'")
Good:
72a2ab46f045
d50da4b5915f (2nd parent: sched/hrtick branch)
Bad:
c50f05bd3c4e (merge commit)
Environment:
- s390 under z/VM
- many CPUs defined (32+)
- boot hangs early with RCU stall (see boot log excerpt below)
[ 0.953192] Freeing unused kernel image (initmem) memory: 2520K
[ 0.962509] Write protected read-only-after-init data: 188k
[ 0.962884] Checked W+X mappings: passed, no W+X pages found
[ 0.962888] Run /init as init process
P+q6E616D65 \[ 29.503589] systemd[1]: Inserted module 'autofs4'
hangs here, no logs ...
[ 98.619693] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 98.619708] rcu: 4-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=25/26 fqs=0 (false positive?)
[ 98.619715] rcu: 5-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=25/26 fqs=0 (false positive?)
[ 98.619720] rcu: 6-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=23/24 fqs=0 (false positive?)
[ 98.619724] rcu: 7-...!: (1 GPs behind) idle=04e8/0/0x0 softirq=28/29 fqs=0 (false positive?)
[ 98.619729] rcu: 8-...!: (1 GPs behind) idle=0568/0/0x0 softirq=27/28 fqs=0 (false positive?)
[ 98.619733] rcu: 9-...!: (1 GPs behind) idle=04e8/0/0x0 softirq=35/36 fqs=0 (false positive?)
[ 98.619737] rcu: 10-...!: (1 GPs behind) idle=0518/0/0x0 softirq=33/34 fqs=0 (false positive?)
[ 98.619742] rcu: 11-...!: (1 GPs behind) idle=0448/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619745] rcu: 12-...!: (1 GPs behind) idle=0488/0/0x0 softirq=23/24 fqs=0 (false positive?)
[ 98.619749] rcu: 13-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619753] rcu: 14-...!: (0 ticks this GP) idle=0498/0/0x0 softirq=26/26 fqs=0 (false positive?)
[ 98.619758] rcu: 15-...!: (1 GPs behind) idle=0448/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619762] rcu: 16-...!: (1 GPs behind) idle=0458/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619766] rcu: 17-...!: (1 GPs behind) idle=0438/0/0x0 softirq=21/22 fqs=0 (false positive?)
[ 98.619770] rcu: 18-...!: (1 GPs behind) idle=03e8/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619774] rcu: 19-...!: (1 GPs behind) idle=0458/0/0x0 softirq=22/23 fqs=0 (false positive?)
[ 98.619778] rcu: 20-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=24/25 fqs=0 (false positive?)
[ 98.619782] rcu: 21-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=23/24 fqs=0 (false positive?)
[ 98.619786] rcu: 22-...!: (1 GPs behind) idle=04d8/0/0x0 softirq=23/24 fqs=0 (false positive?)
[ 98.619790] rcu: 23-...!: (1 GPs behind) idle=0b68/0/0x0 softirq=29/30 fqs=0 (false positive?)
[ 98.619794] rcu: 24-...!: (1 GPs behind) idle=0508/0/0x0 softirq=25/26 fqs=0 (false positive?)
[ 98.619798] rcu: 25-...!: (1 GPs behind) idle=10c8/0/0x0 softirq=38/39 fqs=0 (false positive?)
[ 98.619802] rcu: 28-...!: (1 GPs behind) idle=0518/0/0x0 softirq=25/26 fqs=0 (false positive?)
[ 98.619806] rcu: 30-...!: (1 GPs behind) idle=0518/0/0x0 softirq=20/21 fqs=0 (false positive?)
[ 98.619810] rcu: 31-...!: (1 GPs behind) idle=0528/0/0x0 softirq=23/24 fqs=0 (false positive?)
[ 98.619815] rcu: (detected by 0, t=69116 jiffies, g=-1019, q=494 ncpus=32)
[ 98.619818] Task dump for CPU 4:
[ 98.619820] task:swapper/4 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x0
[ 98.619825] Call Trace:
[ 98.619828] [<000001b02c9afd98>] 0x1b02c9afd98
[ 98.619834] Task dump for CPU 5:
[ 98.619835] task:swapper/5 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x0
[ 98.619838] Call Trace:
[ 98.619839] [<000001b02c9b7d98>] 0x1b02c9b7d98
[ 98.619841] Task dump for CPU 6:
...
[ 98.619963] Task dump for CPU 31:
[ 98.619964] task:swapper/31 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x00001000
[ 98.619967] Call Trace:
[ 98.619967] [<000001b02ca87d98>] 0x1b02ca87d98
[ 98.619969] rcu: rcu_sched kthread timer wakeup didn't happen for 69113 jiffies! g-1019 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[ 98.620052] rcu: Possible timer handling issue on cpu=3 timer-softirq=12
[ 98.620055] rcu: rcu_sched kthread starved for 69116 jiffies! g-1019 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
[ 98.620058] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[ 98.620060] rcu: RCU grace-period kthread stack dump:
[ 98.620062] task:rcu_sched state:I stack:0 pid:15 tgid:15 ppid:2 task_flags:0x208040 flags:0x00000000
[ 98.620067] Call Trace:
[ 98.620068] [<000002302d9d58ac>] __schedule+0x35c/0x850
[ 98.620077] [<000002302d9d5ddc>] schedule+0x3c/0xc0
[ 98.620080] [<000002302d9de418>] schedule_timeout+0x88/0x120
[ 98.620084] [<000002302cbfd29a>] rcu_gp_fqs_loop+0x69a/0x8a0
[ 98.620090] [<000002302cc03426>] rcu_gp_kthread+0x2d6/0x340
[ 98.620093] [<000002302cb82d18>] kthread+0x148/0x170
[ 98.620098] [<000002302cb01ffc>] __ret_from_fork+0x3c/0x240
[ 98.620103] [<000002302d9dfb82>] ret_from_fork+0xa/0x30
[ 98.620106] rcu: Stack dump where RCU GP kthread last ran:
[ 98.620108] Task dump for CPU 3:
[ 98.620110] task:swapper/3 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x00001000
[ 98.620114] Call Trace:
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-03 18:31 [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick) Mikhail Zaslonko
@ 2026-03-04 16:13 ` Mikhail Zaslonko
2026-03-05 7:49 ` Heiko Carstens
0 siblings, 1 reply; 11+ messages in thread
From: Mikhail Zaslonko @ 2026-03-04 16:13 UTC (permalink / raw)
To: linux-next, linux-s390, Ingo Molnar
Cc: Heiko Carstens, Alexander Egorenkov, Peter Zijlstra,
Thomas Gleixner
Sorry, forgot to Cc a few people.
Adding scheduler maintainers.
Thanks,
Mikhail Zaslonko
On 03-Mar-26 19:31, Mikhail Zaslonko wrote:
> Hello,
>
> we have kernel-next boot hang on s390 starting next-20260302.
>
> I bisected it in linux-next to:
>
> c50f05bd3c4e ("Merge branch into tip/master: 'sched/hrtick'")
>
> Good:
> 72a2ab46f045
> d50da4b5915f (2nd parent: sched/hrtick branch)
>
> Bad:
> c50f05bd3c4e (merge commit)
>
> Environment:
> - s390 under z/VM
> - many CPUs defined (32+)
> - boot hangs early with RCU stall (see boot log excerpt below)
>
>
>
> [ 0.953192] Freeing unused kernel image (initmem) memory: 2520K
> [ 0.962509] Write protected read-only-after-init data: 188k
> [ 0.962884] Checked W+X mappings: passed, no W+X pages found
> [ 0.962888] Run /init as init process
> P+q6E616D65 \[ 29.503589] systemd[1]: Inserted module 'autofs4'
>
> hangs here, no logs ...
>
> [ 98.619693] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> [ 98.619708] rcu: 4-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=25/26 fqs=0 (false positive?)
> [ 98.619715] rcu: 5-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=25/26 fqs=0 (false positive?)
> [ 98.619720] rcu: 6-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=23/24 fqs=0 (false positive?)
> [ 98.619724] rcu: 7-...!: (1 GPs behind) idle=04e8/0/0x0 softirq=28/29 fqs=0 (false positive?)
> [ 98.619729] rcu: 8-...!: (1 GPs behind) idle=0568/0/0x0 softirq=27/28 fqs=0 (false positive?)
> [ 98.619733] rcu: 9-...!: (1 GPs behind) idle=04e8/0/0x0 softirq=35/36 fqs=0 (false positive?)
> [ 98.619737] rcu: 10-...!: (1 GPs behind) idle=0518/0/0x0 softirq=33/34 fqs=0 (false positive?)
> [ 98.619742] rcu: 11-...!: (1 GPs behind) idle=0448/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619745] rcu: 12-...!: (1 GPs behind) idle=0488/0/0x0 softirq=23/24 fqs=0 (false positive?)
> [ 98.619749] rcu: 13-...!: (1 GPs behind) idle=04a8/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619753] rcu: 14-...!: (0 ticks this GP) idle=0498/0/0x0 softirq=26/26 fqs=0 (false positive?)
> [ 98.619758] rcu: 15-...!: (1 GPs behind) idle=0448/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619762] rcu: 16-...!: (1 GPs behind) idle=0458/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619766] rcu: 17-...!: (1 GPs behind) idle=0438/0/0x0 softirq=21/22 fqs=0 (false positive?)
> [ 98.619770] rcu: 18-...!: (1 GPs behind) idle=03e8/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619774] rcu: 19-...!: (1 GPs behind) idle=0458/0/0x0 softirq=22/23 fqs=0 (false positive?)
> [ 98.619778] rcu: 20-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=24/25 fqs=0 (false positive?)
> [ 98.619782] rcu: 21-...!: (1 GPs behind) idle=04b8/0/0x0 softirq=23/24 fqs=0 (false positive?)
> [ 98.619786] rcu: 22-...!: (1 GPs behind) idle=04d8/0/0x0 softirq=23/24 fqs=0 (false positive?)
> [ 98.619790] rcu: 23-...!: (1 GPs behind) idle=0b68/0/0x0 softirq=29/30 fqs=0 (false positive?)
> [ 98.619794] rcu: 24-...!: (1 GPs behind) idle=0508/0/0x0 softirq=25/26 fqs=0 (false positive?)
> [ 98.619798] rcu: 25-...!: (1 GPs behind) idle=10c8/0/0x0 softirq=38/39 fqs=0 (false positive?)
> [ 98.619802] rcu: 28-...!: (1 GPs behind) idle=0518/0/0x0 softirq=25/26 fqs=0 (false positive?)
> [ 98.619806] rcu: 30-...!: (1 GPs behind) idle=0518/0/0x0 softirq=20/21 fqs=0 (false positive?)
> [ 98.619810] rcu: 31-...!: (1 GPs behind) idle=0528/0/0x0 softirq=23/24 fqs=0 (false positive?)
> [ 98.619815] rcu: (detected by 0, t=69116 jiffies, g=-1019, q=494 ncpus=32)
> [ 98.619818] Task dump for CPU 4:
> [ 98.619820] task:swapper/4 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x0
> [ 98.619825] Call Trace:
> [ 98.619828] [<000001b02c9afd98>] 0x1b02c9afd98
> [ 98.619834] Task dump for CPU 5:
> [ 98.619835] task:swapper/5 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x0
> [ 98.619838] Call Trace:
> [ 98.619839] [<000001b02c9b7d98>] 0x1b02c9b7d98
> [ 98.619841] Task dump for CPU 6:
>
> ...
>
> [ 98.619963] Task dump for CPU 31:
> [ 98.619964] task:swapper/31 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x00001000
> [ 98.619967] Call Trace:
> [ 98.619967] [<000001b02ca87d98>] 0x1b02ca87d98
> [ 98.619969] rcu: rcu_sched kthread timer wakeup didn't happen for 69113 jiffies! g-1019 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
> [ 98.620052] rcu: Possible timer handling issue on cpu=3 timer-softirq=12
> [ 98.620055] rcu: rcu_sched kthread starved for 69116 jiffies! g-1019 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
> [ 98.620058] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
> [ 98.620060] rcu: RCU grace-period kthread stack dump:
> [ 98.620062] task:rcu_sched state:I stack:0 pid:15 tgid:15 ppid:2 task_flags:0x208040 flags:0x00000000
> [ 98.620067] Call Trace:
> [ 98.620068] [<000002302d9d58ac>] __schedule+0x35c/0x850
> [ 98.620077] [<000002302d9d5ddc>] schedule+0x3c/0xc0
> [ 98.620080] [<000002302d9de418>] schedule_timeout+0x88/0x120
> [ 98.620084] [<000002302cbfd29a>] rcu_gp_fqs_loop+0x69a/0x8a0
> [ 98.620090] [<000002302cc03426>] rcu_gp_kthread+0x2d6/0x340
> [ 98.620093] [<000002302cb82d18>] kthread+0x148/0x170
> [ 98.620098] [<000002302cb01ffc>] __ret_from_fork+0x3c/0x240
> [ 98.620103] [<000002302d9dfb82>] ret_from_fork+0xa/0x30
> [ 98.620106] rcu: Stack dump where RCU GP kthread last ran:
> [ 98.620108] Task dump for CPU 3:
> [ 98.620110] task:swapper/3 state:R running task stack:0 pid:0 tgid:0 ppid:1 task_flags:0x4200042 flags:0x00001000
> [ 98.620114] Call Trace:
>
>
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-04 16:13 ` Mikhail Zaslonko
@ 2026-03-05 7:49 ` Heiko Carstens
2026-03-05 12:12 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Heiko Carstens @ 2026-03-05 7:49 UTC (permalink / raw)
To: Mikhail Zaslonko
Cc: linux-next, linux-s390, Ingo Molnar, Alexander Egorenkov,
Peter Zijlstra, Thomas Gleixner, Paul E. McKenney
On Wed, Mar 04, 2026 at 05:13:56PM +0100, Mikhail Zaslonko wrote:
> Sorry, forgot to Cc a few people.
> Adding scheduler maintainers.
>
> Thanks,
> Mikhail Zaslonko
>
> On 03-Mar-26 19:31, Mikhail Zaslonko wrote:
> > Hello,
> >
> > we have kernel-next boot hang on s390 starting next-20260302.
> >
> > I bisected it in linux-next to:
> >
> > c50f05bd3c4e ("Merge branch into tip/master: 'sched/hrtick'")
> >
> > Good:
> > 72a2ab46f045
> > d50da4b5915f (2nd parent: sched/hrtick branch)
> >
> > Bad:
> > c50f05bd3c4e (merge commit)
> >
> > Environment:
> > - s390 under z/VM
> > - many CPUs defined (32+)
> > - boot hangs early with RCU stall (see boot log excerpt below)
linux-next is currently completely broken because of this.
Looking at one of the numerous dumps of linux-next from 20260303 I can
see that the system hangs early waiting for synchronize_rcu_normal()
to finish:
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 3b7dff6c700 RU 0.0 0 0 [swapper/0]
> 0 0 1 80afc800 RU 0.0 0 0 [swapper/1]
> 0 0 2 80afa400 RU 0.0 0 0 [swapper/2]
> 0 0 3 80b34800 RU 0.0 0 0 [swapper/3]
> 0 0 4 80b32400 RU 0.0 0 0 [swapper/4]
> 0 0 5 80b30000 RU 0.0 0 0 [swapper/5]
> 0 0 6 80b14800 RU 0.0 0 0 [swapper/6]
> 0 0 7 80b50000 RU 0.0 0 0 [swapper/7]
> 0 0 8 80b3c800 RU 0.0 0 0 [swapper/8]
1 0 3 80a80000 UN 0.0 0 0 swapper/0
Everything is idle. PID 1 is in uninterruptible sleep:
crash> bt 1
PID: 1 TASK: 80a80000 CPU: 3 COMMAND: "swapper/0"
#0 [337de307a50] __schedule at 3b7df34708c
#1 [337de307ac8] schedule at 3b7df3475ec
#2 [337de307af8] schedule_timeout at 3b7df34fd78
#3 [337de307b80] __wait_for_common at 3b7df348482
#4 [337de307c18] wait_for_completion_state at 3b7df348658
#5 [337de307c38] __wait_rcu_gp at 3b7de561da8
#6 [337de307c98] synchronize_rcu_normal at 3b7de56ba12
#7 [337de307d78] kern_unmount at 3b7de8a8f86
#8 [337de307da8] do_sysctl_args at 3b7de94f2fa
#9 [337de307e20] kernel_init at 3b7df3448b0
#10 [337de307e38] __ret_from_fork at 3b7de46dffc
#11 [337de307ea0] ret_from_fork at 3b7df3514b2
The corresponding rcu_synchronize structure on the stack:
crash> struct rcu_synchronize 0x337de307d80
struct rcu_synchronize {
head = {
next = 0x822a62a8,
func = 0x3b7de561a90 <wakeme_after_rcu>
},
completion = {
done = 0,
wait = {
lock = {
raw_lock = {
lock = 0
}
},
task_list = {
next = 0x337de307c50,
prev = 0x337de307c50
}
}
},
oldstate = {
rgos_norm = 0,
rgos_exp = 0
}
}
Any idea?
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 7:49 ` Heiko Carstens
@ 2026-03-05 12:12 ` Peter Zijlstra
2026-03-05 12:35 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2026-03-05 12:12 UTC (permalink / raw)
To: Heiko Carstens
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 08:49:22AM +0100, Heiko Carstens wrote:
> On Wed, Mar 04, 2026 at 05:13:56PM +0100, Mikhail Zaslonko wrote:
> > Sorry, forgot to Cc a few people.
> > Adding scheduler maintainers.
> >
> > Thanks,
> > Mikhail Zaslonko
> >
> > On 03-Mar-26 19:31, Mikhail Zaslonko wrote:
> > > Hello,
> > >
> > > we have kernel-next boot hang on s390 starting next-20260302.
> > >
> > > I bisected it in linux-next to:
> > >
> > > c50f05bd3c4e ("Merge branch into tip/master: 'sched/hrtick'")
> > >
> > > Good:
> > > 72a2ab46f045
> > > d50da4b5915f (2nd parent: sched/hrtick branch)
> > >
> > > Bad:
> > > c50f05bd3c4e (merge commit)
> > >
> > > Environment:
> > > - s390 under z/VM
> > > - many CPUs defined (32+)
> > > - boot hangs early with RCU stall (see boot log excerpt below)
>
> linux-next is currently completely broken because of this.
Turns out, that aside from something weird with ACPI PCI routing, these
two patches:
https://lkml.kernel.org/r/87bjh4zies.ffs@tglx
https://lkml.kernel.org/r/87cy1jsa4m.ffs@tglx
Make tip/sched/core, which includes:
1b8b1bb2a2fa (tip/sched/core) Merge branch 'linus' into sched/core, to resolve conflicts
c1455a120f7e Merge branch 'sched/hrtick'
work reliably on my SPR. But since you don't have
CLOCK_SOURCE_HAS_COUPLED_CLOCK_EVENT (yet) this should not affect you.
> Any idea?
Well, that all looks like timers are going missing. Which matches with
Sven saying that disabling HRTIMER_REARM_DEFERRED makes it go again.
I'm just not sure I can see why things would break between
sched/hrtick (GOOD) and 1b8b1bb2a2fa (BAD).
Looking at the diff (eg git diff d50da4b5915f..c50f05bd3c4e) show some
idle time changes to s390 and various kernel/ changes, but nothing that
stands out to me :/
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 12:12 ` Peter Zijlstra
@ 2026-03-05 12:35 ` Peter Zijlstra
2026-03-05 12:45 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2026-03-05 12:35 UTC (permalink / raw)
To: Heiko Carstens
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 01:12:01PM +0100, Peter Zijlstra wrote:
> > Any idea?
>
> Well, that all looks like timers are going missing. Which matches with
> Sven saying that disabling HRTIMER_REARM_DEFERRED makes it go again.
>
> I'm just not sure I can see why things would break between
> sched/hrtick (GOOD) and 1b8b1bb2a2fa (BAD).
>
> Looking at the diff (eg git diff d50da4b5915f..c50f05bd3c4e) show some
> idle time changes to s390 and various kernel/ changes, but nothing that
> stands out to me :/
That s390 idle time code..
0d785e2c324c ("s390/idle: Fix cpu idle exit cpu time accounting")
That moves this_cpu_add() from inside irq_enter_rcu() / irq_exit_rcu()
to outside of it.
Your this_cpu_add() as preempt_enable(), which does a preemption check.
Moving that before irq_enter_rcu() means it doesn't see HARDIRQ_OFFSET
in preempt_count(). As such, it might actually call into schedule() from
hardirq context.
Or am I missing something?
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 12:35 ` Peter Zijlstra
@ 2026-03-05 12:45 ` Peter Zijlstra
2026-03-05 13:07 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2026-03-05 12:45 UTC (permalink / raw)
To: Heiko Carstens
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 01:35:05PM +0100, Peter Zijlstra wrote:
> On Thu, Mar 05, 2026 at 01:12:01PM +0100, Peter Zijlstra wrote:
>
> > > Any idea?
> >
> > Well, that all looks like timers are going missing. Which matches with
> > Sven saying that disabling HRTIMER_REARM_DEFERRED makes it go again.
> >
> > I'm just not sure I can see why things would break between
> > sched/hrtick (GOOD) and 1b8b1bb2a2fa (BAD).
> >
> > Looking at the diff (eg git diff d50da4b5915f..c50f05bd3c4e) show some
> > idle time changes to s390 and various kernel/ changes, but nothing that
> > stands out to me :/
>
> That s390 idle time code..
>
> 0d785e2c324c ("s390/idle: Fix cpu idle exit cpu time accounting")
>
> That moves this_cpu_add() from inside irq_enter_rcu() / irq_exit_rcu()
> to outside of it.
>
> Your this_cpu_add() as preempt_enable(), which does a preemption check.
>
> Moving that before irq_enter_rcu() means it doesn't see HARDIRQ_OFFSET
> in preempt_count(). As such, it might actually call into schedule() from
> hardirq context.
>
> Or am I missing something?
N/m, it turns into __this_cpu_add() and that doesn't have
preempt_enable().
00d8b035eb71 ("s390/idle: Slightly optimize idle time accounting")
Is actually a correctness fix afaict.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 12:45 ` Peter Zijlstra
@ 2026-03-05 13:07 ` Peter Zijlstra
2026-03-05 15:02 ` Heiko Carstens
0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2026-03-05 13:07 UTC (permalink / raw)
To: Heiko Carstens
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 01:45:01PM +0100, Peter Zijlstra wrote:
> On Thu, Mar 05, 2026 at 01:35:05PM +0100, Peter Zijlstra wrote:
> > On Thu, Mar 05, 2026 at 01:12:01PM +0100, Peter Zijlstra wrote:
> >
> > > > Any idea?
> > >
> > > Well, that all looks like timers are going missing. Which matches with
> > > Sven saying that disabling HRTIMER_REARM_DEFERRED makes it go again.
> > >
> > > I'm just not sure I can see why things would break between
> > > sched/hrtick (GOOD) and 1b8b1bb2a2fa (BAD).
> > >
> > > Looking at the diff (eg git diff d50da4b5915f..c50f05bd3c4e) show some
> > > idle time changes to s390 and various kernel/ changes, but nothing that
> > > stands out to me :/
> >
> > That s390 idle time code..
> >
> > 0d785e2c324c ("s390/idle: Fix cpu idle exit cpu time accounting")
> >
> > That moves this_cpu_add() from inside irq_enter_rcu() / irq_exit_rcu()
> > to outside of it.
> >
> > Your this_cpu_add() as preempt_enable(), which does a preemption check.
> >
> > Moving that before irq_enter_rcu() means it doesn't see HARDIRQ_OFFSET
> > in preempt_count(). As such, it might actually call into schedule() from
> > hardirq context.
> >
> > Or am I missing something?
>
> N/m, it turns into __this_cpu_add() and that doesn't have
> preempt_enable().
>
> 00d8b035eb71 ("s390/idle: Slightly optimize idle time accounting")
>
> Is actually a correctness fix afaict.
Another change is that you clear I and E in the PSW bit before
irq_enter_rcu(), which, per:
7e641e52cf5f ("softirq: Prepare for deferred hrtimer rearming")
can re-arm the timer.
So where previously it would re-arm and still have the I/E bits set, so
the timer could fire, they are now disabled.
I really don't know if this is a problem; I'm clutching at s390 straws
here that I really don't know much about.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 13:07 ` Peter Zijlstra
@ 2026-03-05 15:02 ` Heiko Carstens
2026-03-05 17:24 ` Mikhail Zaslonko
0 siblings, 1 reply; 11+ messages in thread
From: Heiko Carstens @ 2026-03-05 15:02 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 02:07:25PM +0100, Peter Zijlstra wrote:
> > > Moving that before irq_enter_rcu() means it doesn't see HARDIRQ_OFFSET
> > > in preempt_count(). As such, it might actually call into schedule() from
> > > hardirq context.
> > >
> > > Or am I missing something?
> >
> > N/m, it turns into __this_cpu_add() and that doesn't have
> > preempt_enable().
> >
> > 00d8b035eb71 ("s390/idle: Slightly optimize idle time accounting")
> >
> > Is actually a correctness fix afaict.
Yes. Those dependencies become quite subtle when calling early into C code.
> Another change is that you clear I and E in the PSW bit before
> irq_enter_rcu(), which, per:
>
> 7e641e52cf5f ("softirq: Prepare for deferred hrtimer rearming")
>
> can re-arm the timer.
>
> So where previously it would re-arm and still have the I/E bits set, so
> the timer could fire, they are now disabled.
>
> I really don't know if this is a problem; I'm clutching at s390 straws
> here that I really don't know much about.
That's the old PSW where the interrupt happened, not the one with which the
CPU is running with, and shouldn't have any effect.
...but reverting that commit actually does fix it for me. I don't see
immediately any code which checks if the _previous_ context had irqs enabled
or disabled, since that is the only difference.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 15:02 ` Heiko Carstens
@ 2026-03-05 17:24 ` Mikhail Zaslonko
2026-03-05 19:48 ` Heiko Carstens
0 siblings, 1 reply; 11+ messages in thread
From: Mikhail Zaslonko @ 2026-03-05 17:24 UTC (permalink / raw)
To: Heiko Carstens, Peter Zijlstra
Cc: linux-next, linux-s390, Ingo Molnar, Alexander Egorenkov,
Thomas Gleixner, Paul E. McKenney
On 05-Mar-26 16:02, Heiko Carstens wrote:
> On Thu, Mar 05, 2026 at 02:07:25PM +0100, Peter Zijlstra wrote:
>>>> Moving that before irq_enter_rcu() means it doesn't see HARDIRQ_OFFSET
>>>> in preempt_count(). As such, it might actually call into schedule() from
>>>> hardirq context.
>>>>
>>>> Or am I missing something?
>>>
>>> N/m, it turns into __this_cpu_add() and that doesn't have
>>> preempt_enable().
>>>
>>> 00d8b035eb71 ("s390/idle: Slightly optimize idle time accounting")
>>>
>>> Is actually a correctness fix afaict.
>
> Yes. Those dependencies become quite subtle when calling early into C code.
>
>> Another change is that you clear I and E in the PSW bit before
>> irq_enter_rcu(), which, per:
>>
>> 7e641e52cf5f ("softirq: Prepare for deferred hrtimer rearming")
>>
>> can re-arm the timer.
>>
>> So where previously it would re-arm and still have the I/E bits set, so
>> the timer could fire, they are now disabled.
>>
>> I really don't know if this is a problem; I'm clutching at s390 straws
>> here that I really don't know much about.
>
> That's the old PSW where the interrupt happened, not the one with which the
> CPU is running with, and shouldn't have any effect.
>
> ...but reverting that commit actually does fix it for me.
For me it doesn't. But reverting this one helps:
15dd3a948855 ("hrtimer: Push reprogramming timers into the interrupt return path")
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 17:24 ` Mikhail Zaslonko
@ 2026-03-05 19:48 ` Heiko Carstens
2026-03-06 8:41 ` Peter Zijlstra
0 siblings, 1 reply; 11+ messages in thread
From: Heiko Carstens @ 2026-03-05 19:48 UTC (permalink / raw)
To: Mikhail Zaslonko
Cc: Peter Zijlstra, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 06:24:48PM +0100, Mikhail Zaslonko wrote:
> >> Another change is that you clear I and E in the PSW bit before
> >> irq_enter_rcu(), which, per:
> >>
> >> 7e641e52cf5f ("softirq: Prepare for deferred hrtimer rearming")
> >>
> >> can re-arm the timer.
> >>
> >> So where previously it would re-arm and still have the I/E bits set, so
> >> the timer could fire, they are now disabled.
> >>
> >> I really don't know if this is a problem; I'm clutching at s390 straws
> >> here that I really don't know much about.
> >
> > That's the old PSW where the interrupt happened, not the one with which the
> > CPU is running with, and shouldn't have any effect.
> >
> > ...but reverting that commit actually does fix it for me.
>
> For me it doesn't. But reverting this one helps:
> 15dd3a948855 ("hrtimer: Push reprogramming timers into the interrupt return path")
Mikhail, which commit did you revert? The above discussion is about commit
d8b5cf9c6314 ("s390/irq/idle: Remove psw bits early"), which is not
explicitly mentioned.
But it is indeed broken: irqentry_exit() has a regs_irqs_disabled() check,
which I broke with this commit - but "only" for idle exit.
Result: no hrtimer_rearm_deferred() being called on idle exit.
Oh well. Peter, thanks for pointing to this broken commit. Will be reverted.
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick)
2026-03-05 19:48 ` Heiko Carstens
@ 2026-03-06 8:41 ` Peter Zijlstra
0 siblings, 0 replies; 11+ messages in thread
From: Peter Zijlstra @ 2026-03-06 8:41 UTC (permalink / raw)
To: Heiko Carstens
Cc: Mikhail Zaslonko, linux-next, linux-s390, Ingo Molnar,
Alexander Egorenkov, Thomas Gleixner, Paul E. McKenney
On Thu, Mar 05, 2026 at 08:48:21PM +0100, Heiko Carstens wrote:
> Mikhail, which commit did you revert? The above discussion is about commit
> d8b5cf9c6314 ("s390/irq/idle: Remove psw bits early"), which is not
> explicitly mentioned.
>
> But it is indeed broken: irqentry_exit() has a regs_irqs_disabled() check,
> which I broke with this commit - but "only" for idle exit.
> Result: no hrtimer_rearm_deferred() being called on idle exit.
>
> Oh well. Peter, thanks for pointing to this broken commit. Will be reverted.
No problem; digging through unknown arch code is what keeps you sharp
;-)
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2026-03-06 8:48 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-03 18:31 [linux-next][s390] Boot hang after merge c50f05bd3c4e (sched/hrtick) Mikhail Zaslonko
2026-03-04 16:13 ` Mikhail Zaslonko
2026-03-05 7:49 ` Heiko Carstens
2026-03-05 12:12 ` Peter Zijlstra
2026-03-05 12:35 ` Peter Zijlstra
2026-03-05 12:45 ` Peter Zijlstra
2026-03-05 13:07 ` Peter Zijlstra
2026-03-05 15:02 ` Heiko Carstens
2026-03-05 17:24 ` Mikhail Zaslonko
2026-03-05 19:48 ` Heiko Carstens
2026-03-06 8:41 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox