The Power9 host booting problem with OpenBMC kernel 5.7.x

["Alexander A. Filippov" <a.filippov@yadro.com>]

[-- Attachment #1: Type: text/plain, Size: 3203 bytes --]

Since the kernel in OpenBMC was updated to 5.7.x we have a problem with the P9
hosts booting.
On host with one Power9 CPU the failure happens during the Petitboot is trying
to initialize the network and it leads to host restarts.
On host with two Power9 CPU the same failure happens during OS booting. It
increases boot time, but at the end the host OS is completely started.

In both cases the error looks like:

[   22.564986] rcu: INFO: rcu_sched self-detected stall on CPU
[   22.565013] rcu:     7-....: (2155 ticks this GP) idle=1da/0/0x3 softirq=92/126 fqs=1050 
[   22.565048]  (t=2100 jiffies g=-1131 q=31)
[   22.565071] NMI backtrace for cpu 7
[   22.565084] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 5.6.16 #2
[   22.565110] Call Trace:
[   22.565134] [c000003ff463f280] [c000000000c457d8] dump_stack+0xbc/0x104 (unreliable)
[   22.565165] [c000003ff463f2c0] [c000000000c50af4] nmi_cpu_backtrace+0x104/0x130
[   22.565213] [c000003ff463f340] [c000000000c50bf4] nmi_trigger_cpumask_backtrace+0xd4/0x1d0
[   22.565244] [c000003ff463f3e0] [c0000000000679a8] arch_trigger_cpumask_backtrace+0x28/0x40
[   22.565293] [c000003ff463f400] [c0000000001c504c] rcu_dump_cpu_stacks+0xe0/0x154
[   22.565342] [c000003ff463f450] [c0000000001c35e8] rcu_sched_clock_irq+0x408/0xaa0
[   22.565381] [c000003ff463f530] [c0000000001d2774] update_process_times+0x44/0x90
[   22.565410] [c000003ff463f560] [c0000000001e81c8] tick_periodic+0xf8/0x120
[   22.565448] [c000003ff463f590] [c0000000001e822c] tick_handle_periodic+0x3c/0xd0
[   22.565488] [c000003ff463f5d0] [c00000000002cad0] timer_interrupt+0x1d0/0x300
[   22.565528] [c000003ff463f630] [c00000000000e6f8] fast_exception_return+0x1a8/0x1cc
[   22.565570] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
                   LR = arch_local_irq_restore+0x5c/0x90
[   22.565605] [c000003ff463f930] [c000000000164220] vtime_account_irq_enter+0x40/0x70 (unreliable)
[   22.565657] [c000003ff463f950] [c000000000c6d038] __do_softirq+0xd8/0x474
[   22.565704] [c000003ff463fa50] [c000000000121968] irq_exit+0x88/0x100
[   22.565741] [c000003ff463fa80] [c00000000002cbc4] timer_interrupt+0x2c4/0x300
[   22.565781] [c000003ff463fae0] [c00000000000e6f8] fast_exception_return+0x1a8/0x1cc
[   22.565831] --- interrupt: 901 at arch_local_irq_restore+0x7c/0x90
                   LR = arch_local_irq_restore+0x48/0x90
[   22.565896] [c000003ff463fde0] [0000000000000080] 0x80 (unreliable)
[   22.565934] [c000003ff463fe00] [c0000000000242a8] arch_cpu_idle+0xb8/0x150
[   22.565971] [c000003ff463fe30] [c000000000c6c234] default_idle_call+0x64/0x78
[   22.566020] [c000003ff463fe50] [c000000000164ee8] do_idle+0x208/0x3f0
[   22.566067] [c000003ff463fed0] [c000000000165304] cpu_startup_entry+0x44/0x50
[   22.566105] [c000003ff463ff00] [c0000000000542b4] start_secondary+0x614/0x620
[   22.566145] [c000003ff463ff90] [c00000000000b354] start_secondary_prolog+0x10/0x14


So, I have two questions:
- Could you please, check if Romulus is also affected by this issue?
- Do you have any idea what is going wrong?

I've attached the tarball with full logs.
- poopsy is a system with two Power9 CPU
- whoopsy is a system with one Power9 CPU

--
Regards,
Alexander

[-- Attachment #2: nicole-broken-merge.tar.gz --]
[-- Type: application/gzip, Size: 90810 bytes --]