Re: Console Hang on SMP Secondary CPU Bringup

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2026-09-23 12:02:31 [-0400], Derek Barbosa wrote:
> In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
> defconfig and additional HV drivers enabled), installing, then booting a kernel
> in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
> boot may result in a unrecoverable "hang"

Does this also occur in v6.6-rt15? If so, can you check v7.3-rc5 and if
it does not occur, try other RT kernels where this started happening?
This HyperV is something I can't try here, can I?

…
> In any case, the system is completely unresponsive over the any serial consoles,
> responding to neither SysRQs nor NMIs, even with the appropriate cmdline
> arguments to invoke such behavior.

Does this run in qemu where you could attach gdb and check what the CPUs
are doing?

> It appears as though the behavior in question has been bisected to af3a8162581e
> ("serial: 8250: Switch to nbcon console")
> 
> <snip>
> $ git describe --contains af3a8162581e
> 
> v6.6.156-rt79-rebase~63
> 
> $ git show v6.6.156-rt79
> </snip>
> 
> and is not reproducible prior to this point (it has not been bisectable to any
> HV/HyperV/Xen/etc commits _yet_).
> 
> IIRC, this commit was reverted in mainline for the time being, as a significant
> amount of work was/is being done in tty/serial, etc...

As far as I remember, it was dropped because other driver fall apart.

> It is not entirely clear what effect the commit introduces with regard to the
> smp secondary CPU masking/bringup, and why it would cause a
> seemingly-unrecoverable state prior to the cpuhp callbacks.

It could be unrelated to the driver, just some printk corner case.

> IIUC (and from what I have seen in the traces) at this point in the init
> process, we've disabled earlycon/earlyserial, and have brought up nbcon.
…
> I have seen in some cases where nbcon 8250 can be slightly slower at boot time,
> which could introduce the possibility of some contention between the increased
> console verbosity and replacement of send_IPI_all/self() with
> hv_send_ipi_all/self() (this is just conjecturing).

Not sure how this is related. Maybe it is a wakeup for the nbcon thread
on the remote CPU.

…
> Specifically, I was curious if serial8250_console_write_thread() could
> potentially be the culprit, especially given the fact that we are running in
> CONFIG_PREEMPT_VOLUNTARY as opposed to CONFIG_PREEMPT_RT.

So we are not doing PREEMPT_RT?

> My understanding is shaky at best here, but specifically:
> 
> <snip>
> 	while (!nbcon_enter_unsafe(wctxt))
> 		nbcon_reacquire(wctxt);
> 
> 	/* Finally, wait for transmitter to become empty and restore IER. */
> 	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
> 	if (em485) {
> 		mdelay(port->rs485.delay_rts_after_send);
> 		if (em485->tx_stopped)
> 			up->rs485_stop_tx(up);
> 	}
> 	serial_port_out(port, UART_IER, ier);
> </snip>
> 
> Is it possible that we just spin here? That we somehow lost ownership at some
> point during the secondary CPU startup sequence and cannot restore UART_IER?
> Would this explain the silent console + hang at PID1?

The regular printk path acquires the uart lock. If my memory serves me
right, the take over takes place in atomic context (say from panic()).

> If that is not the case (very plausible) -- what I am still very unsure of, is
> how the atomic 8250 implementation here could affect the rdmsr()s in
> save/get_fixed_ranges(), to the point of intermittently stalling in the Hyper-V
> VM (MTRR virtualization / SMT sibling) environment?

Me, too. It is just all callback to CPU0 with waiting until it is done.
If CPU1 stalls then CPU0 did not finish… CPU0 should be more or less
idle here.

…
> Many thanks in advance,

The printk version is a bit old but still functional :) If the problem
does not occur in a later kernel then could think about backporting that
printk version back to v6.6 (unless there is a single obvious cause).

Sebastian