Console Hang on SMP Secondary CPU Bringup

Console Hang on SMP Secondary CPU Bringup

[Derek Barbosa]

TLDR: In v6.6-rt-rebase booting a kernel in a paravirtualized VM environment can
result in hangs in the early smp/smpboot secondary CPU bringup.

Hello,

In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
defconfig and additional HV drivers enabled), installing, then booting a kernel
in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
boot may result in a unrecoverable "hang"

<snip>
[    0.644309] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz (family: 0x6, model: 0x55, stepping: 0x4)
[    0.644565] Performance Events: unsupported p6 CPU model 85 no PMU driver, software events only.
[    0.645350] signal: max sigframe size: 3632
[    0.646414] rcu: Hierarchical SRCU implementation.
[    0.646417] rcu:     Max phase no-delay instances is 400.
[    0.652387] NMI watchdog: Perf NMI watchdog permanently disabled
[    0.653406] smp: Bringing up secondary CPUs ...
</snip>

which is only recoverable via rebooting the VM (and hoping that it doesn't hang
again).

The issue is most reliably reproduced on smaller systems with a high enough
loglevel (in this case, it reproduces most reliably with 8 in the cmdline). In
my testing so far, 2vCPUs with 8GiB of memory have a fairly high "hit rate" of
~30% with the following reproducer:

$ for f in `seq 1 50`; do echo try $f; ssh $VM_IP sudo reboot; sleep 90s; done

a reboot script attached to a systemd timer/service also suffices.

In any case, the system is completely unresponsive over the any serial consoles,
responding to neither SysRQs nor NMIs, even with the appropriate cmdline
arguments to invoke such behavior.

It appears as though the behavior in question has been bisected to af3a8162581e
("serial: 8250: Switch to nbcon console")

<snip>
$ git describe --contains af3a8162581e

v6.6.156-rt79-rebase~63

$ git show v6.6.156-rt79
</snip>

and is not reproducible prior to this point (it has not been bisectable to any
HV/HyperV/Xen/etc commits _yet_).

IIRC, this commit was reverted in mainline for the time being, as a significant
amount of work was/is being done in tty/serial, etc...

It is not entirely clear what effect the commit introduces with regard to the
smp secondary CPU masking/bringup, and why it would cause a
seemingly-unrecoverable state prior to the cpuhp callbacks.

IIUC (and from what I have seen in the traces) at this point in the init
process, we've disabled earlycon/earlyserial, and have brought up nbcon.

<snip>
void start_kernel(void)
{
    ...

	/*
	 * HACK ALERT! This is early. We're enabling the console before
	 * we've done PCI setups etc, and console_init() must be aware of
	 * this. But we do want output early, in case something goes wrong.
	 */
	console_init();
	if (panic_later)
		panic("Too many boot %s vars at `%s'", panic_later,
		      panic_param);

	lockdep_init();

	/*
	 * Need to run this when irqs are enabled, because it wants
	 * to self-test [hard/soft]-irqs on/off lock inversion bugs
	 * too:
	 */
	locking_selftest();

	/* Do the rest non-__init'ed, we're now alive */
	arch_call_rest_init();
</snip>

it is only then that rest_init() is called, which

<snip>
void __init __weak __noreturn arch_call_rest_init(void)
{
	rest_init();
}

...

noinline void __ref __noreturn rest_init(void)
{
	struct task_struct *tsk;
	int pid;

	rcu_scheduler_starting();
	/*
	 * We need to spawn init first so that it obtains pid 1, however
	 * the init task will end up wanting to create kthreads, which, if
	 * we schedule it before we create kthreadd, will OOPS.
	 */
	pid = user_mode_thread(kernel_init, NULL, CLONE_FS);

...

static noinline void __init kernel_init_freeable(void)
{
	/* Now the scheduler is fully set up and can do blocking allocations */
	gfp_allowed_mask = __GFP_BITS_MASK;

	/*
	 * init can allocate pages on any node
	 */
	set_mems_allowed(node_states[N_MEMORY]);

	cad_pid = get_pid(task_pid(current));

	smp_prepare_cpus(setup_max_cpus);

	workqueue_init();

	init_mm_internals();

	rcu_init_tasks_generic();
	do_pre_smp_initcalls();
	lockup_detector_init();

	smp_init(); ## INIT secondary CPUS HERE

...
</snip>

I have seen in some cases where nbcon 8250 can be slightly slower at boot time,
which could introduce the possibility of some contention between the increased
console verbosity and replacement of send_IPI_all/self() with
hv_send_ipi_all/self() (this is just conjecturing).

I am unsure what the precise interaction between the console (at this stage of
the init) and smp/smpboot secondary CPU bringup is.

I went and added a few pr_info()'s (I am still hacking on a system with
tracefs and trace_printk setup for more info) and was able to retrieve the
following from the serial console prior to hanging:

<snip>
[    0.705259] smp: bringing up secondary cpus ...
[    0.705267] parallel true!
[    0.705271] smpboot: parallel cpu startup enabled: 0x80000000
[    0.705278] masks: smt_max=2 present=0-1 primary=0 first_present=0 first_primary=0
[    0.705285] primary block kick: tmp=0 ncpus=8192
[    0.705290] in cpuhp_bringup_mask, still alive before iterator macro.
[    0.705295] in cpu 0. ncpus: 8192 
[    0.712205] in cpuhp_bringup_mask, still alive before iterator macro.
[    0.712212] in cpu 0. ncpus: 8192 
[    0.714186] secondary block kick: tmp=0 ncpus=8191
[    0.714193] still alive before 2nd bringup_mask
[    0.714195] in cpuhp_bringup_mask, still alive before iterator macro.
[    0.714212] in cpu 1. ncpus: 8191 
[    0.718185] cpuhp up cpu1 state threads:prepare (1)
[    0.719271] cpuhp up cpu1 state perf:prepare (2)
[    0.720186] cpuhp up cpu1 state mm/page_alloc:pcp (33)
[    0.721184] cpuhp up cpu1 state random:prepare (40)
[    0.721191] cpuhp up cpu1 state workqueue:prepare (41)
[    0.723239] cpuhp up cpu1 state hrtimers:prepare (43)
[    0.723246] cpuhp up cpu1 state smpcfd:prepare (46)
[    0.725187] cpuhp up cpu1 state relay:prepare (47)
[    0.726183] cpuhp up cpu1 state rcu/tree:prepare (50)
[    0.726189] cpuhp up cpu1 state trace/rb:prepare (62)
[    0.728189] cpuhp up cpu1 state timers:prepare (68)
[    0.728206] cpuhp up cpu1 state cpu:kick_ap (91)
[    0.728213] smpboot: ++++++++++++++++++++=_---cpu up  1
[    0.728218] smpboot: still allive before mtrr_save_state
[    0.728221] getting cpumask in mtrr_save_state ### HANG HERE
</snip>

jumping down the rabbit-hole of cpuhp_bp_*_ap:

cpuhp_bringup_mask(mask, ncpus, CPUHP_BP_KICK_AP) kernel/cpu.c
└─ for_each_cpu(cpu, mask)
   └─ cpu_up(cpu, CPUHP_BP_KICK_AP)
      ├─ try_online_node() / cpu_maps_update_begin() / cpu_bootable()
      └─ _cpu_up(cpu, 0, CPUHP_BP_KICK_AP)
         ├─ cpus_write_lock()
         ├─ cpuhp_set_state()
         └─ cpuhp_up_callbacks(cpu, st, CPUHP_BP_KICK_AP)
            └─ cpuhp_invoke_callback_range(true, cpu, st, target)
               └─ __cpuhp_invoke_callback_range(true, cpu, st, target, false)
                  └─ while (cpuhp_next_state(...)) ...
                     └─ cpuhp_invoke_callback(cpu, CPUHP_BP_KICK_AP, true, NULL, NULL)
                        └─ cb = step->startup.single
                           ret = cb(cpu)
                           └─ cpuhp_kick_ap_alive(cpu)
                              └─ arch_cpuhp_kick_ap_alive(cpu, idle_thread_get(cpu))
                                 └─ smp_ops.kick_ap_alive(cpu, tidle)  arch/x86/kernel/smpboot.c
                                    └─ native_kick_ap(cpu, tidle)      arch/x86/kernel/smp.c:290
                                       ├─ apic->cpu_present_to_apicid(cpu)
                                       ├─ mtrr_save_state()

I understand that spamming a ton of pr_info's, especially in a latency-sensitive
path such as the cpu-hotplug-bootstrap paths, can widen timing windows
significantly, so please ignore the rambling...

Out of sheer curiosity (insanity?), while waiting for the debug kernel to
compile, I removed the smp_call_function_single() invocation of
mtrr_save_fixed_ranges() in mtrr_save_state() in one my experiments.

<snip>
void mtrr_save_state(void)
{
	int first_cpu;

	if (!mtrr_enabled())
		return;

	pr_info("getting first cpu in cpumask in mtrr_save_state");
	first_cpu = cpumask_first(cpu_online_mask);
	pr_info("saving range registers in cpu %u", first_cpu);
    /* COMMENTED OUT */
	//smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
}
</snip>

I could've been bleary eyed and lost track of "which kernel was running where"
-- but I was unable to reproduce the issue at all here.

Anyway, I am hoping to collect some meaningful results from the trace data. In the
meantime, I was revisiting the subject of the bisect and the changes introduced to
8250_port.c.

Specifically, I was curious if serial8250_console_write_thread() could
potentially be the culprit, especially given the fact that we are running in
CONFIG_PREEMPT_VOLUNTARY as opposed to CONFIG_PREEMPT_RT.

My understanding is shaky at best here, but specifically:

<snip>
	while (!nbcon_enter_unsafe(wctxt))
		nbcon_reacquire(wctxt);

	/* Finally, wait for transmitter to become empty and restore IER. */
	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
	if (em485) {
		mdelay(port->rs485.delay_rts_after_send);
		if (em485->tx_stopped)
			up->rs485_stop_tx(up);
	}
	serial_port_out(port, UART_IER, ier);
</snip>

Is it possible that we just spin here? That we somehow lost ownership at some
point during the secondary CPU startup sequence and cannot restore UART_IER?
Would this explain the silent console + hang at PID1?

If that is not the case (very plausible) -- what I am still very unsure of, is
how the atomic 8250 implementation here could affect the rdmsr()s in
save/get_fixed_ranges(), to the point of intermittently stalling in the Hyper-V
VM (MTRR virtualization / SMT sibling) environment?

<snip>
static void get_fixed_ranges(mtrr_type *frs)
{
	unsigned int *p = (unsigned int *)frs;
	int i;

	k8_check_syscfg_dram_mod_en();

	rdmsr(MSR_MTRRfix64K_00000, p[0], p[1]);

	for (i = 0; i < 2; i++)
		rdmsr(MSR_MTRRfix16K_80000 + i, p[2 + i * 2], p[3 + i * 2]);
	for (i = 0; i < 8; i++)
		rdmsr(MSR_MTRRfix4K_C0000 + i, p[6 + i * 2], p[7 + i * 2]);
}

void mtrr_save_fixed_ranges(void *info)
{
	return;
	if (boot_cpu_has(X86_FEATURE_MTRR))
		get_fixed_ranges(mtrr_state.fixed_ranges);
}
</snip>

I would appreciate the pointers. I am also very OOTL wrt patches and such that
have hit the mailing lists lately.

Many thanks in advance,

-- 
Derek <debarbos@redhat.com>

Re: Console Hang on SMP Secondary CPU Bringup

[John Ogness]

Hi Derek,

On 2026-09-23, Derek Barbosa <debarbos@redhat.com> wrote:
> It appears as though the behavior in question has been bisected to af3a8162581e
> ("serial: 8250: Switch to nbcon console")

Since you are hacking up the code anyway, here a few things you could
try.

First off, it might be helpful to enable CONFIG_PRINTK_CALLER. This will
print the CPU-ID that the printk message originated from.

To be sure it is timing related due to the nbcon atomic printing, you
could try short-circuiting:

	drivers/tty/serial/8250/8250_port.c:serial8250_console_write_atomic()

so that it immediately does

	return true;

Obviously you will not see any kernel messages on boot, but the machine
should boot and with dmesg you should still see all the messages.

Assuming that boots, you could change the short-circuit in
serial8250_console_write_atomic() to simulate the timing with:

	mdelay(7);
	return true;

If it fails to boot, then we at least know it has nothing to do with the
8250 and is instead either delays in time-sensitive sections or some
sort of contention bug in the core nbcon code.

> <snip>
> 	while (!nbcon_enter_unsafe(wctxt))
> 		nbcon_reacquire(wctxt);
>
> 	/* Finally, wait for transmitter to become empty and restore IER. */
> 	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
> 	if (em485) {
> 		mdelay(port->rs485.delay_rts_after_send);
> 		if (em485->tx_stopped)
> 			up->rs485_stop_tx(up);
> 	}
> 	serial_port_out(port, UART_IER, ier);
> </snip>
>
> Is it possible that we just spin here?

That loop means that the console has lost ownership. A console can only
lose ownership if a WARN or panic were to occur at that moment. I assume
you have neither.

With QEMU it is easy to attach gdb and dump the kernel logs using the
gdb scripts. I don't know if Hyper-V has anything like this. But it
could help if there is one more printk message you are not seeing. When
printk() is called, it locklessly inserts the message and then tries to
print it to the console.

John Ogness

Re: Console Hang on SMP Secondary CPU Bringup

[Derek Barbosa]

Hi John,

On Mon, Sep 28, 2026 at 04:06:22PM +0206, John Ogness wrote:
> Hi Derek,
> 
> On 2026-09-23, Derek Barbosa <debarbos@redhat.com> wrote:
> > It appears as though the behavior in question has been bisected to af3a8162581e
> > ("serial: 8250: Switch to nbcon console")
> 
> Since you are hacking up the code anyway, here a few things you could
> try.
> 
> First off, it might be helpful to enable CONFIG_PRINTK_CALLER. This will
> print the CPU-ID that the printk message originated from.
> 
> To be sure it is timing related due to the nbcon atomic printing, you
> could try short-circuiting:
> 
> 	drivers/tty/serial/8250/8250_port.c:serial8250_console_write_atomic()
> 
> so that it immediately does
> 
> 	return true;
> 
> Obviously you will not see any kernel messages on boot, but the machine
> should boot and with dmesg you should still see all the messages.
> 
> Assuming that boots, you could change the short-circuit in
> serial8250_console_write_atomic() to simulate the timing with:
> 
> 	mdelay(7);
> 	return true;
> 
> If it fails to boot, then we at least know it has nothing to do with the
> 8250 and is instead either delays in time-sensitive sections or some
> sort of contention bug in the core nbcon code.

Thanks for the suggestions. I will give them a spin and report back when I can.

> 
> > <snip>
> > 	while (!nbcon_enter_unsafe(wctxt))
> > 		nbcon_reacquire(wctxt);
> >
> > 	/* Finally, wait for transmitter to become empty and restore IER. */
> > 	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
> > 	if (em485) {
> > 		mdelay(port->rs485.delay_rts_after_send);
> > 		if (em485->tx_stopped)
> > 			up->rs485_stop_tx(up);
> > 	}
> > 	serial_port_out(port, UART_IER, ier);
> > </snip>
> >
> > Is it possible that we just spin here?
> 
> That loop means that the console has lost ownership. A console can only
> lose ownership if a WARN or panic were to occur at that moment. I assume
> you have neither.

Thanks for the note. It was frankly just a wild guess, I was only "reviewing"
the code that was introduced by that specific commit. I figured someone else
would be a lot better at pointing out my logic errors :)

> 
> With QEMU it is easy to attach gdb and dump the kernel logs using the
> gdb scripts. I don't know if Hyper-V has anything like this. But it
> could help if there is one more printk message you are not seeing. When
> printk() is called, it locklessly inserts the message and then tries to
> print it to the console.

I am not sure _what_ other debugging tools Azure has, frankly. From what I've
seen/can-see, there aren't nearly as many knobs and buttons as you would get
from just running plain-old-HyperV on a supported host as opposed to the
multi-tenant "cloud" platform interface. I will take note of this if I stumble
upon something similar by chance.

> 
> John Ogness
> 
-- 
Derek <debarbos@redhat.com>

Re: Console Hang on SMP Secondary CPU Bringup

[Derek Barbosa]

Hi John,

On Mon, Sep 28, 2026 at 04:17:51PM -0400, Derek Barbosa wrote:
> Hi John,
> 
> On Mon, Sep 28, 2026 at 04:06:22PM +0206, John Ogness wrote:
> > Hi Derek,
> > 
> > On 2026-09-23, Derek Barbosa <debarbos@redhat.com> wrote:
> > > It appears as though the behavior in question has been bisected to af3a8162581e
> > > ("serial: 8250: Switch to nbcon console")
> > 
> > Since you are hacking up the code anyway, here a few things you could
> > try.
> > 
> > First off, it might be helpful to enable CONFIG_PRINTK_CALLER. This will
> > print the CPU-ID that the printk message originated from.
> > 
> > To be sure it is timing related due to the nbcon atomic printing, you
> > could try short-circuiting:
> > 
> > 	drivers/tty/serial/8250/8250_port.c:serial8250_console_write_atomic()
> > 
> > so that it immediately does
> > 
> > 	return true;
> > 

I will be running more trials of the reproducer to confirm, but so far, we still
hang. I wasn't able to extract some traces, hopefully will be able to soon...

> > Obviously you will not see any kernel messages on boot, but the machine
> > should boot and with dmesg you should still see all the messages.
> > 
> > Assuming that boots, you could change the short-circuit in
> > serial8250_console_write_atomic() to simulate the timing with:
> > 
> > 	mdelay(7);
> > 	return true;
> > 
> > If it fails to boot, then we at least know it has nothing to do with the
> > 8250 and is instead either delays in time-sensitive sections or some
> > sort of contention bug in the core nbcon code.

I haven't had the time to try the second one yet.

FWIW, Clark suggested it may be worth setting keep_bootcon in the commandline.

He also suspects if the hang is really just that we missed an IPI ack from the
Hyper-V code and now the whole stack is wedged, with the kernel still running.

I'm off to LPC, so I will be radio silent for another day or two while I have
these spin on the Azure VMs.


Cheers,
-- 
Derek <debarbos@redhat.com>

Re: Console Hang on SMP Secondary CPU Bringup

[John Ogness]

On 2026-10-02, Derek Barbosa <debarbos@redhat.com> wrote:
>> To be sure it is timing related due to the nbcon atomic printing, you
>> could try short-circuiting:
>> 
>> 	drivers/tty/serial/8250/8250_port.c:serial8250_console_write_atomic()
>> 
>> so that it immediately does
>> 
>> 	return true;
>> 
>
> I will be running more trials of the reproducer to confirm, but so
> far, we still hang. I wasn't able to extract some traces, hopefully
> will be able to soon...

That is surprising.

For the v6.6-rt you can manually add:

#define CONFIG_SERIAL_8250_LEGACY_CONSOLE

to drivers/tty/serial/8250/8250.h and this should cause the driver to
run in legacy mode. If you add this macro to 8250.h, does the hang go
away?

> FWIW, Clark suggested it may be worth setting keep_bootcon in the
> commandline.

If you use keep_bootcon, then the 8250 will operate in a special mode
that is compatible with the bootcons. It is not a solution, even if it
worked.

> I'm off to LPC, so I will be radio silent for another day or two while
> I have these spin on the Azure VMs.

Sebastian and I are also off to LPC, so we can continue the discussion
in person. :-)

John

Re: Console Hang on SMP Secondary CPU Bringup

[Sebastian Andrzej Siewior]

On 2026-09-23 12:02:31 [-0400], Derek Barbosa wrote:
> In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
> defconfig and additional HV drivers enabled), installing, then booting a kernel
> in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
> boot may result in a unrecoverable "hang"

Does this also occur in v6.6-rt15? If so, can you check v7.3-rc5 and if
it does not occur, try other RT kernels where this started happening?
This HyperV is something I can't try here, can I?

…
> In any case, the system is completely unresponsive over the any serial consoles,
> responding to neither SysRQs nor NMIs, even with the appropriate cmdline
> arguments to invoke such behavior.

Does this run in qemu where you could attach gdb and check what the CPUs
are doing?

> It appears as though the behavior in question has been bisected to af3a8162581e
> ("serial: 8250: Switch to nbcon console")
> 
> <snip>
> $ git describe --contains af3a8162581e
> 
> v6.6.156-rt79-rebase~63
> 
> $ git show v6.6.156-rt79
> </snip>
> 
> and is not reproducible prior to this point (it has not been bisectable to any
> HV/HyperV/Xen/etc commits _yet_).
> 
> IIRC, this commit was reverted in mainline for the time being, as a significant
> amount of work was/is being done in tty/serial, etc...

As far as I remember, it was dropped because other driver fall apart.

> It is not entirely clear what effect the commit introduces with regard to the
> smp secondary CPU masking/bringup, and why it would cause a
> seemingly-unrecoverable state prior to the cpuhp callbacks.

It could be unrelated to the driver, just some printk corner case.

> IIUC (and from what I have seen in the traces) at this point in the init
> process, we've disabled earlycon/earlyserial, and have brought up nbcon.
…
> I have seen in some cases where nbcon 8250 can be slightly slower at boot time,
> which could introduce the possibility of some contention between the increased
> console verbosity and replacement of send_IPI_all/self() with
> hv_send_ipi_all/self() (this is just conjecturing).

Not sure how this is related. Maybe it is a wakeup for the nbcon thread
on the remote CPU.

…
> Specifically, I was curious if serial8250_console_write_thread() could
> potentially be the culprit, especially given the fact that we are running in
> CONFIG_PREEMPT_VOLUNTARY as opposed to CONFIG_PREEMPT_RT.

So we are not doing PREEMPT_RT?

> My understanding is shaky at best here, but specifically:
> 
> <snip>
> 	while (!nbcon_enter_unsafe(wctxt))
> 		nbcon_reacquire(wctxt);
> 
> 	/* Finally, wait for transmitter to become empty and restore IER. */
> 	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
> 	if (em485) {
> 		mdelay(port->rs485.delay_rts_after_send);
> 		if (em485->tx_stopped)
> 			up->rs485_stop_tx(up);
> 	}
> 	serial_port_out(port, UART_IER, ier);
> </snip>
> 
> Is it possible that we just spin here? That we somehow lost ownership at some
> point during the secondary CPU startup sequence and cannot restore UART_IER?
> Would this explain the silent console + hang at PID1?

The regular printk path acquires the uart lock. If my memory serves me
right, the take over takes place in atomic context (say from panic()).

> If that is not the case (very plausible) -- what I am still very unsure of, is
> how the atomic 8250 implementation here could affect the rdmsr()s in
> save/get_fixed_ranges(), to the point of intermittently stalling in the Hyper-V
> VM (MTRR virtualization / SMT sibling) environment?

Me, too. It is just all callback to CPU0 with waiting until it is done.
If CPU1 stalls then CPU0 did not finish… CPU0 should be more or less
idle here.

…
> Many thanks in advance,

The printk version is a bit old but still functional :) If the problem
does not occur in a later kernel then could think about backporting that
printk version back to v6.6 (unless there is a single obvious cause).

Sebastian

Re: Console Hang on SMP Secondary CPU Bringup

[Derek Barbosa]

Hi Sebastian,

On Mon, Sep 28, 2026 at 04:08:37PM +0200, Sebastian Andrzej Siewior wrote:
> On 2026-09-23 12:02:31 [-0400], Derek Barbosa wrote:
> > In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
> > defconfig and additional HV drivers enabled), installing, then booting a kernel
> > in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
> > boot may result in a unrecoverable "hang"
> 
> Does this also occur in v6.6-rt15? If so, can you check v7.3-rc5 and if
> it does not occur, try other RT kernels where this started happening?

You mean the non-rebase branch, correct? I can throw that on the TODO as well.

Just recently I tested a 7.2 kernel with PREEMPT_RT (and some other Fedora
configs) and was able to reproduce the issue. I need to confirm this though,
running some more trials and collecting data. I've been a little
scatter-brained.

> This HyperV is something I can't try here, can I?

Hmm. IIRC you either need a Windows host with the functionality enabled, or
access to some sort of Azure cloud platform [0]. It is similar to other compute
providers in terms of signup, etc.

> 
> …
> > In any case, the system is completely unresponsive over the any serial consoles,
> > responding to neither SysRQs nor NMIs, even with the appropriate cmdline
> > arguments to invoke such behavior.
> 
> Does this run in qemu where you could attach gdb and check what the CPUs
> are doing?
> 

I John's email with something similar. TLDR: I don't know what other functionalities
of Hyper-V the "Azure Cloud VMs" expose, I will play around with the knobs and
fiddle with some buttons here and there...

> > It appears as though the behavior in question has been bisected to af3a8162581e
> > ("serial: 8250: Switch to nbcon console")
> > 
> > <snip>
> > $ git describe --contains af3a8162581e
> > 
> > v6.6.156-rt79-rebase~63
> > 
> > $ git show v6.6.156-rt79
> > </snip>
> > 
> > and is not reproducible prior to this point (it has not been bisectable to any
> > HV/HyperV/Xen/etc commits _yet_).
> > 
> > IIRC, this commit was reverted in mainline for the time being, as a significant
> > amount of work was/is being done in tty/serial, etc...
> 
> As far as I remember, it was dropped because other driver fall apart.
> 
> > It is not entirely clear what effect the commit introduces with regard to the
> > smp secondary CPU masking/bringup, and why it would cause a
> > seemingly-unrecoverable state prior to the cpuhp callbacks.
> 
> It could be unrelated to the driver, just some printk corner case.
> 
> > IIUC (and from what I have seen in the traces) at this point in the init
> > process, we've disabled earlycon/earlyserial, and have brought up nbcon.
> …
> > I have seen in some cases where nbcon 8250 can be slightly slower at boot time,
> > which could introduce the possibility of some contention between the increased
> > console verbosity and replacement of send_IPI_all/self() with
> > hv_send_ipi_all/self() (this is just conjecturing).
> 
> Not sure how this is related. Maybe it is a wakeup for the nbcon thread
> on the remote CPU.
> 
> …
> > Specifically, I was curious if serial8250_console_write_thread() could
> > potentially be the culprit, especially given the fact that we are running in
> > CONFIG_PREEMPT_VOLUNTARY as opposed to CONFIG_PREEMPT_RT.
> 
> So we are not doing PREEMPT_RT?

The issue was reproducible in both :)

But seeing as how the code still exists in this tree, I figured this may be the
best place to report it.

> 
> > My understanding is shaky at best here, but specifically:
> > 
> > <snip>
> > 	while (!nbcon_enter_unsafe(wctxt))
> > 		nbcon_reacquire(wctxt);
> > 
> > 	/* Finally, wait for transmitter to become empty and restore IER. */
> > 	wait_for_xmitr(up, UART_LSR_BOTH_EMPTY);
> > 	if (em485) {
> > 		mdelay(port->rs485.delay_rts_after_send);
> > 		if (em485->tx_stopped)
> > 			up->rs485_stop_tx(up);
> > 	}
> > 	serial_port_out(port, UART_IER, ier);
> > </snip>
> > 
> > Is it possible that we just spin here? That we somehow lost ownership at some
> > point during the secondary CPU startup sequence and cannot restore UART_IER?
> > Would this explain the silent console + hang at PID1?
> 
> The regular printk path acquires the uart lock. If my memory serves me
> right, the take over takes place in atomic context (say from panic()).
> 

Thanks for the note.

> > If that is not the case (very plausible) -- what I am still very unsure of, is
> > how the atomic 8250 implementation here could affect the rdmsr()s in
> > save/get_fixed_ranges(), to the point of intermittently stalling in the Hyper-V
> > VM (MTRR virtualization / SMT sibling) environment?
> 
> Me, too. It is just all callback to CPU0 with waiting until it is done.
> If CPU1 stalls then CPU0 did not finish… CPU0 should be more or less
> idle here.

Yes, that checks out. In fact, further testing reveals that this was a red
herring anyway.

Shortly after I sent that mail, a kernel that I had just finished building with
the mtrr_save_.*() functionalities disabled -- as this appears to be disabled on
HV guests anyway [1] -- had quickly hit the issue again (unrelated, but after
RTFM, it appears that the use of PATs are more common in "modern" kernels than
MTRRs to begin with).

It hangs right at the "cpuhp up cpu1 state cpu:kick_ap" message I had inserted.

I will play around with some of John's suggestions, and a few more ideas I had
in mind.

>  
> …
> > Many thanks in advance,
> 
> The printk version is a bit old but still functional :) If the problem
> does not occur in a later kernel then could think about backporting that
> printk version back to v6.6 (unless there is a single obvious cause).

It very much is! I am pretty positive that we're only seeing the bug in
this weird corner case of HyperV (Linux guest), nbcon + high loglevel.

Thank you for the heads up. I will report back with more results + findings
soon. The bug is just pretty tedious to reproduce.

> 
> Sebastian
> 

[0] https://azure.microsoft.com/en-us/pricing/purchase-options/azure-account?icid=virtual-machines
[1] https://lore.kernel.org/all/20230306163425.8324-1-jgross@suse.com/

-- 
Derek <debarbos@redhat.com>

Re: Console Hang on SMP Secondary CPU Bringup

[Sebastian Andrzej Siewior]

On 2026-09-28 16:54:31 [-0400], Derek Barbosa wrote:
> Hi Sebastian,
Hi Derek,

> On Mon, Sep 28, 2026 at 04:08:37PM +0200, Sebastian Andrzej Siewior wrote:
> > On 2026-09-23 12:02:31 [-0400], Derek Barbosa wrote:
> > > In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
> > > defconfig and additional HV drivers enabled), installing, then booting a kernel
> > > in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
> > > boot may result in a unrecoverable "hang"
> > 
> > Does this also occur in v6.6-rt15? If so, can you check v7.3-rc5 and if
> > it does not occur, try other RT kernels where this started happening?
> 
> You mean the non-rebase branch, correct? I can throw that on the TODO as well.

I don't care so much about rebase vs non-rebase but the old v6.6-rt15 vs
current. Just to ensure that nothing slipped in while moving forward. I
would assume that the rebase and non-rebase branch are equal for a fiven
version.

> Just recently I tested a 7.2 kernel with PREEMPT_RT (and some other Fedora
> configs) and was able to reproduce the issue. I need to confirm this though,
> running some more trials and collecting data. I've been a little
> scatter-brained.

Oh. That is good because it is not something that exists just in this
old version of printk but also in the recent one.

> > This HyperV is something I can't try here, can I?
> 
> Hmm. IIRC you either need a Windows host with the functionality enabled, or
> access to some sort of Azure cloud platform [0]. It is similar to other compute
> providers in terms of signup, etc.

I have neither.

> > Does this run in qemu where you could attach gdb and check what the CPUs
> > are doing?
> > 
> 
> I John's email with something similar. TLDR: I don't know what other functionalities
> of Hyper-V the "Azure Cloud VMs" expose, I will play around with the knobs and
> fiddle with some buttons here and there...

Ah, I forget that gdb and so on does not work in the other world.

> > So we are not doing PREEMPT_RT?
> 
> The issue was reproducible in both :)
> 
> But seeing as how the code still exists in this tree, I figured this may be the
> best place to report it.

The logic should be widely the same but changed in a few details to what
we have now in v7.3-rc1. Also the print logic is different in RT vs
non-RT (due to threaded printing and not).

> It hangs right at the "cpuhp up cpu1 state cpu:kick_ap" message I had inserted.
> 
> I will play around with some of John's suggestions, and a few more ideas I had
> in mind.

The question is, does it hang on its own or is related to printk message
it issues.

> > The printk version is a bit old but still functional :) If the problem
> > does not occur in a later kernel then could think about backporting that
> > printk version back to v6.6 (unless there is a single obvious cause).
> 
> It very much is! I am pretty positive that we're only seeing the bug in
> this weird corner case of HyperV (Linux guest), nbcon + high loglevel.
> 
> Thank you for the heads up. I will report back with more results + findings
> soon. The bug is just pretty tedious to reproduce.

okay, thanks.

Sebastian

Re: Console Hang on SMP Secondary CPU Bringup

[Derek Barbosa]

Hey Sebastian,

On Tue, Sep 29, 2026 at 11:14:15AM +0200, Sebastian Andrzej Siewior wrote:
> On 2026-09-28 16:54:31 [-0400], Derek Barbosa wrote:
> > Hi Sebastian,
> Hi Derek,
> 
> > On Mon, Sep 28, 2026 at 04:08:37PM +0200, Sebastian Andrzej Siewior wrote:
> > > On 2026-09-23 12:02:31 [-0400], Derek Barbosa wrote:
> > > > In the v6.6-rt-rebase branch in the linux-stable-rt kernel trees, building (with
> > > > defconfig and additional HV drivers enabled), installing, then booting a kernel
> > > > in a paravirtualized Virtual Machine environment (HyperV in this case) via UEFI
> > > > boot may result in a unrecoverable "hang"
> > > 
> > > Does this also occur in v6.6-rt15? If so, can you check v7.3-rc5 and if
> > > it does not occur, try other RT kernels where this started happening?
> > 
> > You mean the non-rebase branch, correct? I can throw that on the TODO as well.
> 
> I don't care so much about rebase vs non-rebase but the old v6.6-rt15 vs
> current. Just to ensure that nothing slipped in while moving forward. I
> would assume that the rebase and non-rebase branch are equal for a fiven
> version.

I haven't been able to run as many trials as I would like, but it is
reproducible at v6.6-rt15.

> 
> > Just recently I tested a 7.2 kernel with PREEMPT_RT (and some other Fedora
> > configs) and was able to reproduce the issue. I need to confirm this though,
> > running some more trials and collecting data. I've been a little
> > scatter-brained.
> 
> Oh. That is good because it is not something that exists just in this
> old version of printk but also in the recent one.
> 
> > > This HyperV is something I can't try here, can I?
> > 
> > Hmm. IIRC you either need a Windows host with the functionality enabled, or
> > access to some sort of Azure cloud platform [0]. It is similar to other compute
> > providers in terms of signup, etc.
> 
> I have neither.
> 
> > > Does this run in qemu where you could attach gdb and check what the CPUs
> > > are doing?
> > > 
> > 
> > I John's email with something similar. TLDR: I don't know what other functionalities
> > of Hyper-V the "Azure Cloud VMs" expose, I will play around with the knobs and
> > fiddle with some buttons here and there...
> 
> Ah, I forget that gdb and so on does not work in the other world.
> 
> > > So we are not doing PREEMPT_RT?
> > 
> > The issue was reproducible in both :)
> > 
> > But seeing as how the code still exists in this tree, I figured this may be the
> > best place to report it.
> 
> The logic should be widely the same but changed in a few details to what
> we have now in v7.3-rc1. Also the print logic is different in RT vs
> non-RT (due to threaded printing and not).
> 

I am currently spinning the reproducer on some v7.3-rc4 kernels (both RT and
!RT) and have been able to reproduce at both.


7.3-rc4+rt
<snip>
[    0.097652] Console: colour dummy device 80x25
[    0.097656] printk: legacy console [tty1] enabled
[    0.977756] printk: console [ttyS0] enabled
[    0.977760] printk: legacy bootconsole [earlyser0] disabled
[    0.977839] ACPI: Core revision 20260408
[    0.986610] Failed to register legacy timer interrupt
[    0.990016] APIC: Switch to symmetric I/O mode setup
[    1.002458] x2apic enabled
[    1.012987] APIC: Switched APIC routing to: physical x2apic
[    1.016900] Hyper-V: Hypervisor Build 10.0.26102.1438-1-0
[    1.020785] Hyper-V: enabling crash_kexec_post_notifiers
[    1.024336] Hyper-V: Using IPI hypercalls
[    1.026985] APIC: send_IPI() replaced with hv_send_ipi()
[    1.030622] APIC: send_IPI_mask() replaced with hv_send_ipi_mask()
[    1.035561] APIC: send_IPI_mask_allbutself() replaced with hv_send_ipi_mask_allbutself()
[    1.040568] APIC: send_IPI_allbutself() replaced with hv_send_ipi_allbutself()
[    1.045725] APIC: send_IPI_all() replaced with hv_send_ipi_all()
[    1.049794] APIC: send_IPI_self() replaced with hv_send_ipi_self()
[    1.053820] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.37 BogoMIPS (lpj=2294685)
[    1.054817] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    1.054817] Last level dTLB entries: 4KB 64, 2MB 32, 4MB 32, 1GB 4
[    1.054817] mitigations: Enabled attack vectors: user_kernel, user_user, guest_host, guest_guest, SMT mitigations: auto
[    1.054817] Speculative Store Bypass: Vulnerable
[    1.054817] Spectre V2 : Mitigation: Retpolines
[    1.054817] ITS: Mitigation: Aligned branch/return thunks
[    1.054817] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    1.054817] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
[    1.054817] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode
[    1.054817] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    1.054817] Spectre V2 : Spectre v2 / SpectreRSB: Filling RSB on context switch and VMEXIT
[    1.054817] active return thunk: its_return_thunk
[    1.054817] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    1.054817] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    1.054817] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    1.054817] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    1.054817] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    1.054817] pid_max: default: 32768 minimum: 301
[    1.054817] landlock: Up and running.
[    1.054817] Yama: becoming mindful.
[    1.054817] SELinux:  Initializing.
[    1.054817] LSM support for eBPF active
[    1.054817] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    1.054817] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    1.054817] VFS: Finished mounting rootfs on nullfs
[    1.054817] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[    1.055239] Performance Events: unsupported CPU family 6 model 79 no PMU driver, software events only.
[    1.055872] signal: max sigframe size: 1776
[    1.057165] rcu: Hierarchical SRCU implementation.
[    1.057167] rcu:     Max phase no-delay instances is 400.
[    1.063689] Timer migration: 1 hierarchy levels; 8 children per group; 1 crossnode level
[    1.076550] NMI watchdog: Perf NMI watchdog permanently disabled
[    1.076631] smp: Bringing up secondary CPUs .. ## HANG HERE
</snip>

I would like to run some more trials to truly confirm that this behavior is
consistently reproducible. I've noted that at 7.3-rc4+rt, it takes significantly
more time to reproduce, as opposed to 6.6-rc4+rt (nearly 4x as long).

I was unable to recover a full trace for 7.3-rc4, but it was orders of magnitude
longer, closer to 4 hours to reproduce.

At that point, I am worried that we are just hitting some flakiness caused by
repeated/consecutive boots -- not actually reproducing the console hang -- so I
cannot confidently declare that this is an issue that also affects mainline,
_yet_.

I will be running more trials to confirm.

> > It hangs right at the "cpuhp up cpu1 state cpu:kick_ap" message I had inserted.
> > 
> > I will play around with some of John's suggestions, and a few more ideas I had
> > in mind.
> 
> The question is, does it hang on its own or is related to printk message
> it issues.
> 
> > > The printk version is a bit old but still functional :) If the problem
> > > does not occur in a later kernel then could think about backporting that
> > > printk version back to v6.6 (unless there is a single obvious cause).
> > 
> > It very much is! I am pretty positive that we're only seeing the bug in
> > this weird corner case of HyperV (Linux guest), nbcon + high loglevel.
> > 
> > Thank you for the heads up. I will report back with more results + findings
> > soon. The bug is just pretty tedious to reproduce.
> 
> okay, thanks.
> 
> Sebastian
> 

Cheers,

-- 
Derek <debarbos@redhat.com>