Re: test 1: was: Re: A Comparison of printk between upstream and linux-rt-devel

[Petr Mladek <pmladek@suse.com>]

On Fri 2024-08-23 15:06:57, Derek Barbosa wrote:
> Hi,
> 
> On Fri, Aug 23, 2024 at 12:19:13PM +0200, Petr Mladek wrote:
> > Could you please also share the kernel command line? I can't find it
> > anywhere.
> > 
> > Especially I am interested whether it:
> > 
> >   + wanted to show backtraces on all CPUs via "panic_print" parameter.
> 
> Sure, here's the output of cat /proc/cmdline on the stock kernel
> 
> BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.11.0-0.rc3.20240814git6b0f8db921ab.32.eln142.x86_64 
> root=/dev/mapper/cs_rt--qe--06-root ro 
> crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M 
> resume=/dev/mapper/cs_rt--qe--06-swap rd.lvm.lv=cs_rt-qe-06/root rd.lvm.lv=cs_rt-qe-06/swap 
> console=ttyS0,115200
> 
> >   + did a crashdump or a reboot.
>  
> Yes, with kexec-tools and kdump configured, we succesfully booted into the
> kdump kernel and rebooted every time as well as generated vmcores.
>
> In fact, examining said vmvcore-dmesg.txt log, we are able to see the final
> trace printed. It is simply through the serial console that we are unable to.

I see. This probably explains why the result of the stock kernel was so bad.

There are two tricks which increases the chance to see panic
messages on legacy consoles:

    1. bust_spinlocks() causes that con->write() callbacks ignore
       port->lock.

       This helps only when console_trylock() succeeds in printk().
       Also the serial port must be in a state which allows to see
       the data written by con->write().

       This does not help in NMI


    2. console_flush_on_panic() ignores even console_lock()
       It tries to call console drivers even in NMI context.

       This would likely allow to see the panic bracktrace even
       on stock kernel. But it _not_ called when crashdump is
       called.

> I will attach the vmcore-dmesg.txt. I also just did another run of
> console_blast, capturing the output from invocation to reboot.
> >
> >   + used also another console (graphics).
> 
> No, I solely used a serial console at ttyS0 at 115200 baud.

I see.

> > Do you miss the backtrace from the panic-CPU or non-panic-CPUs or
> > both?
> 
> I am not 100% certain. I included the two attachments, which may have your
> answer.
> >
> > The dump of the backtraces on non-panic-CPUs might have been affected
> > by the regression fixed earlier this week via
> > https://lore.kernel.org/r/20240812072703.339690-1-takakura@valinux.co.jp
> 
> Excellent. Is this included in the latest rt-devel tree? I can build that kernel
> and run it through the gambit of tests again.

Not needed. The tests were not affected by the bug because you did not use
"panic_printk" parameter on the command line.

> > Did the system reboot in the end?
> > Or does it got stuck somewhere?
> 
> The system always reboots.

Great.

> If you have any other questions, please let me know. I would be happy to re-do
> these tests with different kernels, configs + other variables and controls.

It might be interesting to redo the 1st test without the crashdump.
I wonder if console_flush_on_panic() would allow to see the panic
backtrace with the stock kernel.

But it is not important. The most important thing is that the new
nbcon consoles are clearly better and more reliable at least in
the tested scenarios.

Best Regards,
Petr