lost serial characters on boot

lost serial characters on boot

[Andrew Cooper]

Hi,

I think we've got a bug with the serial console.  I'm reliably loosing a
small part of the boot message. e.g.:

(XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
(XEN) build-id: d1536ff6698b3e3ba307d551abf
 Family 25 (0x19 Stepping 1 (raw 00a00f11)
(XEN) BSP microcode revision: 0x0a001137

whereas from xl dmesg after the system boots, we see things correctly:

(XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
(XEN) build-id: d1536ff6698b3b8968e883fc39e3ba307d551abf
(XEN) CPU Vendor: AMD, Family 25 (0x19), Model 1 (0x1), Stepping 1 (raw
00a00f11)
(XEN) BSP microcode revision: 0x0a001137


The lost bit of text differs even when rebooting the same Xen binary,
but it's always around this area, very early on boot.

I've been seeing this for a little while, but one of my common dev
machines is showing it fairly reliably at the moment.

Checking a transcript of the terminal, we're missing characters, rather
than having additional control characters causing the corruption.

I suppose I need to stare at console_init_preirq() some more, but if
anyone has ideas, I'm all ears.

~Andrew

Re: lost serial characters on boot

[Frediano Ziglio]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

On Fri, 17 Oct 2025, 18:16 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:

> Hi,
>
> I think we've got a bug with the serial console.  I'm reliably loosing a
> small part of the boot message. e.g.:
>
> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
> (XEN) build-id: d1536ff6698b3e3ba307d551abf
>  Family 25 (0x19 Stepping 1 (raw 00a00f11)
> (XEN) BSP microcode revision: 0x0a001137
>
> whereas from xl dmesg after the system boots, we see things correctly:
>
> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
> (XEN) build-id: d1536ff6698b3b8968e883fc39e3ba307d551abf
> (XEN) CPU Vendor: AMD, Family 25 (0x19), Model 1 (0x1), Stepping 1 (raw
> 00a00f11)
> (XEN) BSP microcode revision: 0x0a001137
>
>
> The lost bit of text differs even when rebooting the same Xen binary,
> but it's always around this area, very early on boot.
>
> I've been seeing this for a little while, but one of my common dev
> machines is showing it fairly reliably at the moment.
>
> Checking a transcript of the terminal, we're missing characters, rather
> than having additional control characters causing the corruption.
>
> I suppose I need to stare at console_init_preirq() some more, but if
> anyone has ideas, I'm all ears.
>
> ~Andrew
>
Hi,
   I would guess the code for early boot is different and is not waiting
correctly if the serial is able to accept new characters so the hardware
buffer is overwritten.

Frediano

>

[-- Attachment #2: Type: text/html, Size: 2118 bytes --]

Re: lost serial characters on boot

[Frediano Ziglio]

On Sat, Oct 18, 2025 at 10:35 AM Frediano Ziglio <freddy77@gmail.com> wrote:
>
> On Fri, 17 Oct 2025, 18:16 Andrew Cooper, <andrew.cooper3@citrix.com> wrote:
>>
>> Hi,
>>
>> I think we've got a bug with the serial console.  I'm reliably loosing a
>> small part of the boot message. e.g.:
>>
>> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
>> (XEN) build-id: d1536ff6698b3e3ba307d551abf
>>  Family 25 (0x19 Stepping 1 (raw 00a00f11)
>> (XEN) BSP microcode revision: 0x0a001137
>>
>> whereas from xl dmesg after the system boots, we see things correctly:
>>
>> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
>> (XEN) build-id: d1536ff6698b3b8968e883fc39e3ba307d551abf
>> (XEN) CPU Vendor: AMD, Family 25 (0x19), Model 1 (0x1), Stepping 1 (raw
>> 00a00f11)
>> (XEN) BSP microcode revision: 0x0a001137
>>
>>
>> The lost bit of text differs even when rebooting the same Xen binary,
>> but it's always around this area, very early on boot.
>>
>> I've been seeing this for a little while, but one of my common dev
>> machines is showing it fairly reliably at the moment.
>>
>> Checking a transcript of the terminal, we're missing characters, rather
>> than having additional control characters causing the corruption.
>>
>> I suppose I need to stare at console_init_preirq() some more, but if
>> anyone has ideas, I'm all ears.
>>
>> ~Andrew
>
> Hi,
>    I would guess the code for early boot is different and is not waiting correctly if the serial is able to accept new characters so the hardware buffer is overwritten.
>

Or we manage to reach some code like:

        if ( port->tx_quench )
        {
            /* Buffer filled and we are dropping characters. */
            if ( (port->txbufp - port->txbufc) > (serial_txbufsz / 2) )
                return;
            port->tx_quench = 0;
        }

(__serial_putc)

Frediano

Re: lost serial characters on boot

[Frediano Ziglio]

On Fri, Oct 17, 2025 at 6:16 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
> Hi,
>
> I think we've got a bug with the serial console.  I'm reliably loosing a
> small part of the boot message. e.g.:
>
> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
> (XEN) build-id: d1536ff6698b3e3ba307d551abf
>  Family 25 (0x19 Stepping 1 (raw 00a00f11)
> (XEN) BSP microcode revision: 0x0a001137
>
> whereas from xl dmesg after the system boots, we see things correctly:
>
> (XEN) Latest ChangeSet: Fri Oct 17 17:24:59 2025 git:5e79cfcd2100
> (XEN) build-id: d1536ff6698b3b8968e883fc39e3ba307d551abf
> (XEN) CPU Vendor: AMD, Family 25 (0x19), Model 1 (0x1), Stepping 1 (raw
> 00a00f11)
> (XEN) BSP microcode revision: 0x0a001137
>
>
> The lost bit of text differs even when rebooting the same Xen binary,
> but it's always around this area, very early on boot.
>
> I've been seeing this for a little while, but one of my common dev
> machines is showing it fairly reliably at the moment.
>
> Checking a transcript of the terminal, we're missing characters, rather
> than having additional control characters causing the corruption.
>
> I suppose I need to stare at console_init_preirq() some more, but if
> anyone has ideas, I'm all ears.
>
> ~Andrew
>

Which kind of serial do you have ? Standard ns16550 ?

Frediano