x86: Boot failure on select chromebooks with v6.15-rc5

x86: Boot failure on select chromebooks with v6.15-rc5

[Vignesh Raman]

Hi,

With v6.15-rc5, the below chromebooks were not booting with the 
following message. These tests were run in drm-ci.

Starting kernel ...
[    1.843801]  ? __pfx_kernel_init+0x10/0x10
[    1.909838]  ? __pfx_kernel_init+0x10/0x10

https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1420485

Failing jobs:
amdgpu:stoney (AMD Stoney Ridge chipset): 
https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000926

i915:amly (64 bit Intel Whiskey Lake): 
https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000929

i915:whl (64 bit Intel Amber Lake): 
https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000931

These tests were passing till v6.14-rc7, and the issue was seen starting 
in v6.15-rc1. This issue is seen only with these 3 boards and does not 
affect other chromebooks (the pipeline log shows the other i915 driver 
tests).

On bisecting the commits, the commit which introduced this issue is,
3181424aeac2f6596534bf43021a10eae294a9b0 x86/early_printk: Add support 
for MMIO-based UARTs

After reverting the below commits in v6.15-rc5, the board boots and 
tests are executed:
3181424aeac2 x86/early_printk: Add support for MMIO-based UARTs
996457176bb7 x86/early_printk: Use 'mmio32' for consistency, fix 
comments (this fixes 3181424aeac2)

pipeline after reverting the commits,
https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1420493

Please could you look into this issue. Thank you.

Regards,
Vignesh

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[Ingo Molnar]

* Vignesh Raman <vignesh.raman@collabora.com> wrote:

> Hi,
> 
> With v6.15-rc5, the below chromebooks were not booting with the following
> message. These tests were run in drm-ci.
> 
> Starting kernel ...
> [    1.843801]  ? __pfx_kernel_init+0x10/0x10
> [    1.909838]  ? __pfx_kernel_init+0x10/0x10
> 
> https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1420485
> 
> Failing jobs:
> amdgpu:stoney (AMD Stoney Ridge chipset):
> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000926
> 
> i915:amly (64 bit Intel Whiskey Lake):
> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000929
> 
> i915:whl (64 bit Intel Amber Lake):
> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000931
> 
> These tests were passing till v6.14-rc7, and the issue was seen starting in
> v6.15-rc1. This issue is seen only with these 3 boards and does not affect
> other chromebooks (the pipeline log shows the other i915 driver tests).
> 
> On bisecting the commits, the commit which introduced this issue is,
> 3181424aeac2f6596534bf43021a10eae294a9b0 x86/early_printk: Add support for
> MMIO-based UARTs
> 
> After reverting the below commits in v6.15-rc5, the board boots and tests
> are executed:
> 3181424aeac2 x86/early_printk: Add support for MMIO-based UARTs
> 996457176bb7 x86/early_printk: Use 'mmio32' for consistency, fix comments
> (this fixes 3181424aeac2)

What boot cmdline does your kernel have? The MMIO-UART patches should 
only have an effect if the feature is specifically enabled via a boot 
option:

+               if (!strncmp(buf, "mmio32", 6)) {
+			buf += 6;
+                       early_mmio_serial_init(buf);
+                       early_console_register(&early_serial_console, keep);
+                       buf += 4;
+               }

The only other change I can see is the moving of an #if line.

Thanks,

	Ingo

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[Vignesh Raman]

Hi Ingo,

On 09/05/25 11:36, Ingo Molnar wrote:
> 
> * Vignesh Raman <vignesh.raman@collabora.com> wrote:
> 
>> Hi,
>>
>> With v6.15-rc5, the below chromebooks were not booting with the following
>> message. These tests were run in drm-ci.
>>
>> Starting kernel ...
>> [    1.843801]  ? __pfx_kernel_init+0x10/0x10
>> [    1.909838]  ? __pfx_kernel_init+0x10/0x10
>>
>> https://gitlab.freedesktop.org/vigneshraman/linux/-/pipelines/1420485
>>
>> Failing jobs:
>> amdgpu:stoney (AMD Stoney Ridge chipset):
>> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000926
>>
>> i915:amly (64 bit Intel Whiskey Lake):
>> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000929
>>
>> i915:whl (64 bit Intel Amber Lake):
>> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76000931
>>
>> These tests were passing till v6.14-rc7, and the issue was seen starting in
>> v6.15-rc1. This issue is seen only with these 3 boards and does not affect
>> other chromebooks (the pipeline log shows the other i915 driver tests).
>>
>> On bisecting the commits, the commit which introduced this issue is,
>> 3181424aeac2f6596534bf43021a10eae294a9b0 x86/early_printk: Add support for
>> MMIO-based UARTs
>>
>> After reverting the below commits in v6.15-rc5, the board boots and tests
>> are executed:
>> 3181424aeac2 x86/early_printk: Add support for MMIO-based UARTs
>> 996457176bb7 x86/early_printk: Use 'mmio32' for consistency, fix comments
>> (this fixes 3181424aeac2)
> 
> What boot cmdline does your kernel have? The MMIO-UART patches should
> only have an effect if the feature is specifically enabled via a boot
> option:
> 
> +               if (!strncmp(buf, "mmio32", 6)) {
> +			buf += 6;
> +                       early_mmio_serial_init(buf);
> +                       early_console_register(&early_serial_console, keep);
> +                       buf += 4;
> +               }
> 

amdgpu:stoney:
earlyprintk=uart8250,mmio32,0xfedc6000,115200n8  console=ttyS0,115200n8 
root=/dev/nfs rw 
nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598802/extract-nfsrootfs-wgn1xjer,tcp,hard,v3 
init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp 
tftpserverip=192.168.201.1

i915:amly:
earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8 
root=/dev/nfs rw 
nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598804/extract-nfsrootfs-5rlm_b6z,tcp,hard,v3 
init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp 
tftpserverip=192.168.201.1

i915:whl:
earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8 
root=/dev/nfs rw 
nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598833/extract-nfsrootfs-3w0w5_mi,tcp,hard,v3 
init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp 
tftpserverip=192.168.201.1

Regards,
Vignesh

> The only other change I can see is the moving of an #if line.
> 
> Thanks,
> 
> 	Ingo

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[Ingo Molnar]

* Vignesh Raman <vignesh.raman@collabora.com> wrote:

> > What boot cmdline does your kernel have? The MMIO-UART patches should
> > only have an effect if the feature is specifically enabled via a boot
> > option:
> > 
> > +               if (!strncmp(buf, "mmio32", 6)) {
> > +			buf += 6;
> > +                       early_mmio_serial_init(buf);
> > +                       early_console_register(&early_serial_console, keep);
> > +                       buf += 4;
> > +               }
> > 
> 
> amdgpu:stoney:
> earlyprintk=uart8250,mmio32,0xfedc6000,115200n8  console=ttyS0,115200n8
> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598802/extract-nfsrootfs-wgn1xjer,tcp,hard,v3
> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> tftpserverip=192.168.201.1
> 
> i915:amly:
> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598804/extract-nfsrootfs-5rlm_b6z,tcp,hard,v3
> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> tftpserverip=192.168.201.1
> 
> i915:whl:
> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598833/extract-nfsrootfs-3w0w5_mi,tcp,hard,v3
> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> tftpserverip=192.168.201.1

Well, if you remove the earlyprintk option then it will boot fine, 
right?

The earlyprintk=mmio32 in v6.15 is a new debugging feature that was 
tested on a single board by Denis Mukhin AFAIK, and it may or may not 
work on your particular UART - even assuming that all the parameters 
are correct.

Thanks,

	Ingo

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[Vignesh Raman]

Hi Ingo,

On 09/05/25 12:07, Ingo Molnar wrote:
> 
> * Vignesh Raman <vignesh.raman@collabora.com> wrote:
> 
>>> What boot cmdline does your kernel have? The MMIO-UART patches should
>>> only have an effect if the feature is specifically enabled via a boot
>>> option:
>>>
>>> +               if (!strncmp(buf, "mmio32", 6)) {
>>> +			buf += 6;
>>> +                       early_mmio_serial_init(buf);
>>> +                       early_console_register(&early_serial_console, keep);
>>> +                       buf += 4;
>>> +               }
>>>
>>
>> amdgpu:stoney:
>> earlyprintk=uart8250,mmio32,0xfedc6000,115200n8  console=ttyS0,115200n8
>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598802/extract-nfsrootfs-wgn1xjer,tcp,hard,v3
>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>> tftpserverip=192.168.201.1
>>
>> i915:amly:
>> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598804/extract-nfsrootfs-5rlm_b6z,tcp,hard,v3
>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>> tftpserverip=192.168.201.1
>>
>> i915:whl:
>> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598833/extract-nfsrootfs-3w0w5_mi,tcp,hard,v3
>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>> tftpserverip=192.168.201.1
> 
> Well, if you remove the earlyprintk option then it will boot fine,
> right?

Yes, it works when mmio32 option is removed.

https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76005338

earlyprintk=uart8250,0xde000000,115200n8  console=ttyS0,115200n8 
root=/dev/nfs rw 
nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18599938/extract-nfsrootfs-neuejjq0,tcp,hard,v3 
init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp 
tftpserverip=192.168.201.1

> 
> The earlyprintk=mmio32 in v6.15 is a new debugging feature that was
> tested on a single board by Denis Mukhin AFAIK, and it may or may not
> work on your particular UART - even assuming that all the parameters
> are correct.

So the earlyprintk=mmio32 debugging feature is needed only for v6.15 and 
is not necessary in previous kernels (e.g., v6.14 and earlier). Is my 
understanding correct?

Regards,
Vignesh

> 
> Thanks,
> 
> 	Ingo

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[dmkhn]

Hi, 

On Fri, May 09, 2025 at 01:07:54PM +0530, Vignesh Raman wrote:
> Hi Ingo,
> 
> On 09/05/25 12:07, Ingo Molnar wrote:
> > 
> > * Vignesh Raman <vignesh.raman@collabora.com> wrote:
> > 
> > > > What boot cmdline does your kernel have? The MMIO-UART patches should
> > > > only have an effect if the feature is specifically enabled via a boot
> > > > option:
> > > > 
> > > > +               if (!strncmp(buf, "mmio32", 6)) {
> > > > +			buf += 6;
> > > > +                       early_mmio_serial_init(buf);
> > > > +                       early_console_register(&early_serial_console, keep);
> > > > +                       buf += 4;
> > > > +               }
> > > > 
> > > 
> > > amdgpu:stoney:
> > > earlyprintk=uart8250,mmio32,0xfedc6000,115200n8  console=ttyS0,115200n8
> > > root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598802/extract-nfsrootfs-wgn1xjer,tcp,hard,v3
> > > init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> > > tftpserverip=192.168.201.1
> > > 
> > > i915:amly:
> > > earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
> > > root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598804/extract-nfsrootfs-5rlm_b6z,tcp,hard,v3
> > > init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> > > tftpserverip=192.168.201.1
> > > 
> > > i915:whl:
> > > earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
> > > root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598833/extract-nfsrootfs-3w0w5_mi,tcp,hard,v3
> > > init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> > > tftpserverip=192.168.201.1
> > 
> > Well, if you remove the earlyprintk option then it will boot fine,
> > right?
> 
> Yes, it works when mmio32 option is removed.
> 
> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76005338
> 
> earlyprintk=uart8250,0xde000000,115200n8  console=ttyS0,115200n8
> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18599938/extract-nfsrootfs-neuejjq0,tcp,hard,v3
> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
> tftpserverip=192.168.201.1

For the above example, can you please try something like

  earlyprintk=mmio32,0xde000000,nocfg

?

In my case, configuring exact baud rate did not work. I started to dig that,
but did not finish, because `nocfg` worked (firmware happened to configure the
UART correctly). Using `nocfg` was sufficient for the system bringup debugging.

> 
> > 
> > The earlyprintk=mmio32 in v6.15 is a new debugging feature that was
> > tested on a single board by Denis Mukhin AFAIK, and it may or may not
> > work on your particular UART - even assuming that all the parameters
> > are correct.

Correct, I have tested with one board only and with limited UART configuration
combinations.

> 
> So the earlyprintk=mmio32 debugging feature is needed only for v6.15 and is
> not necessary in previous kernels (e.g., v6.14 and earlier). Is my
> understanding correct?
> 
> Regards,
> Vignesh
> 
> > 
> > Thanks,
> > 
> > 	Ingo
> 

Thanks,
Denis

Re: x86: Boot failure on select chromebooks with v6.15-rc5

[Vignesh Raman]

Hi Denis

On 12/05/25 23:00, dmkhn@proton.me wrote:
> Hi,
> 
> On Fri, May 09, 2025 at 01:07:54PM +0530, Vignesh Raman wrote:
>> Hi Ingo,
>>
>> On 09/05/25 12:07, Ingo Molnar wrote:
>>>
>>> * Vignesh Raman <vignesh.raman@collabora.com> wrote:
>>>
>>>>> What boot cmdline does your kernel have? The MMIO-UART patches should
>>>>> only have an effect if the feature is specifically enabled via a boot
>>>>> option:
>>>>>
>>>>> +               if (!strncmp(buf, "mmio32", 6)) {
>>>>> +			buf += 6;
>>>>> +                       early_mmio_serial_init(buf);
>>>>> +                       early_console_register(&early_serial_console, keep);
>>>>> +                       buf += 4;
>>>>> +               }
>>>>>
>>>>
>>>> amdgpu:stoney:
>>>> earlyprintk=uart8250,mmio32,0xfedc6000,115200n8  console=ttyS0,115200n8
>>>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598802/extract-nfsrootfs-wgn1xjer,tcp,hard,v3
>>>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>>>> tftpserverip=192.168.201.1
>>>>
>>>> i915:amly:
>>>> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
>>>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598804/extract-nfsrootfs-5rlm_b6z,tcp,hard,v3
>>>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>>>> tftpserverip=192.168.201.1
>>>>
>>>> i915:whl:
>>>> earlyprintk=uart8250,mmio32,0xde000000,115200n8  console=ttyS0,115200n8
>>>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18598833/extract-nfsrootfs-3w0w5_mi,tcp,hard,v3
>>>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>>>> tftpserverip=192.168.201.1
>>>
>>> Well, if you remove the earlyprintk option then it will boot fine,
>>> right?
>>
>> Yes, it works when mmio32 option is removed.
>>
>> https://gitlab.freedesktop.org/vigneshraman/linux/-/jobs/76005338
>>
>> earlyprintk=uart8250,0xde000000,115200n8  console=ttyS0,115200n8
>> root=/dev/nfs rw nfsroot=192.168.201.1:/var/lib/lava/dispatcher/tmp/18599938/extract-nfsrootfs-neuejjq0,tcp,hard,v3
>> init=/init rootwait usbcore.quirks=0bda:8153:k ip=dhcp
>> tftpserverip=192.168.201.1
> 
> For the above example, can you please try something like
> 
>    earlyprintk=mmio32,0xde000000,nocfg
> 
> ?

This option didn’t work. For now, we’ll use earlycon instead of 
earlyprintk to fix the issue.

Thanks.

Regards,
Vignesh

> 
> In my case, configuring exact baud rate did not work. I started to dig that,
> but did not finish, because `nocfg` worked (firmware happened to configure the
> UART correctly). Using `nocfg` was sufficient for the system bringup debugging.
> 
>>
>>>
>>> The earlyprintk=mmio32 in v6.15 is a new debugging feature that was
>>> tested on a single board by Denis Mukhin AFAIK, and it may or may not
>>> work on your particular UART - even assuming that all the parameters
>>> are correct.
> 
> Correct, I have tested with one board only and with limited UART configuration
> combinations.
> 
>>
>> So the earlyprintk=mmio32 debugging feature is needed only for v6.15 and is
>> not necessary in previous kernels (e.g., v6.14 and earlier). Is my
>> understanding correct?
>>
>> Regards,
>> Vignesh
>>
>>>
>>> Thanks,
>>>
>>> 	Ingo
>>
> 
> Thanks,
> Denis
>