Re: Boot failure with ppc64 port on iMacs G5

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hello,

On Tue, 2024-02-20 at 04:16 +0100, tuxayo wrote:
> I tried snapshots/2024-01-31/debian-12.0.0-ppc64-NETINST-1.iso
> 
> And was able to start booting from usb with:
> boot usb0/disk@1:,\boot\grub\powerpc.elf
> (typed in Open Firmware shell)
> (usb0 is the top port)
> 
> Grub worked, and then I tried default install (the 1st option) and it 
> started loading during like 2 minutes.
> And then it got stuck with some superposition of the messages
> smp_core99_probe
> and
> the stuff before
> DO-QUIESCE finisedBooting Linux via __start() @ 0x0000000002090000 ...

There seems to be a regression in the kernel which affects PowerPC 970 machines,
i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.

If you have the time, I would really appreciate if you could test the various
snapshots and let me know which kernel is the first to not work. I expect that
the breakage occurred somewhere around kernel 6.3 or so.

CC'ing Claudia Neumann who observed this bug before and can maybe share some
additional information.

> Full message of the two iMacs G5 ↓↓↓
> https://transfert.facil.services/r/Ksfq_2VM9_#tDATBcLXzB0zkAEaiqm9gfLfsaXliVJ13rQxKUHgUmA=
> https://transfert.facil.services/r/Zs1h1jEtb2#jufjxv6+1DfHnO3TSfhmYD+teOvY46sGClHyz7SiXd4=

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot failure with ppc64 port on iMacs G5

[tuxayo]

On 24-02-20 10:16, John Paul Adrian Glaubitz wrote:
> There seems to be a regression in the kernel which affects PowerPC 970 machines,
> i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.
> 
> If you have the time, I would really appreciate if you could test the various
> snapshots and let me know which kernel is the first to not work. I expect that
> the breakage occurred somewhere around kernel 6.3 or so.

No problem, I'm going there every week and can do as much testing as 
necessary even if it takes months. We got a donation of a standard PC so 
we have now one publicly usable PC at the library.
So I can take my time to try to make something out of the iMacs.

Results:
- 2023-06-18: ok
   - running strings on iso's vmlinuz yields 6.3.7-1 (2023-06-12)
- 2023-11-16: fail
   - 6.5.10-1 (2023-11-03)

As expected, the regression happened during the big gap between 
snapshots >_<

Anything more I can test?
(cross compiling old kernels is unfortunately too much over my head)

It's weird that it seems the regression is about booting the iso. 
Otherwise, net installs from old iso like what Claudia did wouldn't work 
after rebooting after install.

Cheers,

-- 
tuxayo

Re: Boot failure with ppc64 port on iMacs G5

[Michael Ellerman]

John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> writes:
> On Tue, 2024-02-20 at 04:16 +0100, tuxayo wrote:
>> I tried snapshots/2024-01-31/debian-12.0.0-ppc64-NETINST-1.iso
>> 
>> And was able to start booting from usb with:
>> boot usb0/disk@1:,\boot\grub\powerpc.elf
>> (typed in Open Firmware shell)
>> (usb0 is the top port)
>> 
>> Grub worked, and then I tried default install (the 1st option) and it 
>> started loading during like 2 minutes.
>> And then it got stuck with some superposition of the messages
>> smp_core99_probe
>> and
>> the stuff before
>> DO-QUIESCE finisedBooting Linux via __start() @ 0x0000000002090000 ...
>
> There seems to be a regression in the kernel which affects PowerPC 970 machines,
> i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.

I have a quad G5 that is booting mainline happily.

I used to have an iMac G5 but it died.

> If you have the time, I would really appreciate if you could test the various
> snapshots and let me know which kernel is the first to not work. I expect that
> the breakage occurred somewhere around kernel 6.3 or so.

Can someone send the .config for the kernel in question? I could try
that on my machine here.

cheers

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hi Michael,

On Thu, 2024-02-29 at 17:42 +1100, Michael Ellerman wrote:
> > There seems to be a regression in the kernel which affects PowerPC 970 machines,
> > i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.
> 
> I have a quad G5 that is booting mainline happily.

it's a really tricky problem because it seems to depend on how the kernel image
is booted.

It fails when trying to boot the kernel off the installation CD, i.e. like from here:

> https://cdimage.debian.org/cdimage/ports/snapshots/2024-02-25/debian-12.0.0-ppc64-NETINST-1.iso

but the kernel will boot fine when installing in an existing system which was installed
with an installation CD which uses an older kernel.

> https://cdimage.debian.org/cdimage/ports/snapshots/2023-06-18/debian-12.0.0-ppc64-NETINST-1.iso

I have not really figured out yet what the problem is.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot failure with ppc64 port on iMacs G5

[Michael Ellerman]

John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> writes:
> Hi Michael,
>
> On Thu, 2024-02-29 at 17:42 +1100, Michael Ellerman wrote:
>> > There seems to be a regression in the kernel which affects PowerPC 970 machines,
>> > i.e. PowerMac G5 CPUs. The issue needs to be bisected and reported upstream.
>> 
>> I have a quad G5 that is booting mainline happily.
>
> it's a really tricky problem because it seems to depend on how the kernel image
> is booted.
>
> It fails when trying to boot the kernel off the installation CD, i.e. like from here:
>
>> https://cdimage.debian.org/cdimage/ports/snapshots/2024-02-25/debian-12.0.0-ppc64-NETINST-1.iso
>
> but the kernel will boot fine when installing in an existing system which was installed
> with an installation CD which uses an older kernel.
>
>> https://cdimage.debian.org/cdimage/ports/snapshots/2023-06-18/debian-12.0.0-ppc64-NETINST-1.iso
>
> I have not really figured out yet what the problem is.

OK.

That second iso boots OK for me in qemu. It boots grub and then the
kernel loads and shows:

  Loading ...
  OF stdout device is: /pci@f0000000/mac-io@c/escc@13000/ch-a@13020
  Preparing to boot Linux version 6.3.0-1-powerpc64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.3.0-2) 12.3.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.3.7-1 (2023-06-12)
  Detected machine type: 0000000000000400
  command line: BOOT_IMAGE=/install/vmlinux --- quiet
  memory layout at init:
    memory_limit : 0000000000000000 (16 MB aligned)
    alloc_bottom : 0000000005e70000
    alloc_top    : 0000000030000000
    alloc_top_hi : 0000000080000000
    rmo_top      : 0000000030000000
    ram_top      : 0000000080000000
  copying OF device tree...
  Building dt strings...
  Building dt structure...
  Device tree strings 0x0000000005e80000 -> 0x0000000005e80560
  Device tree struct  0x0000000005e90000 -> 0x0000000005ea0000
  Quiescing Open Firmware ...
  Booting Linux via __start() @ 0x0000000002000000 ...
  Hello World !
  smp_core99_probe
  smp_core99_bringup_done
  Starting system log daemon: syslogd, klogd.

And eventually starts the installer.

That's using no VGA, so possibly there's something wrong with the video
setup on real hardware:

  $ qemu-system-ppc64 -nographic -vga none -M mac99,via=pmu -smp 1 -m 2G -nic user -drive file=$HOME/debian-12.0.0-ppc64-NETINST-1.2023-06-18.iso,format=raw,media=cdrom -boot d

I'll try and find time to test it on my actual G5 next week when I'm in
the office.

cheers

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hi Michael,

On Fri, 2024-03-01 at 12:56 +1100, Michael Ellerman wrote:
> OK.
> 
> That second iso boots OK for me in qemu. It boots grub and then the
> kernel loads and shows:
> 
>   Loading ...
>   OF stdout device is: /pci@f0000000/mac-io@c/escc@13000/ch-a@13020
>   Preparing to boot Linux version 6.3.0-1-powerpc64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.3.0-2) 12.3.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.3.7-1 (2023-06-12)
>   Detected machine type: 0000000000000400
>   command line: BOOT_IMAGE=/install/vmlinux --- quiet
>   memory layout at init:
>     memory_limit : 0000000000000000 (16 MB aligned)
>     alloc_bottom : 0000000005e70000
>     alloc_top    : 0000000030000000
>     alloc_top_hi : 0000000080000000
>     rmo_top      : 0000000030000000
>     ram_top      : 0000000080000000
>   copying OF device tree...
>   Building dt strings...
>   Building dt structure...
>   Device tree strings 0x0000000005e80000 -> 0x0000000005e80560
>   Device tree struct  0x0000000005e90000 -> 0x0000000005ea0000
>   Quiescing Open Firmware ...
>   Booting Linux via __start() @ 0x0000000002000000 ...
>   Hello World !
>   smp_core99_probe
>   smp_core99_bringup_done
>   Starting system log daemon: syslogd, klogd.
> 
> And eventually starts the installer.

Yep, the second, older image works as expected. However, the recent one does not
and I have absolutely no clue why.

> That's using no VGA, so possibly there's something wrong with the video
> setup on real hardware:
> 
>   $ qemu-system-ppc64 -nographic -vga none -M mac99,via=pmu -smp 1 -m 2G -nic user -drive file=$HOME/debian-12.0.0-ppc64-NETINST-1.2023-06-18.iso,format=raw,media=cdrom -boot d
> 
> I'll try and find time to test it on my actual G5 next week when I'm in
> the office.

The video issue can usually be worked around by disabling mode-setting or
passing other kernel options related to the video card. That's not the
main problem here though.

The problem is that the newer image doesn't boot and currently I don't know
why because installing the exact same kernel later from the package manager
into an installed system works yields a bootable system with the latest
kernel.

The installer images are built from the same kernel package which makes the
whole thing even more confusing.

I'm using debian-cd to build the installation images:

> https://salsa.debian.org/images-team/debian-cd/

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot failure with ppc64 port on iMacs G5

[Brad Boyer]

On Fri, Mar 01, 2024 at 03:47:26PM +0100, John Paul Adrian Glaubitz wrote:
> The problem is that the newer image doesn't boot and currently I don't know
> why because installing the exact same kernel later from the package manager
> into an installed system works yields a bootable system with the latest
> kernel.
> 
> The installer images are built from the same kernel package which makes the
> whole thing even more confusing.

Perhaps it's a memory allocation problem before the kernel has fully
configured the MMU? I know in the past the kernel could be touchy
about how memory was mapped in the very early stages of boot. If we
happened to be near some limit, even a small increase in the size of
the kernel might have pushed us over the edge.

I'm not sure of the exact details, but perhaps the memory setup is
slightly different when booted from the CD image. I haven't looked
at recent images, but I know in the past the boot process was
different on CD compared to after the install.

	Brad Boyer
	flar@allandria.com

Re: Boot failure with ppc64 port on iMacs G5

[Michael Ellerman]

John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> writes:
> On Fri, 2024-03-01 at 12:56 +1100, Michael Ellerman wrote:
>> That second iso boots OK for me in qemu. It boots grub and then the
>> kernel loads and shows:
>> 
>>   Loading ...
>>   OF stdout device is: /pci@f0000000/mac-io@c/escc@13000/ch-a@13020
>>   Preparing to boot Linux version 6.3.0-1-powerpc64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.3.0-2) 12.3.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.3.7-1 (2023-06-12)
>>   Detected machine type: 0000000000000400
>>   command line: BOOT_IMAGE=/install/vmlinux --- quiet
>>   memory layout at init:
>>     memory_limit : 0000000000000000 (16 MB aligned)
>>     alloc_bottom : 0000000005e70000
>>     alloc_top    : 0000000030000000
>>     alloc_top_hi : 0000000080000000
>>     rmo_top      : 0000000030000000
>>     ram_top      : 0000000080000000
>>   copying OF device tree...
>>   Building dt strings...
>>   Building dt structure...
>>   Device tree strings 0x0000000005e80000 -> 0x0000000005e80560
>>   Device tree struct  0x0000000005e90000 -> 0x0000000005ea0000
>>   Quiescing Open Firmware ...
>>   Booting Linux via __start() @ 0x0000000002000000 ...
>>   Hello World !
>>   smp_core99_probe
>>   smp_core99_bringup_done
>>   Starting system log daemon: syslogd, klogd.
>> 
>> And eventually starts the installer.
>
> Yep, the second, older image works as expected. However, the recent one does not
> and I have absolutely no clue why.

I actually tested both, and both work, but then I cited the wrong one in
my email >_<

So at least on qemu that newer kernel is OK:

  Preparing to boot Linux version 6.6.15-powerpc64 (debian-kernel@lists.debian.org) (gcc-13 (Debian 13.2.0-13) 13.2.0, GNU ld (GNU Binutils for Debian) 2.42) #1 SMP Debian 6.6.15-2 (2024-02-04)
  ...
  Booting Linux via __start() @ 0x0000000004800000 ...
  Hello World !
  smp_core99_probe
  smp_core99_bringup_done
  Starting system log daemon: syslogd, klogd.


cheers

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hi Michael,

On Wed, 2024-03-06 at 12:57 +1100, Michael Ellerman wrote:
> > Yep, the second, older image works as expected. However, the recent one does not
> > and I have absolutely no clue why.
> 
> I actually tested both, and both work, but then I cited the wrong one in
> my email >_<
> 
> So at least on qemu that newer kernel is OK:
> 
>   Preparing to boot Linux version 6.6.15-powerpc64 (debian-kernel@lists.debian.org) (gcc-13 (Debian 13.2.0-13) 13.2.0, GNU ld (GNU Binutils for Debian) 2.42) #1 SMP Debian 6.6.15-2 (2024-02-04)
>   ...
>   Booting Linux via __start() @ 0x0000000004800000 ...
>   Hello World !
>   smp_core99_probe
>   smp_core99_bringup_done
>   Starting system log daemon: syslogd, klogd.

Did you get around testing the images on real hardware?

And can tell me what command line you used for booting with QEMU?
Maybe that gives us a clue where the problem is.

Users are still reporting boot lockups with kernel 6.6.x.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hi,

On Wed, 2024-03-06 at 12:57 +1100, Michael Ellerman wrote:
> > p, the second, older image works as expected. However, the recent one does not
> > and I have absolutely no clue why.
> 
> I actually tested both, and both work, but then I cited the wrong one in
> my email >_<
> 
> So at least on qemu that newer kernel is OK:
> 
>   Preparing to boot Linux version 6.6.15-powerpc64 (debian-kernel@lists.debian.org) (gcc-13 (Debian 13.2.0-13) 13.2.0, GNU ld (GNU Binutils for Debian) 2.42) #1 SMP Debian 6.6.15-2 (2024-02-04)
>   ...
>   Booting Linux via __start() @ 0x0000000004800000 ...
>   Hello World !
>   smp_core99_probe
>   smp_core99_bringup_done
>   Starting system log daemon: syslogd, klogd.

Here is a freshly generated image using kernel 6.7.12 as the installer kernel:

> https://cdimage.debian.org/cdimage/ports/tests/ppc64-test-20240511/debian-12.0.0-ppc64-NETINST-1.iso

Still hangs for me with QEMU emulating a PPC970.

Will retest once kernel 6.8.x enters Debian unstable.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Boot failure with ppc64 port on iMacs G5

[John Paul Adrian Glaubitz]

Hi Michael,

On Wed, 2024-03-06 at 12:57 +1100, Michael Ellerman wrote:
> > Yep, the second, older image works as expected. However, the recent one does not
> > and I have absolutely no clue why.
> 
> I actually tested both, and both work, but then I cited the wrong one in
> my email >_<
> 
> So at least on qemu that newer kernel is OK:
> 
>   Preparing to boot Linux version 6.6.15-powerpc64 (debian-kernel@lists.debian.org) (gcc-13 (Debian 13.2.0-13) 13.2.0, GNU ld (GNU Binutils for Debian) 2.42) #1 SMP Debian 6.6.15-2 (2024-02-04)
>   ...
>   Booting Linux via __start() @ 0x0000000004800000 ...
>   Hello World !
>   smp_core99_probe
>   smp_core99_bringup_done
>   Starting system log daemon: syslogd, klogd.

Just as a heads-up: This particular issue is related to this patch in the Debian kernel package:

> https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/patches/bugfix/powerpc/fbdev-offb-Update-expected-device-name.patch?ref_type=heads

Removing this patch unbreaks the console on PowerMac G5.

I could also reproduce this issue with the latest ppc64el image on PowerKVM:

> https://cdimage.debian.org/cdimage/daily-builds/daily/current/ppc64el/iso-cd/debian-testing-ppc64el-netinst.iso

Here is a video demonstrating the issue:

> https://people.debian.org/~glaubitz/debian-bug-1033058.mp4

And here is the corresponding bug report:

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085949

This issue can be easily reproduced with QEMU.

Aditi Mishra (CC'ed) is trying to reproduce the issue as well, but so far has been unsuccessful.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913