Boot error since 6.9: Initramfs unpacking failed: invalid magic at start of compressed archive

Boot error since 6.9: Initramfs unpacking failed: invalid magic at start of compressed archive

[Thorsten Leemhuis]

Hi, Thorsten here, the Linux kernel's regression tracker.

I noticed a report about a regression in bugzilla.kernel.org that a user
bisected to the main ARM64 merge commit from the 6.9 cycle. So the real
issue might be somewhere else, but some help would be appreciated. To
quote from https://bugzilla.kernel.org/show_bug.cgi?id=219014 :

> Hi.
> 
> I'm marking as arm64 since git bisect identifies the first problematic commit as:
>   6d75c6f40a03c97e1ecd683ae54e249abb9d922b
> 
> git bisect log is attached.
> 
> Any kernel tagged v6.9x and later fails to boot with the following error:
> 
> [    0.420154] Initramfs unpacking failed: invalid magic at start of compressed archive
> [    0.433178] Freeing initrd memory: 8820K
> ... snip ...
> [    0.701958] List of all partitions:
> [    0.705604] No filesystem could mount root, tried: 
> [    0.705608] 
> [    0.712166] Kernel panic - not syncing: VFS: Unable to mount root fs on "UUID=a889afec-1ced-4493-ab8b-cf1569819c49" or unknown-block(0,0)                                                
> [    0.724903] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.9.8-sheeva64 #sheeva64
> [    0.732356] Hardware name: gti sheeva64 development board (DT)
> [    0.738369] Call trace:
> 
> full start-up log attached.
> 
> kernel 6.8.12 and earlier builds boot no problem. 
> 
> Info:
> 
> - hardware: sheeva64 plug (Marvell Armada 3720 dual core ARMv8 Cortex-A53)
> - uboot version: U-Boot 2017.03-armada-18.09.1-gc338309136
>   - uboot init script with kernel cmdline is attached (I haven't changed this in at least a few years)
> - ext4 rootfs is on external usb drive
> - kernel 6.8.12 and earlier boot without issue (see attached: .config.6.8.12)
> - problem first seen in commit: 6d75c6f40a03c97e1ecd683ae54e249abb9d922b (see attached: git_bisect.log)
> - any kernel tagged 6.9x and later that I've tried fails with above error (see attached: .config.6.9.8.broken)
> - I'm building the kernel by cross-compiling on debian "testing" (`crossbuild-essential-arm64`)
> - the `initrd.img-6.9x` file itself appears valid:
>   - `lsinitramfs` works no problem
>   - `file initrd.img-6.9x` output is the same as the working initrds (`Zstandard compressed data (v0.8+), Dictionary ID: None`)
> 
> The bisected commit 6d75c6f40a03c97e1ecd683ae54e249abb9d922b is a merge of some 130+ arm64-related commits, so I'm having a hard time narrowing this down further. I'm glad to help test/troubleshoot but I'd need someone to give me some pointers.
> 
> thanks.
> 
> Jack Donohue

See the ticket for more details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

P.S.: let me use this mail to also add the report to the list of tracked
regressions to ensure it's doesn't fall through the cracks:

#regzbot introduced: 6d75c6f40a03c97e1ecd683ae54e249abb9d922b
#regzbot title: arm64: Initramfs unpacking failed: invalid magic at
start of compressed archive
#regzbot from: Jack D <wm2vdghq@duck.com>
#regzbot duplicate: https://bugzilla.kernel.org/show_bug.cgi?id=219014
#regzbot ignore-activity

Re: Boot error since 6.9: Initramfs unpacking failed: invalid magic at start of compressed archive

[Will Deacon]

On Mon, Jul 15, 2024 at 08:24:04AM +0200, Thorsten Leemhuis wrote:
> Hi, Thorsten here, the Linux kernel's regression tracker.
> 
> I noticed a report about a regression in bugzilla.kernel.org that a user
> bisected to the main ARM64 merge commit from the 6.9 cycle. So the real
> issue might be somewhere else, but some help would be appreciated. To
> quote from https://bugzilla.kernel.org/show_bug.cgi?id=219014 :
> 
> > Hi.
> > 
> > I'm marking as arm64 since git bisect identifies the first problematic commit as:
> >   6d75c6f40a03c97e1ecd683ae54e249abb9d922b
> > 
> > git bisect log is attached.
> > 
> > Any kernel tagged v6.9x and later fails to boot with the following error:
> > 
> > [    0.420154] Initramfs unpacking failed: invalid magic at start of compressed archive

^^^

My guess (based on times I've seen this sort of thing in the past) is
that the kernel binary grew in size and the broken bootloader is loading
the kernel over the initramfs instead of parsing the 'image_size' field
in the boot header.

Will

Re: Boot error since 6.9: Initramfs unpacking failed: invalid magic at start of compressed archive

[wm2vdghq]

On Monday, July 15, 2024 at 03:04:34 AM PDT, Will Deacon wrote:
> On Mon, Jul 15, 2024 at 08:24:04AM +0200, Thorsten Leemhuis wrote:
> quote from https://bugzilla.kernel.org/show_bug.cgi?id=219014 :
> > 
> > I'm marking as arm64 since git bisect identifies the first problematic commit as:
> >  6d75c6f40a03c97e1ecd683ae54e249abb9d922b
> > 
> > git bisect log is attached.
> > 
> > Any kernel tagged v6.9x and later fails to boot with the following error:
> > 
> > [    0.420154] Initramfs unpacking failed: invalid magic at start of compressed archive

> My guess (based on times I've seen this sort of thing in the past) is
> that the kernel binary grew in size and the broken bootloader is loading
> the kernel over the initramfs instead of parsing the 'image_size' field
> in the boot header.

(I hope I got the email formatting for this list correct. Apologies if I messed it up.)

Yes, that was it. Thank you for your quick (and accurate) assessment. The commit
increased my uncompressed kernel by ~1MB. Not much, but enough to cause an
address conflict.