[Qemu-devel] alpha platform is missing files after initrd load

[Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

qemu: 2.7.x (git head)
platform: Alpha (Clipper)
kernel: 4.7.0
gcc: 6.1

i don't know if its an qemu oder linux kernel problem

i've got an ~360MB big_initrd.cpio and it sometimes happen(seems so) that
there are files missing after the kernel loaded the initrd
this does not happen with the same programs/initrd compiled for the 
sparc64 platform using qemu

i've got a small C/C++-based init that prints some infos about the 
system and
prints the content of the /tools/bin folder and there a some files 
missing that are
definitely in the cpio file

im starting qemu with

alpha-softmmu/qemu-system-alpha -m 1GB -nographic -monitor 
telnet::4440,server,nowait -serial telnet::3000,server -kernel 
clfskernel -append 'console=ttyS0' -initrd big_initrd.cpio

i've found a problem description on stackoverflow going in the same 
direction but im not using an in-kernel-ramfs
http://stackoverflow.com/questions/31524636/files-disappearing-from-initramfs

i've uploaded my complete test here: 
http://www.filedropper.com/testalpha-linux-470gcc-610 (~66MB)

containing files:
big_initrd.cpio -> the big cpio which misses files after load
clfskernel -> linux 4.7.0 (default settings)
config -> used config
init.cpp -> my init
initrd.cpio -> only kernel+init cpio
kernel.out.txt -> output of the kernel and init
start_big.sh -> my starting script for qemu loading big_initrd.cpio
start_small.sh -> my starting script for qemu loading initrd.cpio
System.map -> kernel map file
tools_dir_after_load.txt -> files in /tools/bin (copy&paste+sort from 
kernel.out.txt printdir /tools/bin)
tools_dir_in_cpio.txt -> files in the cpio:/tools/bin

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

On Thu, Oct 20, 2016 at 12:45:37PM +0200, Dennis Luehring wrote:
> qemu: 2.7.x (git head)
> platform: Alpha (Clipper)

Two options:

1. Not many people use Alpha.  You may need to debug this yourself by
   learning about the Linux alpha boot protocol (where the initramfs is
   loaded and how big that region of memory may be).  Then you can
   verify the memory contents after QEMU has loaded the
   kernel/initramfs using monitor commands to read memory.  You may need
   to look at QEMU's kernel/initramfs loading code to see what it's
   doing.

2. If it worked in a previous QEMU version, please use git-bisect(1) to
   find out which commit broke it.

Good luck!

> kernel: 4.7.0
> gcc: 6.1
> 
> i don't know if its an qemu oder linux kernel problem
> 
> i've got an ~360MB big_initrd.cpio and it sometimes happen(seems so) that
> there are files missing after the kernel loaded the initrd
> this does not happen with the same programs/initrd compiled for the sparc64
> platform using qemu
> 
> i've got a small C/C++-based init that prints some infos about the system
> and
> prints the content of the /tools/bin folder and there a some files missing
> that are
> definitely in the cpio file
> 
> im starting qemu with
> 
> alpha-softmmu/qemu-system-alpha -m 1GB -nographic -monitor
> telnet::4440,server,nowait -serial telnet::3000,server -kernel clfskernel
> -append 'console=ttyS0' -initrd big_initrd.cpio
> 
> i've found a problem description on stackoverflow going in the same
> direction but im not using an in-kernel-ramfs
> http://stackoverflow.com/questions/31524636/files-disappearing-from-initramfs
> 
> i've uploaded my complete test here:
> http://www.filedropper.com/testalpha-linux-470gcc-610 (~66MB)
> 
> containing files:
> big_initrd.cpio -> the big cpio which misses files after load
> clfskernel -> linux 4.7.0 (default settings)
> config -> used config
> init.cpp -> my init
> initrd.cpio -> only kernel+init cpio
> kernel.out.txt -> output of the kernel and init
> start_big.sh -> my starting script for qemu loading big_initrd.cpio
> start_small.sh -> my starting script for qemu loading initrd.cpio
> System.map -> kernel map file
> tools_dir_after_load.txt -> files in /tools/bin (copy&paste+sort from
> kernel.out.txt printdir /tools/bin)
> tools_dir_in_cpio.txt -> files in the cpio:/tools/bin
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Richard Henderson]

On 11/04/2016 05:08 AM, Stefan Hajnoczi wrote:
> On Thu, Oct 20, 2016 at 12:45:37PM +0200, Dennis Luehring wrote:
>> qemu: 2.7.x (git head)
>> platform: Alpha (Clipper)
>
> Two options:
>
> 1. Not many people use Alpha.  You may need to debug this yourself by
>    learning about the Linux alpha boot protocol (where the initramfs is
>    loaded and how big that region of memory may be).  Then you can
>    verify the memory contents after QEMU has loaded the
>    kernel/initramfs using monitor commands to read memory.  You may need
>    to look at QEMU's kernel/initramfs loading code to see what it's
>    doing.
>
> 2. If it worked in a previous QEMU version, please use git-bisect(1) to
>    find out which commit broke it.
>
> Good luck!

We debugged this via private mail.

For the 4.7 kernel, we had enough ram to unpack the (large) initrd; with the 
4.8 kernel, we ran out.  The 4.8 kernel did in fact print an (obscure) error 
message to that effect, which had not been noticed.  I consider it a bug that 
the kernel does not treat this like any other failure to mount /root, via panic.

Increasing ram from 1GB to 2GB allowed the 4.8 kernel to succeed in unpacking 
and booting off of the initrd.


r~

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/04/16 17:37, Richard Henderson wrote:
> On 11/04/2016 05:08 AM, Stefan Hajnoczi wrote:
>> On Thu, Oct 20, 2016 at 12:45:37PM +0200, Dennis Luehring wrote:
>>> qemu: 2.7.x (git head)
>>> platform: Alpha (Clipper)
>>
>> Two options:
>>
>> 1. Not many people use Alpha.  You may need to debug this yourself by
>>    learning about the Linux alpha boot protocol (where the initramfs is
>>    loaded and how big that region of memory may be).  Then you can
>>    verify the memory contents after QEMU has loaded the
>>    kernel/initramfs using monitor commands to read memory.  You may need
>>    to look at QEMU's kernel/initramfs loading code to see what it's
>>    doing.
>>
>> 2. If it worked in a previous QEMU version, please use git-bisect(1) to
>>    find out which commit broke it.
>>
>> Good luck!
> 
> We debugged this via private mail.
> 
> For the 4.7 kernel, we had enough ram to unpack the (large) initrd; with
> the 4.8 kernel, we ran out.  The 4.8 kernel did in fact print an
> (obscure) error message to that effect, which had not been noticed.

Was it

  Unpacking initramfs...
  Initramfs unpacking failed: junk in compressed archive

?

If so, then the same has been encountered with the aarch64 target:

https://github.com/libguestfs/libguestfs/commit/c24f242521e8

> I
> consider it a bug that the kernel does not treat this like any other
> failure to mount /root, via panic.

I agree.

And, apparently, it used to be handled with a panic() call, but then that was deemed "policy", and downgraded to a KERN_EMERG message:

commit 73310a169aebe257efdd35a763cce1c7658f40c9
Author: H. Peter Anvin <hpa@linux.intel.com>
Date:   Wed Jan 14 11:28:35 2009 -0800

    init: make initrd/initramfs decompression failure a KERN_EMERG event
    
    Impact: More consistent behaviour, avoid policy in the kernel
    
    Upgrade/downgrade initrd/initramfs decompression failure from
    inconsistently a panic or a KERN_ALERT message to a KERN_EMERG event.
    It is, however, possible do design a system which can recover from
    this (using the kernel builtin code and/or the internal initramfs),
    which means this is policy, not a technical necessity.
    
    A good way to handle this would be to have a panic-level=X option, to
    force a panic on a printk above a certain level.  That is a separate
    patch, however.
    
    Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>

I guess it is "possible to design a system which can recover from this", except noone seems to have bothered, since 2009. (Ditto for the proposed "panic-level=X" alternative.)

I've now briefly considered posting a trivial kernel patch for this, but having learned about the above commit, I don't think so...

Thanks
Laszlo

> Increasing ram from 1GB to 2GB allowed the 4.8 kernel to succeed in
> unpacking and booting off of the initrd.

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Richard Henderson]

On 11/04/2016 01:40 PM, Laszlo Ersek wrote:
>> For the 4.7 kernel, we had enough ram to unpack the (large) initrd; with
>> the 4.8 kernel, we ran out.  The 4.8 kernel did in fact print an
>> (obscure) error message to that effect, which had not been noticed.
>
> Was it
>
>   Unpacking initramfs...
>   Initramfs unpacking failed: junk in compressed archive

No, it was less specific than that.  Something like "unpacking failed: error".

> And, apparently, it used to be handled with a panic() call, but then that
> was deemed "policy", and downgraded to a KERN_EMERG message:

Ouch.  A silly decision, but definitely an uphill fight to get it changed.


r~

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 04.11.2016 um 20:40 schrieb Laszlo Ersek:
> I guess it is "possible to design a system which can recover from this", except noone seems to have bothered, since 2009. (Ditto for the proposed "panic-level=X" alternative.)
>
> I've now briefly considered posting a trivial kernel patch for this, but having learned about the above commit, I don't think so...

i tried to get some response on the linux-kernel or initramfs 
mailinglist 1-2 weeks ago but no one seems to care

but i still think it would help alot to if some more known developers 
would get involved and send in a patch for this - if only for starting a 
discussion about the

situation

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 04.11.2016 um 22:08 schrieb Richard Henderson:
> No, it was less specific than that.  Something like "unpacking failed: error".
>
>> >And, apparently, it used to be handled with a panic() call, but then that
>> >was deemed "policy", and downgraded to a KERN_EMERG message:
> Ouch.  A silly decision, but definitely an uphill fight to get it changed.


using a big(~370MB) initrd.cpio with not enough ram (1GB) in qemu i get 
this kernel message

"Initramfs unpacking failed: write error"

using a small(~14MB) initrd.cpio with just a staticaly linked c based
init for the kernel also with 1GB ram (more then enough) gives me this 
kernel message:

"Initramfs unpacking failed: junk in compressed archive"

and i currently can't say if only the first is evil or both

both cpios created with "find . | cpio -H newc -o initrd.cpio"

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/07/16 08:37, Dennis Luehring wrote:
> Am 04.11.2016 um 22:08 schrieb Richard Henderson:
>> No, it was less specific than that.  Something like "unpacking failed:
>> error".
>>
>>> >And, apparently, it used to be handled with a panic() call, but then
>>> that
>>> >was deemed "policy", and downgraded to a KERN_EMERG message:
>> Ouch.  A silly decision, but definitely an uphill fight to get it
>> changed.
> 
> 
> using a big(~370MB) initrd.cpio with not enough ram (1GB) in qemu i get
> this kernel message
> 
> "Initramfs unpacking failed: write error"
> 
> using a small(~14MB) initrd.cpio with just a staticaly linked c based
> init for the kernel also with 1GB ram (more then enough) gives me this
> kernel message:
> 
> "Initramfs unpacking failed: junk in compressed archive"
> 
> and i currently can't say if only the first is evil or both
> 
> both cpios created with "find . | cpio -H newc -o initrd.cpio"

The second error message is incorrect in its own right (it's just
another symptom of running out of memory; the compressed archive is
valid in fact). But, fixing that would require a more intrusive change I
guess, i.e., to distinguish these two kinds of problems properly.

Thanks
Laszlo

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/07/16 08:35, Dennis Luehring wrote:
> Am 04.11.2016 um 20:40 schrieb Laszlo Ersek:
>> I guess it is "possible to design a system which can recover from
>> this", except noone seems to have bothered, since 2009. (Ditto for the
>> proposed "panic-level=X" alternative.)
>>
>> I've now briefly considered posting a trivial kernel patch for this,
>> but having learned about the above commit, I don't think so...
> 
> i tried to get some response on the linux-kernel or initramfs
> mailinglist 1-2 weeks ago but no one seems to care
> 
> but i still think it would help alot to if some more known developers
> would get involved and send in a patch for this - if only for starting a
> discussion about the
> 
> situation
> 

Please report the issue directly to the author of commit 73310a169aeb,
that is, "H. Peter Anvin" <hpa@zytor.com> (the latter is his current
email address). Please identify the commit to him and feel free to
reference this discussion on qemu-devel, using the mailing list archive.

If Peter still believes the current logic is valid, no work (= patch)
will have been in vain. If Peter agrees the current logic is wrong after
all, then he should be able to write a patch for you that he won't
dislike (so several iterations won't be necessary for the patch).

Thanks
Laszlo

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/07/16 15:15, Laszlo Ersek wrote:
> On 11/07/16 08:35, Dennis Luehring wrote:
>> Am 04.11.2016 um 20:40 schrieb Laszlo Ersek:
>>> I guess it is "possible to design a system which can recover from
>>> this", except noone seems to have bothered, since 2009. (Ditto for the
>>> proposed "panic-level=X" alternative.)
>>>
>>> I've now briefly considered posting a trivial kernel patch for this,
>>> but having learned about the above commit, I don't think so...
>>
>> i tried to get some response on the linux-kernel or initramfs
>> mailinglist 1-2 weeks ago but no one seems to care
>>
>> but i still think it would help alot to if some more known developers
>> would get involved and send in a patch for this - if only for starting a
>> discussion about the
>>
>> situation
>>
> 
> Please report the issue directly to the author of commit 73310a169aeb,
> that is, "H. Peter Anvin" <hpa@zytor.com> (the latter is his current
> email address). Please identify the commit to him and feel free to
> reference this discussion on qemu-devel, using the mailing list archive.

Also, please CC <linux-kernel@vger.kernel.org> on your bug report.

Thanks
Laszlo

> 
> If Peter still believes the current logic is valid, no work (= patch)
> will have been in vain. If Peter agrees the current logic is wrong after
> all, then he should be able to write a patch for you that he won't
> dislike (so several iterations won't be necessary for the patch).
> 
> Thanks
> Laszlo
> 

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 07.11.2016 um 15:12 schrieb Laszlo Ersek:
> The second error message is incorrect in its own right (it's just
> another symptom of running out of memory;
there a two cpios

big(~370MB) initrd.cpio + 1GB ram gives only this message: "Initramfs 
unpacking failed: write error"
small(~14MB) initrd.cpio + 1GB ram gives only this message: "Initramfs 
unpacking failed: junk in compressed archive"

so the second one couldn't be a problem with running out of memory - and 
its not alpha related same error happens with ppc64,sparc64,mips64...

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 07.11.2016 um 15:15 schrieb Laszlo Ersek:
> On 11/07/16 08:35, Dennis Luehring wrote:
> > Am 04.11.2016 um 20:40 schrieb Laszlo Ersek:
> >> I guess it is "possible to design a system which can recover from
> >> this", except noone seems to have bothered, since 2009. (Ditto for the
> >> proposed "panic-level=X" alternative.)
> >>
> >> I've now briefly considered posting a trivial kernel patch for this,
> >> but having learned about the above commit, I don't think so...
> >
> > i tried to get some response on the linux-kernel or initramfs
> > mailinglist 1-2 weeks ago but no one seems to care
> >
> > but i still think it would help alot to if some more known developers
> > would get involved and send in a patch for this - if only for starting a
> > discussion about the
> >
> > situation
> >
>
> Please report the issue directly to the author of commit 73310a169aeb,
> that is, "H. Peter Anvin" <hpa@zytor.com> (the latter is his current
> email address). Please identify the commit to him and feel free to
> reference this discussion on qemu-devel, using the mailing list archive.
>
> If Peter still believes the current logic is valid, no work (= patch)
> will have been in vain. If Peter agrees the current logic is wrong after
> all, then he should be able to write a patch for you that he won't
> dislike (so several iterations won't be necessary for the patch).
>
> Thanks
> Laszlo

will do and refere to this discussion here - thx

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/07/16 15:38, Dennis Luehring wrote:
> Am 07.11.2016 um 15:12 schrieb Laszlo Ersek:
>> The second error message is incorrect in its own right (it's just
>> another symptom of running out of memory;
> there a two cpios
> 
> big(~370MB) initrd.cpio + 1GB ram gives only this message: "Initramfs
> unpacking failed: write error"
> small(~14MB) initrd.cpio + 1GB ram gives only this message: "Initramfs
> unpacking failed: junk in compressed archive"
> 
> so the second one couldn't be a problem with running out of memory - and
> its not alpha related same error happens with ppc64,sparc64,mips64...

The second one *is* related to running out of memory. Initrd
decompression takes unintuitive amounts of memory. And, in practice,
exactly the second error message was seen with the aarch64 target, until
Rich Jones raised the guest RAM size that libguestfs passed to QEMU.

You can try it for yourself: stick with the small (14MB) initrd, and,
say, quadruple the guest RAM size. I'm pretty sure it will succeed.

The difference between the error messages (and the initrd sizes) is that
the initrd decompression runs out of memory at different stages. With
the huge initrd, unpacking likely fails in one of the early stages. With
the smaller one, unpacking likely fails later. I never bothered to
figure out where exactly.

Thanks
Laszlo

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 07.11.2016 um 16:56 schrieb Laszlo Ersek:
>> >so the second one couldn't be a problem with running out of memory - and
>> >its not alpha related same error happens with ppc64,sparc64,mips64...
> The second one*is*  related to running out of memory. Initrd
> decompression takes unintuitive amounts of memory. And, in practice,
> exactly the second error message was seen with the aarch64 target, until
> Rich Jones raised the guest RAM size that libguestfs passed to QEMU.
>
> You can try it for yourself: stick with the small (14MB) initrd, and,
> say, quadruple the guest RAM size. I'm pretty sure it will succeed.

can't give the quest more then 4.2GB ram - my host does not have more available - but

small(~14MB) initrd.cpio + 4GB ram still gives

"Initramfsunpacking failed: junk in compressed archive"

and there is just *one* 14MB init-binary file in the cpio - nothing more, no compression

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Laszlo Ersek]

On 11/07/16 17:11, Dennis Luehring wrote:
> Am 07.11.2016 um 16:56 schrieb Laszlo Ersek:
>>> >so the second one couldn't be a problem with running out of memory -
>>> and
>>> >its not alpha related same error happens with ppc64,sparc64,mips64...
>> The second one*is*  related to running out of memory. Initrd
>> decompression takes unintuitive amounts of memory. And, in practice,
>> exactly the second error message was seen with the aarch64 target, until
>> Rich Jones raised the guest RAM size that libguestfs passed to QEMU.
>>
>> You can try it for yourself: stick with the small (14MB) initrd, and,
>> say, quadruple the guest RAM size. I'm pretty sure it will succeed.
> 
> can't give the quest more then 4.2GB ram - my host does not have more
> available - but
> 
> small(~14MB) initrd.cpio + 4GB ram still gives
> 
> "Initramfsunpacking failed: junk in compressed archive"
> 
> and there is just *one* 14MB init-binary file in the cpio - nothing
> more, no compression
> 

I stand corrected then; the problem you see must be genuinely different
from what Rich saw with the libguestfs appliance initrd.

Maybe the decompressor code doesn't expect to find only one file in the
initrd. Just guessing.

Laszlo

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 07.11.2016 um 18:21 schrieb Laszlo Ersek:
> its not alpha related same error happens with ppc64,sparc64,mips64...

so i tried to reproduce the errors also on x86_64 in qemu

building a tiny linux kernel 4.8.6 (~1MB) and a simple init (~1MB) based on http://mgalgs.github.io/2015/05/16/how-to-build-a-custom-linux-kernel-for-qemu-2015-edition.html

until now (using the same cpio commandline) im unable to get the same errors like on alpha, ppc64,sparc64,mips64

im using cross-linux-from-scratch manual/script for building the non-x86_64 platforms

Re: [Qemu-devel] alpha platform is missing files after initrd load

[Dennis Luehring]

Am 09.11.2016 um 07:34 schrieb Dennis Luehring:
> im using cross-linux-from-scratch manual/script for building the non-x86_64 platforms

fyi

after using the crosscompilers from the gcc-5-crossports packages 
(http://packages.ubuntu.com/source/xenial/gcc-5-cross-ports)

alpha-linux-gnu-gcc-5
sparc64-linux-gnu-gcc-5
mips64-linux-gnuabi64-gcc-5
powerpc64-linux-gnu-gcc-5
sadly no sparc, arm gccs :)

kompiling the kernels using my standard configs and init-programm

alpha - my small initrd cpio boots without "junk in compressed archive" 
message
mips64 - does not compile
ppc64 - does not boot
sparc64 - my small initrd cpio boots without "junk in compressed 
archive" message

so i only get these "junk in compressed archive" messages with my own 
gcc builds
and whats also interesting is that the excutables are 10 times smaller 
(6MB:59MB alpha kernel...) using the prebuild compilers

so i think the warning that it is not easy to build working cross 
compilers for nearly all the qemu platforms from source seems
to be true and i need to check my gcc compilation settings against the 
prebuild ones - never thought it would be
that hard to build linux kernels based on self compiled gccs... in such 
limited spare time :)

but thx for all your help