Warning from free_init_pages with large initrd

Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

Hi,

While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
throws the following warning:

[    0.955216] Unpacking initramfs...
[    5.391977] ------------[ cut here ]------------
[    5.391986] WARNING: CPU: 0 PID: 1 at arch/x86/mm/init.c:671 free_init_pages+0x94/0xa0
[    5.391988] Modules linked in:
[    5.391992] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.0-0.rc7.git2.1.fc26.x86_64 #1
[    5.391993] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
[    5.391995]  0000000000000286 00000000c1677634 ffff9d943abffd58 ffffffffb6467073
[    5.391999]  0000000000000000 0000000000000000 ffff9d943abffd98 ffffffffb60ae7ab
[    5.392001]  0000029fb71cab9d ffff9d9300000000 ffff9d9380000000 ffff9d9329861000
[    5.392003] Call Trace:
[    5.392008]  [<ffffffffb6467073>] dump_stack+0x86/0xc3
[    5.392020]  [<ffffffffb60ae7ab>] __warn+0xcb/0xf0
[    5.392022]  [<ffffffffb60ae8dd>] warn_slowpath_null+0x1d/0x20
[    5.392024]  [<ffffffffb606b094>] free_init_pages+0x94/0xa0
[    5.392027]  [<ffffffffb71cabe3>] ? unpack_to_rootfs+0x29d/0x29d
[    5.392039]  [<ffffffffb71c88cd>] ? set_debug_rodata+0x12/0x12
[    5.392042]  [<ffffffffb71e6aee>] free_initrd_mem+0x21/0x23
[    5.392044]  [<ffffffffb71cacc8>] populate_rootfs+0xe5/0x105
[    5.392046]  [<ffffffffb6002190>] do_one_initcall+0x50/0x180
[    5.392048]  [<ffffffffb71c88cd>] ? set_debug_rodata+0x12/0x12
[    5.392049]  [<ffffffffb71c923b>] kernel_init_freeable+0x200/0x29c
[    5.392053]  [<ffffffffb68e6b4e>] kernel_init+0xe/0x100
[    5.392055]  [<ffffffffb68f7b2f>] ret_from_fork+0x1f/0x40
[    5.392056]  [<ffffffffb68e6b40>] ? rest_init+0x140/0x140
[    5.392062] ---[ end trace 0fbf5e487fcf7e56 ]---
[    6.572994] Freeing initrd memory: 1416828K (ffff9d9329861000 - ffff9d9380000000)

The initrd is big because it holds all the files in the ISO (see
https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was
made). The issue was originally filed on
https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually
against older kernels...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Warning from free_init_pages with large initrd

[H. Peter Anvin]

On 09/24/16 08:32, Sitsofe Wheeler wrote:
> Hi,
> 
> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
> throws the following warning:
> 
[...]
> 
> The initrd is big because it holds all the files in the ISO (see
> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was
> made). The issue was originally filed on
> https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually
> against older kernels...
> 

How big is big?

Re: Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

On 26 September 2016 at 03:14, H. Peter Anvin <hpa@zytor.com> wrote:
> On 09/24/16 08:32, Sitsofe Wheeler wrote:
>>
>> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
>> throws the following warning:
>>
> [...]
>>
>> The initrd is big because it holds all the files in the ISO (see
>> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it was
>> made). The issue was originally filed on
>> https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually
>> against older kernels...
>
> How big is big?

The one that matches that backtrace is 1385 Mbytes and was made from a
Fedora rawhide ISO. Here are some sizes in megabytes from different
versions of Fedora:
960   20/initrd0.img
1416 21/initrd0.img
1379 22/initrd0.img
1430 23/initrd0.img
1492 24/initrd.img

-- 
Sitsofe | http://sucs.org/~sits/

Re: Warning from free_init_pages with large initrd

[H. Peter Anvin]

On September 25, 2016 11:22:04 PM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>On 26 September 2016 at 03:14, H. Peter Anvin <hpa@zytor.com> wrote:
>> On 09/24/16 08:32, Sitsofe Wheeler wrote:
>>>
>>> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
>>> throws the following warning:
>>>
>> [...]
>>>
>>> The initrd is big because it holds all the files in the ISO (see
>>> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it
>was
>>> made). The issue was originally filed on
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually
>>> against older kernels...
>>
>> How big is big?
>
>The one that matches that backtrace is 1385 Mbytes and was made from a
>Fedora rawhide ISO. Here are some sizes in megabytes from different
>versions of Fedora:
>960   20/initrd0.img
>1416 21/initrd0.img
>1379 22/initrd0.img
>1430 23/initrd0.img
>1492 24/initrd.img

And they all fail?

Also, is this BIOS or EFI?

When loading under BIOS there is currently a 2 GB limit with most loaders; it would have been 4 GB except that Grub had a bug (sigh.)
-- 
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.

Re: Warning from free_init_pages with large initrd

[H. Peter Anvin]

On September 25, 2016 11:22:04 PM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>On 26 September 2016 at 03:14, H. Peter Anvin <hpa@zytor.com> wrote:
>> On 09/24/16 08:32, Sitsofe Wheeler wrote:
>>>
>>> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
>>> throws the following warning:
>>>
>> [...]
>>>
>>> The initrd is big because it holds all the files in the ISO (see
>>> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it
>was
>>> made). The issue was originally filed on
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1274966 but was usually
>>> against older kernels...
>>
>> How big is big?
>
>The one that matches that backtrace is 1385 Mbytes and was made from a
>Fedora rawhide ISO. Here are some sizes in megabytes from different
>versions of Fedora:
>960   20/initrd0.img
>1416 21/initrd0.img
>1379 22/initrd0.img
>1430 23/initrd0.img
>1492 24/initrd.img

Also: compressed or uncompressed (if it makes a difference?)
-- 
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.

Re: Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

On 26 September 2016 at 07:47, H. Peter Anvin <hpa@zytor.com> wrote:
> On September 25, 2016 11:22:04 PM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>On 26 September 2016 at 03:14, H. Peter Anvin <hpa@zytor.com> wrote:
>>> On 09/24/16 08:32, Sitsofe Wheeler wrote:
>>>>
>>>> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
>>>> throws the following warning:
>>>>
>>> [...]
>>>>
>>>> The initrd is big because it holds all the files in the ISO (see
>>>> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it
>>was
>>>> made).
>>>
>>> How big is big?
>>
>>The one that matches that backtrace is 1385 Mbytes and was made from a
>>Fedora rawhide ISO. Here are some sizes in megabytes from different
>>versions of Fedora:
>>960   20/initrd0.img
>>1416 21/initrd0.img
>>1379 22/initrd0.img
>>1430 23/initrd0.img
>>1492 24/initrd.img

> And they all fail?
>From at least 23 onwards they all continue to boot but give the
previously mentioned warning in dmesg output.

> Also, is this BIOS or EFI?
Booting from BIOS, not EFI.

> Also: compressed or uncompressed (if it makes a difference?)

file reports the following
20 initrd starts with a gzip header
21-23 initrds start with a cpio header
24 and rawhide initrd starts with an xz header

Looking at livecd-iso-to-pxeboot it seems that the CD's contents is
turned into a gzip'd cpio and then concatenated on the end of the
usual initrd...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Warning from free_init_pages with large initrd

[Randy Dunlap]

On 09/26/16 10:26, Sitsofe Wheeler wrote:
> On 26 September 2016 at 07:47, H. Peter Anvin <hpa@zytor.com> wrote:
>> On September 25, 2016 11:22:04 PM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>> On 26 September 2016 at 03:14, H. Peter Anvin <hpa@zytor.com> wrote:
>>>> On 09/24/16 08:32, Sitsofe Wheeler wrote:
>>>>>
>>>>> While trying to PXE boot a Fedora LiveISO on VMware ESXi the kernel
>>>>> throws the following warning:
>>>>>
>>>> [...]
>>>>>
>>>>> The initrd is big because it holds all the files in the ISO (see
>>>>> https://fedorahosted.org/cobbler/wiki/HowToPxeAnyLiveCd for how it
>>> was
>>>>> made).
>>>>
>>>> How big is big?
>>>
>>> The one that matches that backtrace is 1385 Mbytes and was made from a
>>> Fedora rawhide ISO. Here are some sizes in megabytes from different
>>> versions of Fedora:
>>> 960   20/initrd0.img
>>> 1416 21/initrd0.img
>>> 1379 22/initrd0.img
>>> 1430 23/initrd0.img
>>> 1492 24/initrd.img
> 
>> And they all fail?
> From at least 23 onwards they all continue to boot but give the
> previously mentioned warning in dmesg output.
> 
>> Also, is this BIOS or EFI?
> Booting from BIOS, not EFI.
> 
>> Also: compressed or uncompressed (if it makes a difference?)
> 
> file reports the following
> 20 initrd starts with a gzip header
> 21-23 initrds start with a cpio header
> 24 and rawhide initrd starts with an xz header
> 
> Looking at livecd-iso-to-pxeboot it seems that the CD's contents is
> turned into a gzip'd cpio and then concatenated on the end of the
> usual initrd...
> 

but the warning in free_init_pages() is about alignment, not size...
Maybe the concatenation is bad?

-- 
~Randy

Re: Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for
the history of this thread )

On 26 September 2016 at 20:00, Randy Dunlap <rdunlap@infradead.org> wrote:
>
> but the warning in free_init_pages() is about alignment, not size...
> Maybe the concatenation is bad?

What would l have to pull apart to be able to tell?

Having said that I've just noticed that newer versions of the script
concatenate an ISO to the initrd (rather than another cpio) -
https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
. Unfortunately after rebuilding the initrd with the
livecd-iso-to-pxeboot version from Fedora 24 the warning persists.
CC'ing a few of the tool's authors on this email.

-- 
Sitsofe | http://sucs.org/~sits/

Re: Warning from free_init_pages with large initrd

[Richard W.M. Jones]

On Tue, Sep 27, 2016 at 11:20:06AM +0100, Sitsofe Wheeler wrote:
> (See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for
> the history of this thread )
> 
> On 26 September 2016 at 20:00, Randy Dunlap <rdunlap@infradead.org> wrote:
> >
> > but the warning in free_init_pages() is about alignment, not size...
> > Maybe the concatenation is bad?
> 
> What would l have to pull apart to be able to tell?
> 
> Having said that I've just noticed that newer versions of the script
> concatenate an ISO to the initrd (rather than another cpio) -
> https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
> . Unfortunately after rebuilding the initrd with the
> livecd-iso-to-pxeboot version from Fedora 24 the warning persists.
> CC'ing a few of the tool's authors on this email.

I would just comment that I wrote livecd-iso-to-pxeboot for
a specific purpose -- virt-p2v -- where the size of the ISO
rarely exceeded around 100-200 MB, which is large for an initramfs,
but about an order of magnitude smaller than what you are trying.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: Warning from free_init_pages with large initrd

[H. Peter Anvin]

On September 27, 2016 3:20:06 AM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for
>the history of this thread )
>
>On 26 September 2016 at 20:00, Randy Dunlap <rdunlap@infradead.org>
>wrote:
>>
>> but the warning in free_init_pages() is about alignment, not size...
>> Maybe the concatenation is bad?
>
>What would l have to pull apart to be able to tell?
>
>Having said that I've just noticed that newer versions of the script
>concatenate an ISO to the initrd (rather than another cpio) -
>https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
>. Unfortunately after rebuilding the initrd with the
>livecd-iso-to-pxeboot version from Fedora 24 the warning persists.
>CC'ing a few of the tool's authors on this email.

This really doesn't seem like a good approach, and certainly may cause memory exhaustion at boot time.  Still would be good to track down the problem.

Overall the idea of downloading the whole shebang in the initial TFTP transaction ID a really bad idea.  TFTP is slow and unreliable, and the PXE ROMs are often complete crap in terms of quality.  It would be way better to bring up a small initramfs and then fetch the rest using http or a similar protocol from the very high quality Linux TCP stack.  As far as I know, Fedora has supported this mode for *ages*.
-- 
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.

Re: Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

On 27 September 2016 at 19:36, H. Peter Anvin <hpa@zytor.com> wrote:
> On September 27, 2016 3:20:06 AM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for
>>the history of this thread )
>>
>>On 26 September 2016 at 20:00, Randy Dunlap <rdunlap@infradead.org>
>>wrote:
>>>
>>> but the warning in free_init_pages() is about alignment, not size...
>>> Maybe the concatenation is bad?
>>
>>What would l have to pull apart to be able to tell?
>>
>>Having said that I've just noticed that newer versions of the script
>>concatenate an ISO to the initrd (rather than another cpio) -
>>https://github.com/rhinstaller/livecd-tools/commit/8067be50907da9461e442c11a664c89e066ccac6#diff-88c69e43bb69726c532af3a136cc50e8
>>. Unfortunately after rebuilding the initrd with the
>>livecd-iso-to-pxeboot version from Fedora 24 the warning persists.
>>CC'ing a few of the tool's authors on this email.
>
> This really doesn't seem like a good approach, and certainly may cause memory exhaustion at boot time.  Still would be good to track down the problem.
>
> Overall the idea of downloading the whole shebang in the initial TFTP transaction ID a really bad idea.  TFTP is slow and unreliable, and the PXE ROMs are often complete crap in terms of quality.  It would be way better to bring up a small initramfs and then fetch the rest using http or a similar protocol from the very high quality Linux TCP stack.  As far as I know, Fedora has supported this mode for *ages*.

I wouldn't mind tracking down the problem too but I admit I don't know
where to start on this alignment issue.

As for the process, it can be excruciating slow and can cause memory
exhaustion but the initrd and vmlinuz are actually downloaded over
HTTP courtesy of iPXE (so grabbing 1GByte happens in under a minute
over HTTP) and these days machines have many gigabytes of memory so it
actually works quite well once booted...

I was going to say that Fedora doesn't allow PXE booting of its live
images (I know it always supported something for server installs) but
I've just come across this
https://bugzilla.redhat.com/show_bug.cgi?id=1317780 so perhaps there's
a another route to take in addition to this next time round...

-- 
Sitsofe | http://sucs.org/~sits/

Re: Warning from free_init_pages with large initrd

[Sitsofe Wheeler]

On 27 September 2016 at 22:21, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> On 27 September 2016 at 19:36, H. Peter Anvin <hpa@zytor.com> wrote:
>> On September 27, 2016 3:20:06 AM PDT, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>(See http://www.gossamer-threads.com/lists/linux/kernel/2534175 for
>>>the history of this thread )
>>>
>>>On 26 September 2016 at 20:00, Randy Dunlap <rdunlap@infradead.org>
>>>wrote:
>>>>
>>>> but the warning in free_init_pages() is about alignment, not size...
>>>> Maybe the concatenation is bad?
>>>
>>>What would l have to pull apart to be able to tell?

This turned out to be down to a bug in iPXE initrd alignment which is
fixed in https://git.ipxe.org/ipxe.git/commit/aa11f5deda29f91f5b8592965ed00f4a65002eff
and impacted any initrd.

-- 
Sitsofe | http://sucs.org/~sits/