Re: Cannot boot PVH dom0 with big initrd

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 8135 bytes --]

On Fri, Feb 13, 2026 at 09:40:31PM +0100, Roger Pau Monné wrote:
> On Fri, Feb 13, 2026 at 04:56:39PM +0100, Roger Pau Monné wrote:
> > On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> > > On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > After fixing the xhci crash, I hit another issue - booting with 236MB
> > > > initrd doesn't work, I get:
> > > > 
> > > >     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> > > >     ...
> > > >     (XEN) [    3.593940] Unable to allocate memory with order 0!
> > > >     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> > > >     (XEN) [    3.599884] 
> > > >     (XEN) [    3.602482] ****************************************
> > > >     (XEN) [    3.605272] Panic on CPU 0:
> > > >     (XEN) [    3.607928] Could not construct d0
> > > >     (XEN) [    3.610692] ****************************************
> > > >     (XEN) [    3.613463] 
> > > >     (XEN) [    3.616035] Reboot in five seconds...
> > > >     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> > > > 
> > > > Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> > > > 
> > > > If I skip initrd, then it boots just fine (but dom0 is not happy about
> > > > that). 164MB initrd failed too, but 13MB started ok.
> > > > Just in case, I tried skipping XHCI console, but it didn't change
> > > > anything.
> > > > 
> > > > Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> > > > started from GRUB, using MB2+EFI.
> > > 
> > > Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> > > for passing from the boot loader to Xen) only after copying to the designated
> > > guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> > > its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> > > isn't affected because to load huge initrd there, the kernel has to request
> > > the initrd to not be mapped into the initial allocation.
> > 
> > Right, on PV dom0 we do not copy the image to a new set of pages, we
> > simply assign the pages where the initrd resides to the domain.  We
> > can't populate those pages in the p2m as-is, otherwise we would
> > shatter super pages.
> > 
> > I think the fix below should do it, it's likely the best we can do.
> > Can you please give it a try Marek?
> > 
> > Thanks, Roger.
> > ---
> > diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> > index 0b467fd4a4fc..8e3cb5d0db76 100644
> > --- a/xen/arch/x86/dom0_build.c
> > +++ b/xen/arch/x86/dom0_build.c
> > @@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
> >  
> >      for_each_node_mask ( node, dom0_nodes )
> >          avail += avail_domheap_pages_region(node, 0, 0) +
> > -                 initial_images_nrpages(node);
> > +                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
> >  
> >      /* Reserve memory for further dom0 vcpu-struct allocations... */
> >      avail -= (d->max_vcpus - 1UL)
> 
> I'm working on a more complex patch, that attempts to account the
> memory used by the init images towards the reserved amount that's kept
> by Xen.  This should make accounting a bit better, in that we won't
> end up reserving the Xen memory plus the memory used by the init
> images.
> 
> It's still however a WIP, but would you mind giving it a try?

This one worked :)

    (XEN) [    4.014243] Dom0 memory allocation stats:
    (XEN) [    4.017378] order  0 allocations: 4
    (XEN) [    4.020111] order  1 allocations: 3
    (XEN) [    4.022758] order  2 allocations: 4
    (XEN) [    4.025406] order  3 allocations: 4
    (XEN) [    4.028150] order  4 allocations: 3
    (XEN) [    4.030886] order  5 allocations: 2
    (XEN) [    4.033602] order  6 allocations: 2
    (XEN) [    4.036305] order  7 allocations: 6
    (XEN) [    4.039020] order  8 allocations: 6
    (XEN) [    4.041597] order  9 allocations: 5
    (XEN) [    4.044261] order 10 allocations: 9
    (XEN) [    4.046831] order 11 allocations: 7
    (XEN) [    4.049493] order 12 allocations: 9
    (XEN) [    4.052147] order 13 allocations: 7
    (XEN) [    4.054799] order 14 allocations: 6
    (XEN) [    4.057447] order 15 allocations: 7
    (XEN) [    4.060080] order 16 allocations: 7
    (XEN) [    4.062609] order 17 allocations: 5
    (XEN) [    4.065227] order 18 allocations: 9
    (XEN) [    4.921719] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
    (XEN) [    4.924403] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
    (XEN) [    4.927079] ELF: memory: 0x200000 -> 0x3e00000
    (XEN) [    4.929759] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
    (XEN) [    4.932884] ELF: note: PHYS32_ENTRY = 0x16a2ca0
    (XEN) [    4.935921] ELF: note: GUEST_OS = "linux"
    (XEN) [    4.938953] ELF: note: GUEST_VERSION = "2.6"
    (XEN) [    4.942005] ELF: note: XEN_VERSION = "xen-3.0"
    (XEN) [    4.945077] ELF: note: VIRT_BASE = 0xffffffff80000000
    (XEN) [    4.948133] ELF: note: INIT_P2M = 0x8000000000
    (XEN) [    4.951203] ELF: note: ENTRY = 0xffffffff82d3c160
    (XEN) [    4.954221] ELF: note: FEATURES = "!writable_page_tables"
    (XEN) [    4.957229] ELF: note: PAE_MODE = "yes"
    (XEN) [    4.960175] ELF: note: L1_MFN_VALID
    (XEN) [    4.962775] ELF: note: MOD_START_PFN = 0x1
    (XEN) [    4.965675] ELF: note: PADDR_OFFSET = 0
    (XEN) [    4.968540] ELF: note: SUPPORTED_FEATURES = 0x8801
    (XEN) [    4.971420] ELF: note: LOADER = "generic"
    (XEN) [    4.974303] ELF: note: SUSPEND_CANCEL = 0x1
    (XEN) [    4.977186] ELF: Found PVH image
    (XEN) [    4.979910] ELF: addresses:
    (XEN) [    4.982237]     virt_base        = 0x0
    (XEN) [    4.984694]     elf_paddr_offset = 0x0
    (XEN) [    4.987141]     virt_offset      = 0x0
    (XEN) [    4.989599]     virt_kstart      = 0x200000
    (XEN) [    4.992044]     virt_kend        = 0x3e00000
    (XEN) [    4.994498]     virt_entry       = 0x16a2ca0
    (XEN) [    4.996966]     p2m_base         = 0x8000000000
    (XEN) [    4.999415] ELF: phdr 0 at 0x200000 -> 0x21f3928
    (XEN) [    5.007160] ELF: phdr 1 at 0x2200000 -> 0x3e00000
    (XEN) [    5.055448] Dom0 memory map:
    (XEN) [    5.057763]  [0000000000000000, 000000000009efff] (usable)
    (XEN) [    5.060281]  [000000000009f000, 00000000000fffff] (reserved)
    (XEN) [    5.062812]  [0000000000100000, 000000005471afff] (usable)
    (XEN) [    5.065324]  [000000005471b000, 000000005475bfff] (reserved)
    (XEN) [    5.067891]  [000000005475c000, 0000000063c2dfff] (usable)
    (XEN) [    5.070446]  [0000000063c2e000, 000000006d17afff] (reserved)
    (XEN) [    5.073036]  [000000006d17b000, 000000006d22bfff] (ACPI data)
    (XEN) [    5.075649]  [000000006d22c000, 000000006d2ebfff] (ACPI NVS)
    (XEN) [    5.078267]  [000000006d2ec000, 000000006fffefff] (reserved)
    (XEN) [    5.080908]  [000000006ffff000, 000000006ffffdcb] (usable)
    (XEN) [    5.083560]  [000000006ffffdcc, 000000006ffffe97] (ACPI data)
    (XEN) [    5.086243]  [0000000070000000, 00000000807fffff] (reserved)
    (XEN) [    5.088943]  [00000000c0000000, 00000000cfffffff] (reserved)
    (XEN) [    5.091662]  [00000000fe000000, 00000000fe010fff] (reserved)
    (XEN) [    5.094413]  [00000000fec00000, 00000000fec00fff] (reserved)
    (XEN) [    5.097193]  [00000000fed00000, 00000000fed00fff] (reserved)
    (XEN) [    5.099976]  [00000000fed20000, 00000000fed7ffff] (reserved)
    (XEN) [    5.102779]  [00000000fee00000, 00000000fee00fff] (reserved)
    (XEN) [    5.105615]  [00000000ff000000, 00000001023fffff] (reserved)
    (XEN) [    5.108487]  [0000000102400000, 000000045c89cfff] (usable)
    (XEN) [    5.111354]  [000000045c89d000, 000000047f7fffff] (unusable)
    (XEN) [    5.114263] Initial low memory virq threshold set at 0x4000 pages.
    (XEN) [    5.117108] Scrubbing Free RAM in background

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]