Cannot boot PVH dom0 with big initrd

Cannot boot PVH dom0 with big initrd

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 6616 bytes --]

Hi,

After fixing the xhci crash, I hit another issue - booting with 236MB
initrd doesn't work, I get:

    (XEN) [    3.151856] *** Building a PVH Dom0 ***
    ...
    (XEN) [    3.593940] Unable to allocate memory with order 0!
    (XEN) [    3.597110] Failed to setup Dom0 physical memory map
    (XEN) [    3.599884] 
    (XEN) [    3.602482] ****************************************
    (XEN) [    3.605272] Panic on CPU 0:
    (XEN) [    3.607928] Could not construct d0
    (XEN) [    3.610692] ****************************************
    (XEN) [    3.613463] 
    (XEN) [    3.616035] Reboot in five seconds...
    (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.

Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565

If I skip initrd, then it boots just fine (but dom0 is not happy about
that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change
anything.

Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
started from GRUB, using MB2+EFI.

When it works (12MB initrd), I get the following info:

    (XEN) [    4.123843] Dom0 memory allocation stats:
    (XEN) [    4.126636] order  0 allocations: 4
    (XEN) [    4.129278] order  1 allocations: 3
    (XEN) [    4.132027] order  2 allocations: 4
    (XEN) [    4.134776] order  3 allocations: 3
    (XEN) [    4.137527] order  4 allocations: 3
    (XEN) [    4.140252] order  5 allocations: 3
    (XEN) [    4.142966] order  7 allocations: 2
    (XEN) [    4.145663] order  8 allocations: 3
    (XEN) [    4.148358] order  9 allocations: 3
    (XEN) [    4.151035] order 10 allocations: 4
    (XEN) [    4.153709] order 11 allocations: 7
    (XEN) [    4.156277] order 12 allocations: 9
    (XEN) [    4.158940] order 13 allocations: 6
    (XEN) [    4.161604] order 14 allocations: 6
    (XEN) [    4.164251] order 15 allocations: 7
    (XEN) [    4.166892] order 16 allocations: 6
    (XEN) [    4.169521] order 17 allocations: 4
    (XEN) [    4.172048] order 18 allocations: 10
    (XEN) [    4.994309] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
    (XEN) [    4.997011] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
    (XEN) [    4.999686] ELF: memory: 0x200000 -> 0x3e00000
    (XEN) [    5.002404] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
    (XEN) [    5.005451] ELF: note: PHYS32_ENTRY = 0x16a2ca0
    (XEN) [    5.008519] ELF: note: GUEST_OS = "linux"
    (XEN) [    5.011562] ELF: note: GUEST_VERSION = "2.6"
    (XEN) [    5.014634] ELF: note: XEN_VERSION = "xen-3.0"
    (XEN) [    5.017712] ELF: note: VIRT_BASE = 0xffffffff80000000
    (XEN) [    5.020795] ELF: note: INIT_P2M = 0x8000000000
    (XEN) [    5.023856] ELF: note: ENTRY = 0xffffffff82d3c160
    (XEN) [    5.026924] ELF: note: FEATURES = "!writable_page_tables"
    (XEN) [    5.029976] ELF: note: PAE_MODE = "yes"
    (XEN) [    5.032882] ELF: note: L1_MFN_VALID
    (XEN) [    5.035516] ELF: note: MOD_START_PFN = 0x1
    (XEN) [    5.038442] ELF: note: PADDR_OFFSET = 0
    (XEN) [    5.041250] ELF: note: SUPPORTED_FEATURES = 0x8801
    (XEN) [    5.044169] ELF: note: LOADER = "generic"
    (XEN) [    5.047048] ELF: note: SUSPEND_CANCEL = 0x1
    (XEN) [    5.049931] ELF: Found PVH image
    (XEN) [    5.052712] ELF: addresses:
    (XEN) [    5.055090]     virt_base        = 0x0
    (XEN) [    5.057552]     elf_paddr_offset = 0x0
    (XEN) [    5.060007]     virt_offset      = 0x0
    (XEN) [    5.062476]     virt_kstart      = 0x200000
    (XEN) [    5.064924]     virt_kend        = 0x3e00000
    (XEN) [    5.067380]     virt_entry       = 0x16a2ca0
    (XEN) [    5.069841]     p2m_base         = 0x8000000000
    (XEN) [    5.072319] ELF: phdr 0 at 0x200000 -> 0x21f3928
    (XEN) [    5.080076] ELF: phdr 1 at 0x2200000 -> 0x3e00000
    (XEN) [    5.090182] Dom0 memory map:
    (XEN) [    5.092531]  [0000000000000000, 000000000009efff] (usable)
    (XEN) [    5.095086]  [000000000009f000, 00000000000fffff] (reserved)
    (XEN) [    5.097625]  [0000000000100000, 000000005471afff] (usable)
    (XEN) [    5.100156]  [000000005471b000, 000000005475bfff] (reserved)
    (XEN) [    5.102704]  [000000005475c000, 0000000063c2dfff] (usable)
    (XEN) [    5.105259]  [0000000063c2e000, 000000006d17afff] (reserved)
    (XEN) [    5.107853]  [000000006d17b000, 000000006d22bfff] (ACPI data)
    (XEN) [    5.110459]  [000000006d22c000, 000000006d2ebfff] (ACPI NVS)
    (XEN) [    5.113082]  [000000006d2ec000, 000000006fffefff] (reserved)
    (XEN) [    5.115726]  [000000006ffff000, 000000006ffffdcb] (usable)
    (XEN) [    5.118388]  [000000006ffffdcc, 000000006ffffe97] (ACPI data)
    (XEN) [    5.121080]  [0000000070000000, 00000000807fffff] (reserved)
    (XEN) [    5.123776]  [00000000c0000000, 00000000cfffffff] (reserved)
    (XEN) [    5.126498]  [00000000fe000000, 00000000fe010fff] (reserved)
    (XEN) [    5.129244]  [00000000fec00000, 00000000fec00fff] (reserved)
    (XEN) [    5.132001]  [00000000fed00000, 00000000fed00fff] (reserved)
    (XEN) [    5.134797]  [00000000fed20000, 00000000fed7ffff] (reserved)
    (XEN) [    5.137619]  [00000000fee00000, 00000000fee00fff] (reserved)
    (XEN) [    5.140436]  [00000000ff000000, 00000001023fffff] (reserved)
    (XEN) [    5.143271]  [0000000102400000, 0000000468b34fff] (usable)
    (XEN) [    5.146131]  [0000000468b35000, 000000047f7fffff] (unusable)
    (XEN) [    5.149015] Initial low memory virq threshold set at 0x4000 pages.
    (XEN) [    5.151852] Scrubbing Free RAM in background
    (XEN) [    5.154667] Std. Loglevel: All
    (XEN) [    5.157465] Guest Loglevel: All
    (XEN) [    5.160250] Xen is relinquishing VGA console.
    (XEN) [    5.166762] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
    (XEN) [    5.167550] Re-running stub recovery selftests...
    (XEN) [    5.167692] Fixup #UD[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403b8753
    (XEN) [    5.167914] Fixup #GP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403b8753
    (XEN) [    5.168139] Fixup #SS[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403b8753
    (XEN) [    5.168359] Fixup #BP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403b8753
    (XEN) [    5.168662] Freed 720kB init memory
    (XEN) [    6.884758] d0v0: upcall vector f3

Interestingly, this appear to have worked on Xen 4.19.3.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Cannot boot PVH dom0 with big initrd

[Jan Beulich]

On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> After fixing the xhci crash, I hit another issue - booting with 236MB
> initrd doesn't work, I get:
> 
>     (XEN) [    3.151856] *** Building a PVH Dom0 ***
>     ...
>     (XEN) [    3.593940] Unable to allocate memory with order 0!
>     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
>     (XEN) [    3.599884] 
>     (XEN) [    3.602482] ****************************************
>     (XEN) [    3.605272] Panic on CPU 0:
>     (XEN) [    3.607928] Could not construct d0
>     (XEN) [    3.610692] ****************************************
>     (XEN) [    3.613463] 
>     (XEN) [    3.616035] Reboot in five seconds...
>     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> 
> Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> 
> If I skip initrd, then it boots just fine (but dom0 is not happy about
> that). 164MB initrd failed too, but 13MB started ok.
> Just in case, I tried skipping XHCI console, but it didn't change
> anything.
> 
> Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> started from GRUB, using MB2+EFI.

Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
for passing from the boot loader to Xen) only after copying to the designated
guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
its calculation (adding initial_images_nrpages()'s return value). PV Dom0
isn't affected because to load huge initrd there, the kernel has to request
the initrd to not be mapped into the initial allocation.

Jan

Re: Cannot boot PVH dom0 with big initrd

[Roger Pau Monné]

On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > After fixing the xhci crash, I hit another issue - booting with 236MB
> > initrd doesn't work, I get:
> > 
> >     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> >     ...
> >     (XEN) [    3.593940] Unable to allocate memory with order 0!
> >     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> >     (XEN) [    3.599884] 
> >     (XEN) [    3.602482] ****************************************
> >     (XEN) [    3.605272] Panic on CPU 0:
> >     (XEN) [    3.607928] Could not construct d0
> >     (XEN) [    3.610692] ****************************************
> >     (XEN) [    3.613463] 
> >     (XEN) [    3.616035] Reboot in five seconds...
> >     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> > 
> > Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> > 
> > If I skip initrd, then it boots just fine (but dom0 is not happy about
> > that). 164MB initrd failed too, but 13MB started ok.
> > Just in case, I tried skipping XHCI console, but it didn't change
> > anything.
> > 
> > Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> > started from GRUB, using MB2+EFI.
> 
> Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> for passing from the boot loader to Xen) only after copying to the designated
> guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> isn't affected because to load huge initrd there, the kernel has to request
> the initrd to not be mapped into the initial allocation.

Right, on PV dom0 we do not copy the image to a new set of pages, we
simply assign the pages where the initrd resides to the domain.  We
can't populate those pages in the p2m as-is, otherwise we would
shatter super pages.

I think the fix below should do it, it's likely the best we can do.
Can you please give it a try Marek?

Thanks, Roger.
---
diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 0b467fd4a4fc..8e3cb5d0db76 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
 
     for_each_node_mask ( node, dom0_nodes )
         avail += avail_domheap_pages_region(node, 0, 0) +
-                 initial_images_nrpages(node);
+                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
 
     /* Reserve memory for further dom0 vcpu-struct allocations... */
     avail -= (d->max_vcpus - 1UL)

Re: Cannot boot PVH dom0 with big initrd

[Roger Pau Monné]

On Fri, Feb 13, 2026 at 04:56:39PM +0100, Roger Pau Monné wrote:
> On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> > On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > > 
> > > After fixing the xhci crash, I hit another issue - booting with 236MB
> > > initrd doesn't work, I get:
> > > 
> > >     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> > >     ...
> > >     (XEN) [    3.593940] Unable to allocate memory with order 0!
> > >     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> > >     (XEN) [    3.599884] 
> > >     (XEN) [    3.602482] ****************************************
> > >     (XEN) [    3.605272] Panic on CPU 0:
> > >     (XEN) [    3.607928] Could not construct d0
> > >     (XEN) [    3.610692] ****************************************
> > >     (XEN) [    3.613463] 
> > >     (XEN) [    3.616035] Reboot in five seconds...
> > >     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> > > 
> > > Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> > > 
> > > If I skip initrd, then it boots just fine (but dom0 is not happy about
> > > that). 164MB initrd failed too, but 13MB started ok.
> > > Just in case, I tried skipping XHCI console, but it didn't change
> > > anything.
> > > 
> > > Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> > > started from GRUB, using MB2+EFI.
> > 
> > Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> > for passing from the boot loader to Xen) only after copying to the designated
> > guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> > its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> > isn't affected because to load huge initrd there, the kernel has to request
> > the initrd to not be mapped into the initial allocation.
> 
> Right, on PV dom0 we do not copy the image to a new set of pages, we
> simply assign the pages where the initrd resides to the domain.  We
> can't populate those pages in the p2m as-is, otherwise we would
> shatter super pages.
> 
> I think the fix below should do it, it's likely the best we can do.
> Can you please give it a try Marek?
> 
> Thanks, Roger.
> ---
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index 0b467fd4a4fc..8e3cb5d0db76 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
>  
>      for_each_node_mask ( node, dom0_nodes )
>          avail += avail_domheap_pages_region(node, 0, 0) +
> -                 initial_images_nrpages(node);
> +                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
>  
>      /* Reserve memory for further dom0 vcpu-struct allocations... */
>      avail -= (d->max_vcpus - 1UL)

I'm working on a more complex patch, that attempts to account the
memory used by the init images towards the reserved amount that's kept
by Xen.  This should make accounting a bit better, in that we won't
end up reserving the Xen memory plus the memory used by the init
images.

It's still however a WIP, but would you mind giving it a try?

Thanks, Roger.
---
diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 0b467fd4a4fc..3d54af197188 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -325,10 +325,18 @@ unsigned long __init dom0_paging_pages(const struct domain *d,
  * If allocation isn't specified, reserve 1/16th of available memory for
  * things like DMA buffers. This reservation is clamped to a maximum of 128MB.
  */
-static unsigned long __init default_nr_pages(unsigned long avail)
+static unsigned long __init default_nr_pages(unsigned long avail,
+                                             unsigned long init_images)
 {
-    return avail - (pv_shim ? pv_shim_mem(avail)
-                            : min(avail / 16, 128UL << (20 - PAGE_SHIFT)));
+    unsigned long rsvd = min(avail / 16, 128UL << (20 - PAGE_SHIFT));
+
+    /*
+     * Account for memory consumed by initial images as if it was part of the
+     * reserved amount.
+     */
+    rsvd -= rsvd <= init_images ? rsvd : init_images;
+
+    return avail - (pv_shim ? pv_shim_mem(avail) : rsvd);
 }
 
 unsigned long __init dom0_compute_nr_pages(
@@ -336,14 +344,28 @@ unsigned long __init dom0_compute_nr_pages(
 {
     nodeid_t node;
     unsigned long avail = 0, nr_pages, min_pages, max_pages, iommu_pages = 0;
+    unsigned long init_images = 0;
 
     /* The ordering of operands is to work around a clang5 issue. */
     if ( CONFIG_DOM0_MEM[0] && !dom0_mem_set )
         parse_dom0_mem(CONFIG_DOM0_MEM);
 
     for_each_node_mask ( node, dom0_nodes )
-        avail += avail_domheap_pages_region(node, 0, 0) +
-                 initial_images_nrpages(node);
+    {
+        avail += avail_domheap_pages_region(node, 0, 0);
+        init_images += initial_images_nrpages(node);
+    }
+
+    if ( is_pv_domain(d) )
+    {
+        /*
+         * For PV domains the initrd pages are directly assigned to the
+         * guest, and hence the initrd size counts as free memory that can
+         * be used by the domain.  Set to 0 to prevent further adjustments.
+         */
+        avail += init_images;
+        init_images = 0;
+    }
 
     /* Reserve memory for further dom0 vcpu-struct allocations... */
     avail -= (d->max_vcpus - 1UL)
@@ -367,7 +389,8 @@ unsigned long __init dom0_compute_nr_pages(
     {
         unsigned long cpu_pages;
 
-        nr_pages = get_memsize(&dom0_size, avail) ?: default_nr_pages(avail);
+        nr_pages = get_memsize(&dom0_size, avail) ?:
+                   default_nr_pages(avail, init_images);
 
         /*
          * Clamp according to min/max limits and available memory
@@ -385,7 +408,8 @@ unsigned long __init dom0_compute_nr_pages(
             avail -= cpu_pages - iommu_pages;
     }
 
-    nr_pages = get_memsize(&dom0_size, avail) ?: default_nr_pages(avail);
+    nr_pages = get_memsize(&dom0_size, avail) ?:
+               default_nr_pages(avail, init_images);
     min_pages = get_memsize(&dom0_min_size, avail);
     max_pages = get_memsize(&dom0_max_size, avail);
 

Re: Cannot boot PVH dom0 with big initrd

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 6310 bytes --]

On Fri, Feb 13, 2026 at 04:56:39PM +0100, Roger Pau Monné wrote:
> On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> > On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > > 
> > > After fixing the xhci crash, I hit another issue - booting with 236MB
> > > initrd doesn't work, I get:
> > > 
> > >     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> > >     ...
> > >     (XEN) [    3.593940] Unable to allocate memory with order 0!
> > >     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> > >     (XEN) [    3.599884] 
> > >     (XEN) [    3.602482] ****************************************
> > >     (XEN) [    3.605272] Panic on CPU 0:
> > >     (XEN) [    3.607928] Could not construct d0
> > >     (XEN) [    3.610692] ****************************************
> > >     (XEN) [    3.613463] 
> > >     (XEN) [    3.616035] Reboot in five seconds...
> > >     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> > > 
> > > Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> > > 
> > > If I skip initrd, then it boots just fine (but dom0 is not happy about
> > > that). 164MB initrd failed too, but 13MB started ok.
> > > Just in case, I tried skipping XHCI console, but it didn't change
> > > anything.
> > > 
> > > Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> > > started from GRUB, using MB2+EFI.
> > 
> > Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> > for passing from the boot loader to Xen) only after copying to the designated
> > guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> > its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> > isn't affected because to load huge initrd there, the kernel has to request
> > the initrd to not be mapped into the initial allocation.
> 
> Right, on PV dom0 we do not copy the image to a new set of pages, we
> simply assign the pages where the initrd resides to the domain.  We
> can't populate those pages in the p2m as-is, otherwise we would
> shatter super pages.
> 
> I think the fix below should do it, it's likely the best we can do.
> Can you please give it a try Marek?

With this, it's different:

    (XEN) [    4.510345] Dom0 memory allocation stats:
    (XEN) [    4.513110] order  0 allocations: 1
    (XEN) [    4.515847] order  1 allocations: 1
    (XEN) [    4.518593] order  2 allocations: 2
    (XEN) [    4.521329] order  3 allocations: 2
    (XEN) [    4.524076] order  4 allocations: 1
    (XEN) [    4.526799] order  5 allocations: 1
    (XEN) [    4.529518] order  6 allocations: 1
    (XEN) [    4.532118] order  7 allocations: 2
    (XEN) [    4.534807] order  8 allocations: 2
    (XEN) [    4.537483] order  9 allocations: 1
    (XEN) [    4.540052] order 10 allocations: 2
    (XEN) [    4.542618] order 11 allocations: 2
    (XEN) [    4.545276] order 12 allocations: 1
    (XEN) [    4.547949] order 13 allocations: 1
    (XEN) [    4.550501] order 14 allocations: 1
    (XEN) [    4.553147] order 15 allocations: 1
    (XEN) [    5.393487] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
    (XEN) [    5.396196] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
    (XEN) [    5.398891] ELF: memory: 0x200000 -> 0x3e00000
    (XEN) [    5.401592] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
    (XEN) [    5.404632] ELF: note: PHYS32_ENTRY = 0x16a2ca0
    (XEN) [    5.407695] ELF: note: GUEST_OS = "linux"
    (XEN) [    5.410748] ELF: note: GUEST_VERSION = "2.6"
    (XEN) [    5.413810] ELF: note: XEN_VERSION = "xen-3.0"
    (XEN) [    5.416891] ELF: note: VIRT_BASE = 0xffffffff80000000
    (XEN) [    5.419976] ELF: note: INIT_P2M = 0x8000000000
    (XEN) [    5.423062] ELF: note: ENTRY = 0xffffffff82d3c160
    (XEN) [    5.426155] ELF: note: FEATURES = "!writable_page_tables"
    (XEN) [    5.429260] ELF: note: PAE_MODE = "yes"
    (XEN) [    5.432343] ELF: note: L1_MFN_VALID
    (XEN) [    5.434952] ELF: note: MOD_START_PFN = 0x1
    (XEN) [    5.437975] ELF: note: PADDR_OFFSET = 0
    (XEN) [    5.440928] ELF: note: SUPPORTED_FEATURES = 0x8801
    (XEN) [    5.443871] ELF: note: LOADER = "generic"
    (XEN) [    5.446785] ELF: note: SUSPEND_CANCEL = 0x1
    (XEN) [    5.449696] ELF: Found PVH image
    (XEN) [    5.452574] ELF: addresses:
    (XEN) [    5.455002]     virt_base        = 0x0
    (XEN) [    5.457569]     elf_paddr_offset = 0x0
    (XEN) [    5.460102]     virt_offset      = 0x0
    (XEN) [    5.462569]     virt_kstart      = 0x200000
    (XEN) [    5.465049]     virt_kend        = 0x3e00000
    (XEN) [    5.467515]     virt_entry       = 0x16a2ca0
    (XEN) [    5.469983]     p2m_base         = 0x8000000000
    (XEN) [    5.472465] ELF: phdr 0 at 0x200000 -> 0x21f3928
    (XEN) [    5.480548] ELF: phdr 1 at 0x2200000 -> 0x3e00000
    (XEN) [    5.487915] Unable to find a memory region to load initrd and metadata
    (XEN) [    5.490466] Failed to load Dom0 kernel
    (XEN) [    5.492954] 
    (XEN) [    5.495266] ****************************************
    (XEN) [    5.497755] Panic on CPU 0:
    (XEN) [    5.500065] Could not construct d0
    (XEN) [    5.502470] ****************************************
    (XEN) [    5.504897] 
    (XEN) [    5.507134] Reboot in five seconds...
    (XEN) [   10.517191] Resetting with ACPI MEMORY or I/O RESET_REG.

FWIW memory map is in the full console dump linked earlier.

I'll test the other patch next.

> ---
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index 0b467fd4a4fc..8e3cb5d0db76 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
>  
>      for_each_node_mask ( node, dom0_nodes )
>          avail += avail_domheap_pages_region(node, 0, 0) +
> -                 initial_images_nrpages(node);
> +                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
>  
>      /* Reserve memory for further dom0 vcpu-struct allocations... */
>      avail -= (d->max_vcpus - 1UL)
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Cannot boot PVH dom0 with big initrd

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 8135 bytes --]

On Fri, Feb 13, 2026 at 09:40:31PM +0100, Roger Pau Monné wrote:
> On Fri, Feb 13, 2026 at 04:56:39PM +0100, Roger Pau Monné wrote:
> > On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> > > On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > > 
> > > > After fixing the xhci crash, I hit another issue - booting with 236MB
> > > > initrd doesn't work, I get:
> > > > 
> > > >     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> > > >     ...
> > > >     (XEN) [    3.593940] Unable to allocate memory with order 0!
> > > >     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> > > >     (XEN) [    3.599884] 
> > > >     (XEN) [    3.602482] ****************************************
> > > >     (XEN) [    3.605272] Panic on CPU 0:
> > > >     (XEN) [    3.607928] Could not construct d0
> > > >     (XEN) [    3.610692] ****************************************
> > > >     (XEN) [    3.613463] 
> > > >     (XEN) [    3.616035] Reboot in five seconds...
> > > >     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> > > > 
> > > > Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> > > > 
> > > > If I skip initrd, then it boots just fine (but dom0 is not happy about
> > > > that). 164MB initrd failed too, but 13MB started ok.
> > > > Just in case, I tried skipping XHCI console, but it didn't change
> > > > anything.
> > > > 
> > > > Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> > > > started from GRUB, using MB2+EFI.
> > > 
> > > Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> > > for passing from the boot loader to Xen) only after copying to the designated
> > > guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> > > its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> > > isn't affected because to load huge initrd there, the kernel has to request
> > > the initrd to not be mapped into the initial allocation.
> > 
> > Right, on PV dom0 we do not copy the image to a new set of pages, we
> > simply assign the pages where the initrd resides to the domain.  We
> > can't populate those pages in the p2m as-is, otherwise we would
> > shatter super pages.
> > 
> > I think the fix below should do it, it's likely the best we can do.
> > Can you please give it a try Marek?
> > 
> > Thanks, Roger.
> > ---
> > diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> > index 0b467fd4a4fc..8e3cb5d0db76 100644
> > --- a/xen/arch/x86/dom0_build.c
> > +++ b/xen/arch/x86/dom0_build.c
> > @@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
> >  
> >      for_each_node_mask ( node, dom0_nodes )
> >          avail += avail_domheap_pages_region(node, 0, 0) +
> > -                 initial_images_nrpages(node);
> > +                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
> >  
> >      /* Reserve memory for further dom0 vcpu-struct allocations... */
> >      avail -= (d->max_vcpus - 1UL)
> 
> I'm working on a more complex patch, that attempts to account the
> memory used by the init images towards the reserved amount that's kept
> by Xen.  This should make accounting a bit better, in that we won't
> end up reserving the Xen memory plus the memory used by the init
> images.
> 
> It's still however a WIP, but would you mind giving it a try?

This one worked :)

    (XEN) [    4.014243] Dom0 memory allocation stats:
    (XEN) [    4.017378] order  0 allocations: 4
    (XEN) [    4.020111] order  1 allocations: 3
    (XEN) [    4.022758] order  2 allocations: 4
    (XEN) [    4.025406] order  3 allocations: 4
    (XEN) [    4.028150] order  4 allocations: 3
    (XEN) [    4.030886] order  5 allocations: 2
    (XEN) [    4.033602] order  6 allocations: 2
    (XEN) [    4.036305] order  7 allocations: 6
    (XEN) [    4.039020] order  8 allocations: 6
    (XEN) [    4.041597] order  9 allocations: 5
    (XEN) [    4.044261] order 10 allocations: 9
    (XEN) [    4.046831] order 11 allocations: 7
    (XEN) [    4.049493] order 12 allocations: 9
    (XEN) [    4.052147] order 13 allocations: 7
    (XEN) [    4.054799] order 14 allocations: 6
    (XEN) [    4.057447] order 15 allocations: 7
    (XEN) [    4.060080] order 16 allocations: 7
    (XEN) [    4.062609] order 17 allocations: 5
    (XEN) [    4.065227] order 18 allocations: 9
    (XEN) [    4.921719] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
    (XEN) [    4.924403] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
    (XEN) [    4.927079] ELF: memory: 0x200000 -> 0x3e00000
    (XEN) [    4.929759] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
    (XEN) [    4.932884] ELF: note: PHYS32_ENTRY = 0x16a2ca0
    (XEN) [    4.935921] ELF: note: GUEST_OS = "linux"
    (XEN) [    4.938953] ELF: note: GUEST_VERSION = "2.6"
    (XEN) [    4.942005] ELF: note: XEN_VERSION = "xen-3.0"
    (XEN) [    4.945077] ELF: note: VIRT_BASE = 0xffffffff80000000
    (XEN) [    4.948133] ELF: note: INIT_P2M = 0x8000000000
    (XEN) [    4.951203] ELF: note: ENTRY = 0xffffffff82d3c160
    (XEN) [    4.954221] ELF: note: FEATURES = "!writable_page_tables"
    (XEN) [    4.957229] ELF: note: PAE_MODE = "yes"
    (XEN) [    4.960175] ELF: note: L1_MFN_VALID
    (XEN) [    4.962775] ELF: note: MOD_START_PFN = 0x1
    (XEN) [    4.965675] ELF: note: PADDR_OFFSET = 0
    (XEN) [    4.968540] ELF: note: SUPPORTED_FEATURES = 0x8801
    (XEN) [    4.971420] ELF: note: LOADER = "generic"
    (XEN) [    4.974303] ELF: note: SUSPEND_CANCEL = 0x1
    (XEN) [    4.977186] ELF: Found PVH image
    (XEN) [    4.979910] ELF: addresses:
    (XEN) [    4.982237]     virt_base        = 0x0
    (XEN) [    4.984694]     elf_paddr_offset = 0x0
    (XEN) [    4.987141]     virt_offset      = 0x0
    (XEN) [    4.989599]     virt_kstart      = 0x200000
    (XEN) [    4.992044]     virt_kend        = 0x3e00000
    (XEN) [    4.994498]     virt_entry       = 0x16a2ca0
    (XEN) [    4.996966]     p2m_base         = 0x8000000000
    (XEN) [    4.999415] ELF: phdr 0 at 0x200000 -> 0x21f3928
    (XEN) [    5.007160] ELF: phdr 1 at 0x2200000 -> 0x3e00000
    (XEN) [    5.055448] Dom0 memory map:
    (XEN) [    5.057763]  [0000000000000000, 000000000009efff] (usable)
    (XEN) [    5.060281]  [000000000009f000, 00000000000fffff] (reserved)
    (XEN) [    5.062812]  [0000000000100000, 000000005471afff] (usable)
    (XEN) [    5.065324]  [000000005471b000, 000000005475bfff] (reserved)
    (XEN) [    5.067891]  [000000005475c000, 0000000063c2dfff] (usable)
    (XEN) [    5.070446]  [0000000063c2e000, 000000006d17afff] (reserved)
    (XEN) [    5.073036]  [000000006d17b000, 000000006d22bfff] (ACPI data)
    (XEN) [    5.075649]  [000000006d22c000, 000000006d2ebfff] (ACPI NVS)
    (XEN) [    5.078267]  [000000006d2ec000, 000000006fffefff] (reserved)
    (XEN) [    5.080908]  [000000006ffff000, 000000006ffffdcb] (usable)
    (XEN) [    5.083560]  [000000006ffffdcc, 000000006ffffe97] (ACPI data)
    (XEN) [    5.086243]  [0000000070000000, 00000000807fffff] (reserved)
    (XEN) [    5.088943]  [00000000c0000000, 00000000cfffffff] (reserved)
    (XEN) [    5.091662]  [00000000fe000000, 00000000fe010fff] (reserved)
    (XEN) [    5.094413]  [00000000fec00000, 00000000fec00fff] (reserved)
    (XEN) [    5.097193]  [00000000fed00000, 00000000fed00fff] (reserved)
    (XEN) [    5.099976]  [00000000fed20000, 00000000fed7ffff] (reserved)
    (XEN) [    5.102779]  [00000000fee00000, 00000000fee00fff] (reserved)
    (XEN) [    5.105615]  [00000000ff000000, 00000001023fffff] (reserved)
    (XEN) [    5.108487]  [0000000102400000, 000000045c89cfff] (usable)
    (XEN) [    5.111354]  [000000045c89d000, 000000047f7fffff] (unusable)
    (XEN) [    5.114263] Initial low memory virq threshold set at 0x4000 pages.
    (XEN) [    5.117108] Scrubbing Free RAM in background

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Cannot boot PVH dom0 with big initrd

[Jan Beulich]

On 13.02.2026 16:56, Roger Pau Monné wrote:
> On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
>> On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
>>> Hi,
>>>
>>> After fixing the xhci crash, I hit another issue - booting with 236MB
>>> initrd doesn't work, I get:
>>>
>>>     (XEN) [    3.151856] *** Building a PVH Dom0 ***
>>>     ...
>>>     (XEN) [    3.593940] Unable to allocate memory with order 0!
>>>     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
>>>     (XEN) [    3.599884] 
>>>     (XEN) [    3.602482] ****************************************
>>>     (XEN) [    3.605272] Panic on CPU 0:
>>>     (XEN) [    3.607928] Could not construct d0
>>>     (XEN) [    3.610692] ****************************************
>>>     (XEN) [    3.613463] 
>>>     (XEN) [    3.616035] Reboot in five seconds...
>>>     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
>>>
>>> Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
>>>
>>> If I skip initrd, then it boots just fine (but dom0 is not happy about
>>> that). 164MB initrd failed too, but 13MB started ok.
>>> Just in case, I tried skipping XHCI console, but it didn't change
>>> anything.
>>>
>>> Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
>>> started from GRUB, using MB2+EFI.
>>
>> Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
>> for passing from the boot loader to Xen) only after copying to the designated
>> guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
>> its calculation (adding initial_images_nrpages()'s return value). PV Dom0
>> isn't affected because to load huge initrd there, the kernel has to request
>> the initrd to not be mapped into the initial allocation.
> 
> Right, on PV dom0 we do not copy the image to a new set of pages, we
> simply assign the pages where the initrd resides to the domain.  We
> can't populate those pages in the p2m as-is, otherwise we would
> shatter super pages.
> 
> I think the fix below should do it, it's likely the best we can do.

That's at best a workaround imo. We definitely can do better, and the bigger
the initrd, the more important it may be that we actually do. One option may
be to similarly use the pages directly (i.e. by assigning rather than
copying), accepting that we may not be able to use large page mappings then
for the respective GFN range.

Jan

> Can you please give it a try Marek?
> 
> Thanks, Roger.
> ---
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index 0b467fd4a4fc..8e3cb5d0db76 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(
>  
>      for_each_node_mask ( node, dom0_nodes )
>          avail += avail_domheap_pages_region(node, 0, 0) +
> -                 initial_images_nrpages(node);
> +                 is_pv_domain(d) ? initial_images_nrpages(node) : 0;
>  
>      /* Reserve memory for further dom0 vcpu-struct allocations... */
>      avail -= (d->max_vcpus - 1UL)
> 

Re: Cannot boot PVH dom0 with big initrd

[Roger Pau Monné]

On Mon, Feb 16, 2026 at 09:11:29AM +0100, Jan Beulich wrote:
> On 13.02.2026 16:56, Roger Pau Monné wrote:
> > On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
> >> On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
> >>> Hi,
> >>>
> >>> After fixing the xhci crash, I hit another issue - booting with 236MB
> >>> initrd doesn't work, I get:
> >>>
> >>>     (XEN) [    3.151856] *** Building a PVH Dom0 ***
> >>>     ...
> >>>     (XEN) [    3.593940] Unable to allocate memory with order 0!
> >>>     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
> >>>     (XEN) [    3.599884] 
> >>>     (XEN) [    3.602482] ****************************************
> >>>     (XEN) [    3.605272] Panic on CPU 0:
> >>>     (XEN) [    3.607928] Could not construct d0
> >>>     (XEN) [    3.610692] ****************************************
> >>>     (XEN) [    3.613463] 
> >>>     (XEN) [    3.616035] Reboot in five seconds...
> >>>     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
> >>>
> >>> Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
> >>>
> >>> If I skip initrd, then it boots just fine (but dom0 is not happy about
> >>> that). 164MB initrd failed too, but 13MB started ok.
> >>> Just in case, I tried skipping XHCI console, but it didn't change
> >>> anything.
> >>>
> >>> Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
> >>> started from GRUB, using MB2+EFI.
> >>
> >> Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
> >> for passing from the boot loader to Xen) only after copying to the designated
> >> guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
> >> its calculation (adding initial_images_nrpages()'s return value). PV Dom0
> >> isn't affected because to load huge initrd there, the kernel has to request
> >> the initrd to not be mapped into the initial allocation.
> > 
> > Right, on PV dom0 we do not copy the image to a new set of pages, we
> > simply assign the pages where the initrd resides to the domain.  We
> > can't populate those pages in the p2m as-is, otherwise we would
> > shatter super pages.
> > 
> > I think the fix below should do it, it's likely the best we can do.
> 
> That's at best a workaround imo. We definitely can do better, and the bigger
> the initrd, the more important it may be that we actually do.

See the second patch I gave to Marek.  That one is slightly better by
accounting for the initial images space as part of the reserved space.

> One option may
> be to similarly use the pages directly (i.e. by assigning rather than
> copying), accepting that we may not be able to use large page mappings then
> for the respective GFN range.

Hm, there's always going to be a trade-off.  I think I would prefer
having 1G pages in the p2m, rather than a bit more memory due to
direct adding the initrd into the p2m.

Thanks, Roger.

Re: Cannot boot PVH dom0 with big initrd

[Jan Beulich]

On 16.02.2026 09:40, Roger Pau Monné wrote:
> On Mon, Feb 16, 2026 at 09:11:29AM +0100, Jan Beulich wrote:
>> On 13.02.2026 16:56, Roger Pau Monné wrote:
>>> On Fri, Feb 13, 2026 at 09:56:42AM +0100, Jan Beulich wrote:
>>>> On 13.02.2026 05:02, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> After fixing the xhci crash, I hit another issue - booting with 236MB
>>>>> initrd doesn't work, I get:
>>>>>
>>>>>     (XEN) [    3.151856] *** Building a PVH Dom0 ***
>>>>>     ...
>>>>>     (XEN) [    3.593940] Unable to allocate memory with order 0!
>>>>>     (XEN) [    3.597110] Failed to setup Dom0 physical memory map
>>>>>     (XEN) [    3.599884] 
>>>>>     (XEN) [    3.602482] ****************************************
>>>>>     (XEN) [    3.605272] Panic on CPU 0:
>>>>>     (XEN) [    3.607928] Could not construct d0
>>>>>     (XEN) [    3.610692] ****************************************
>>>>>     (XEN) [    3.613463] 
>>>>>     (XEN) [    3.616035] Reboot in five seconds...
>>>>>     (XEN) [    8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
>>>>>
>>>>> Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
>>>>>
>>>>> If I skip initrd, then it boots just fine (but dom0 is not happy about
>>>>> that). 164MB initrd failed too, but 13MB started ok.
>>>>> Just in case, I tried skipping XHCI console, but it didn't change
>>>>> anything.
>>>>>
>>>>> Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is
>>>>> started from GRUB, using MB2+EFI.
>>>>
>>>> Hmm, yes, there's an ordering issue: Of course we free initrd space (as used
>>>> for passing from the boot loader to Xen) only after copying to the designated
>>>> guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in
>>>> its calculation (adding initial_images_nrpages()'s return value). PV Dom0
>>>> isn't affected because to load huge initrd there, the kernel has to request
>>>> the initrd to not be mapped into the initial allocation.
>>>
>>> Right, on PV dom0 we do not copy the image to a new set of pages, we
>>> simply assign the pages where the initrd resides to the domain.  We
>>> can't populate those pages in the p2m as-is, otherwise we would
>>> shatter super pages.
>>>
>>> I think the fix below should do it, it's likely the best we can do.
>>
>> That's at best a workaround imo. We definitely can do better, and the bigger
>> the initrd, the more important it may be that we actually do.
> 
> See the second patch I gave to Marek.  That one is slightly better by
> accounting for the initial images space as part of the reserved space.

Will check; didn't make it there, yet.

>> One option may
>> be to similarly use the pages directly (i.e. by assigning rather than
>> copying), accepting that we may not be able to use large page mappings then
>> for the respective GFN range.
> 
> Hm, there's always going to be a trade-off.  I think I would prefer
> having 1G pages in the p2m, rather than a bit more memory due to
> direct adding the initrd into the p2m.

"A bit more" may not get it, when considering e.g. a 2Gb initrd on a 4Gb
system. And of course "use pages directly" is only the simplest of the
possible approaches. We could "shift" the initrd some, or we could make
sure to allocate a 2Mb or 1Gb aligned region to hold it right from the
beginning.

Jan

Re: Cannot boot PVH dom0 with big initrd

[Jan Beulich]

On 13.02.2026 21:40, Roger Pau Monné wrote:
> @@ -336,14 +344,28 @@ unsigned long __init dom0_compute_nr_pages(
>  {
>      nodeid_t node;
>      unsigned long avail = 0, nr_pages, min_pages, max_pages, iommu_pages = 0;
> +    unsigned long init_images = 0;
>  
>      /* The ordering of operands is to work around a clang5 issue. */
>      if ( CONFIG_DOM0_MEM[0] && !dom0_mem_set )
>          parse_dom0_mem(CONFIG_DOM0_MEM);
>  
>      for_each_node_mask ( node, dom0_nodes )
> -        avail += avail_domheap_pages_region(node, 0, 0) +
> -                 initial_images_nrpages(node);
> +    {
> +        avail += avail_domheap_pages_region(node, 0, 0);
> +        init_images += initial_images_nrpages(node);
> +    }
> +
> +    if ( is_pv_domain(d) )
> +    {
> +        /*
> +         * For PV domains the initrd pages are directly assigned to the
> +         * guest, and hence the initrd size counts as free memory that can
> +         * be used by the domain.  Set to 0 to prevent further adjustments.
> +         */
> +        avail += init_images;
> +        init_images = 0;
> +    }

Just to mention: It's still only "may be directly assigned", because of the
PV32 special requirements.

Jan