Excluding init_on_free for pages for initial balloon down (Xen)

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 3650 bytes --]

Hi,

Some time ago I made a change to disable scrubbing pages that are
ballooned out during system boot. I'll paste the whole commit message as
it's relevant here:

    197ecb3802c0 xen/balloon: add runtime control for scrubbing ballooned out pages

    Scrubbing pages on initial balloon down can take some time, especially
    in nested virtualization case (nested EPT is slow). When HVM/PVH guest is
    started with memory= significantly lower than maxmem=, all the extra
    pages will be scrubbed before returning to Xen. But since most of them
    weren't used at all at that point, Xen needs to populate them first
    (from populate-on-demand pool). In nested virt case (Xen inside KVM)
    this slows down the guest boot by 15-30s with just 1.5GB needed to be
    returned to Xen.
    
    Add runtime parameter to enable/disable it, to allow initially disabling
    scrubbing, then enable it back during boot (for example in initramfs).
    Such usage relies on assumption that a) most pages ballooned out during
    initial boot weren't used at all, and b) even if they were, very few
    secrets are in the guest at that time (before any serious userspace
    kicks in).
    Convert CONFIG_XEN_SCRUB_PAGES to CONFIG_XEN_SCRUB_PAGES_DEFAULT (also
    enabled by default), controlling default value for the new runtime
    switch.

Now, I face the same issue with init_on_free/init_on_alloc (not sure
which one applies here, probably the latter one), which several
distributions enable by default. The result is (see timestamps):

    [2026-02-24 01:12:55] [    7.485151] xen:balloon: Waiting for initial ballooning down having finished.
    [2026-02-24 01:14:14] [   86.581510] xen:balloon: Initial ballooning down finished.

But here the situation is a bit more complicated:
init_on_free/init_on_alloc applies to any pages, not just those for
balloon driver. I see two approaches to solve the issue:
1. Similar to xen_scrub_pages=, add a runtime switch for
   init_on_free/init_on_alloc, then force them off during boot, and
   re-enable early in initramfs.
2. Somehow adjust balloon driver to bypass init_on_alloc when ballooning
   a page out.

The first approach is likely easier to implement, but also has some
drawbacks: it may result in some kernel structures that are allocated
early to remain with garbage data in uninitialized places. While it may
not matter during early boot, such structures may survive for quite some
time, and maybe attacker can use them later on to exploit some other
bug. This wasn't really a concern with xen_scrub_pages, as those pages
were immediately ballooned out.

The second approach sounds architecturally better, and maybe
init_on_alloc could be always bypassed during balloon out? The balloon
driver can scrub the page on its own already (which is enabled by
default). That of course assumes the issue is only about init_on_alloc,
not init_on_free (or both) - which I haven't really confirmed yet...
If going this way, I see the balloon driver does basically
alloc_page(GFP_BALLOON), where GFP_BALLOON is:

    /* When ballooning out (allocating memory to return to Xen) we don't really
       want the kernel to try too hard since that can trigger the oom killer. */
    #define GFP_BALLOON \
        (GFP_HIGHUSER | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC)

Would that be about adding some new flag here? Or maybe there is already
one for this purpose?

Any opinions?

PS issue tracked at https://github.com/QubesOS/qubes-issues/issues/10723

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]