Re: [PATCH v1 0/8] kho: add read-only Kexec Handover mode

[Pasha Tatashin <pasha.tatashin@soleen.com>]

On 10-19 11:43, Mike Rapoport wrote:
> Hi Pasha,
> 
> Some brief comments before digging in.
> 
> On Sun, Oct 18, 2026 at 10:11:03PM -0400, Pasha Tatashin wrote:
> > This series introduces a read-only Kexec Handover mode, enabled at boot
> > via 'kho_ro=<pa>' pointing to the physical address of the KHO FDT.
> > 
> > In standard KHO, the target kernel takes ownership of preserved memory
> > during early boot, initializes struct pages for preserved folios, and
> > reclaims KHO metadata and preserved pages back into the page allocator
> > as they are restored.
> > 
> > In some scenarios (such as a caretaker, inspection kernel, firmware), a
> > kernel needs to inspect and consume preserved KHO subtrees, radix trees,
> > and serialized metadata without taking ownership of or modifying the KHO
> > state, leaving the preserved KHO tree intact for a subsequent kernel.
> > In this mode, the KHO physical memory lies outside the kernel's direct
> > map.
> > 
> > To support this cleanly without duplicating KHO traversal logic:
> > - Patch 1 exposes the physical address of the outgoing KHO FDT via
> >   /sys/kernel/debug/kho/out/fdt_phys so userspace tooling can pass it
> >   to a 'kho_ro=<pa>' kernel.
> 
> We already pass the physical address, why adding a boolean kho_ro is not
> enough?

In standard KHO, we pass the physical address as part of the normal 
kexec_file_load() reboot path. In that flow, the outgoing kernel shuts 
down or serializes all devices into LUO state, releases machine control, 
and hands over all machine resources to the next kernel, which becomes the 
sole "Machine Owner" (MO) and owns all machine resources.

Here, we are using a model where a minimal "Linux Caretaker" (LC) kernel 
runs concurrently while the MO is still active. We spin off LC on 
dedicated CPUs that have been offlined from the MO scheduler, giving LC 
read-only access to KHO so it can temporarily take care of VM while MO 
undergoes a standard KHO-preserved kexec reboot (i.e. Live Update). Once 
the new MO kernel boots up, it takes back the resources from LC and 
retrieves everything normally from LUO/KHO.

LC is never the machine owner: it has no PCI support compiled in, does 
not handle device interrupts, cannot handle page faults, etc. It only 
exists as a temporary caretaker. LC is booted entirely inside a small 
LUO-preserved memfd area (in our use case, LC's RAM is only ~128MB, 
while the host machine has terabytes of memory). Because LC's direct map 
only covers its own tiny memory footprint (for isolation reasons, and to 
save on struct pages), the KHO FDT, radix tree, and preserved VM memory 
all reside outside of LC's direct map.

LC is configured and launched directly from userspace while MO is still 
running. Therefore, LC does not receive bootloader handoff data from an 
outgoing kernel, as MO is not even shutting down yet. Userspace 
therefore reads needs PA address of KHO FDT and explicitly passes 
'kho_ro=<pa>' on LC's kernel command line.

> 
> > - Patch 2 moves the shared radix-tree key encoding and indexing,
> >   vmalloc flag conversion, and FDT validation and subtree lookup helpers
> >   to kernel/liveupdate/kexec_handover_internal.h so they can be shared
> >   between standard and read-only KHO.
> > - Patch 3 hardens kho_restore_vmalloc() by introducing
> >   kho_vmalloc_validate() in kernel/liveupdate/kexec_handover_internal.h
> >   to validate the preservation metadata header and flags before
> >   restoring the vmalloc area.
> 
> The diff for kexec_handover_internal.h 
> 
> >  kernel/liveupdate/kexec_handover_internal.h   | 171 +++++++-
> 
> seems a wee too large. Maybe some of this should move to a .c file?

Agreed. I originally kept them in the header as static inlines because 
they are mostly small bit manipulations and helper routines without loops, 
but moving them into a dedicated kernel/liveupdate/kho_core.c is OK.

Also, for v2, I am planning to allow CONFIG_KEXEC_HANDOVER and 
CONFIG_KEXEC_HANDOVER_READONLY to co-exist in the same build instead of 
being mutually exclusive. That way we won't need two different kernels 
for vmtests. In that model, common KHO API will use function vectors 
dynamiclly decided at early boot depending on which mode is activated 
('kho=on' vs 'kho_ro=<pa>').

> > - Patch 4 avoids modifying ser->next and bs->head_pa in kho_block_add()
> >   and kho_block_set_shrink_one() when operating on an incoming block
> >   set, as the chain is already linked and incoming blocks must not be
> >   modified in KHO read-only mode.
> > - Patch 5 adds CONFIG_KEXEC_HANDOVER_READONLY and
> >   kernel/liveupdate/kexec_handover_ro.c. When booted with
> >   'kho_ro=<pa>', preserved KHO metadata is mapped read-only
> >   (PAGE_KERNEL_RO), with the FDT and radix-tree nodes mapped lazily on
> >   first visit and kept cached, providing read-only implementations of
> >   kho_retrieve_subtree(), kho_restore_vmalloc(),
> >   kho_ro_lookup_folio_order(), kho_ro_map_folio(), and kho_ro_unmap().
> 
> What are the expected users of these APIs?

The primary use case is achieving ultra-low disruption Live Update 
(<30ms VM blackout time) while running a minimal VMM inside the caretaker 
kernel. This VMM keeps VMs responsive by servicing software-only, fully 
virtualized devices that do not require host hardware interrupts or global 
machine state (such as virtio-console, pvpanic, virtio-rng, etc.).

Thanks,
Pasha