Re: [PATCH v4 09/14] kexec: Add documentation for KHO

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Feb 06, 2025 at 03:27:49PM +0200, Mike Rapoport wrote:

> +KHO introduces a new concept to its device tree: ``mem`` properties. A
> +``mem`` property can be inside any subnode in the device tree. 

I do not think this is a good idea.

It should be core infrastructure, totally unrelated to any per-device
fdt nodes, to carry the memory map.

IOW a full DT that looks something more like:

/dts-v1/;

/ {
  compatible = "linux-kho,v1";
  allocated-memory {
        <>
  };

  ftracebuffer {
  	       compatible = "linux-kho,ftracem,v1";
	       ftrace-buffer-phys = <..>;
	       ftrace-buffer-len = <..>;
	       ..etc..
  };
};

Where allocated_memory will remove all memory from the buddy allocator
very early on in an efficient way. that process should not be walking
the fdt to find mem nodes.

> +After boot, drivers can call the kho subsystem to transfer ownership of memory
> +that was reserved via a ``mem`` property to themselves to continue using memory
> +from the previous execution.

And this transfer should be done by phys that the node itself
describes.

Ie if ftrace has a single high order folio to store it's ftrace buffer
then I would expect code like:

allocate ftrace:
  buffer = folio_alloc(..);

activate callback:
   kho_preserve_folio(buffer)
   fdt...("ftrace-buffer-phys", virt_to_phys(buffer))

restore callback:
   buffer_phys = fdt..("ftrace-buffer-phys")
   buffer = kho_restore_folio(buffer_phys)
   [..]

destroy ftrace:
   folio_put(buffer);

And kho will take care to restore the struct folio, put back the
order, etc, etc.

Similar for slab.

I think this sort of memory-based operation should be the very basic
core building primitive here.

So the allocated-memory node should preserve information about KHO'd
folios, their order and so on.

It doesn't matter what part of the FDT owns those folios, all the core
kernel should do is keep track of them and at some point check that
all preserved folios have been claimed.

> +We guarantee that we always have such regions through the scratch regions: On
> +first boot KHO allocates several physically contiguous memory regions. Since
> +after kexec these regions will be used by early memory allocations, there is a
> +scratch region per NUMA node plus a scratch region to satisfy allocations
> +requests that do not require particilar NUMA node assignment.

This plan sounds great, way better than the pmem approaches/etc.

> +To enable user space based kexec file loader, the kernel needs to be able to
> +provide the device tree that describes the previous kernel's state before
> +performing the actual kexec. The process of generating that device tree is
> +called serialization. When the device tree is generated, some properties
> +of the system may become immutable because they are already written down
> +in the device tree. That state is called the KHO active phase.

This should have a whole state diagram as we've talked a few
times. There is alot more to worry about here than just 'activate'.

Jason