Re: [RFCv2 PATCH 0/6] Support memory hotplug/unplug for TDX CoCo guests

["David Hildenbrand (Arm)" <david@kernel.org>]

On 8/27/26 10:18, Duan, Zhenzhong wrote:
> 
> 
>> -----Original Message-----
>> From: David Hildenbrand (Arm) <david@kernel.org>
>> Subject: Re: [RFCv2 PATCH 0/6] Support memory hotplug/unplug for TDX CoCo
>> guests
>>
>> On 6/23/26 12:17, Zhenzhong Duan wrote:
>>> This RFCv2 series implements comprehensive support for virtio-mem and ACPI
>>> DIMM memory hotplug/unplug in Intel TDX confidential computing guests.
>>> It explores the start-private memory approach utilizing the native
>>> TDG.MEM.PAGE.RELEASE API.
>>>
>>> We are seeking feedback from Kiryl on the CoCo guest implementation, MM
>>> experts on DIMM & virio-mem memory hotplug integration and broader
>>> virtio/CoCo community input on the overall approach. We are not seeking
>>> x86 maintainer review at this stage.
>>>
>>> == Changes from RFC v1 ==
>>>
>>> - Eliminated callback infrastructure: Dropped plug callback and replaced
>>>   unplug callback with platform-level unaccept function into core MM
>>>   hotplug and virtio-mem subsystems.
>>> - Added comprehensive bitmap tracking: Introduced a "plugged" bitmap
>>>   alongside the unaccepted bitmap to track populated hotplug memory
>>>   states to support load_unaligned_zeropad().
>>> - Enhanced SRAT parsing: Extended the EFI stub to parse ACPI SRAT tables
>>>   early, ensuring hotpluggable ranges are tracked from initial boot.
>>>
>>> For more introduction about the background or other efforts in community,
>>> please check the RFCv1 cover letter [1].
>>>
>>> == Technical Approach ==
>>>
>>> - Early SRAT Integration: A lightweight EFI stub parser scans ACPI SRAT
>>>   tables to identify hotpluggable ranges and adjust bitmap boundaries
>>>   early, avoiding the overhead of the full ACPI subsystem.
>>> - Comprehensive Bitmap Tracking: Introduces a "plugged" bitmap right
>>>   after the unaccepted bitmap. Both static and hotplugged memory are
>>>   tracked, allowing the guest to map which ranges are populated by the
>>>   VMM. This prevents acceptance beyond plugged memory boundaries due to
>>>   load_unaligned_zeropad() operations.
>>> - Platform Extensibility: Exposes generic CoCo memory interfaces. Other
>>>   confidential platforms (like AMD SEV-SNP) can easily adopt this by
>>>   hooking their specific mechanisms into arch_unaccept_memory().
>>> - Hotplug & Guest Control: Integrates platform-level unaccept logic
>>>   into ACPI hotplug and virtio-mem handlers. Uses TDG.MEM.PAGE.RELEASE
>>>   for TDX to explicitly set memory to the "unaccepted" state during
>>>   unplug, removing host hole-punching dependencies.
>>> - Kexec Handover: Leverages existing EFI mechanisms to seamlessly hand
>>>   over both the extended unaccepted bitmap and the new plugged bitmap
>>>   across kexec boundaries.
>>>
>>> == Testing ==
>>>
>>> - dimm and virtio-mem memory hotplug/unplug
>>> - lazy and eager accept
>>> - kexec/kdump with hotplugged memory
>>>
>>> This is tested with Marc-André Lureau's newest qemu series [2]
>>
>> What's the status of this?
> 
> Marc's QEMU series is merged.
> For this series, following feedback from Kirill and Pratik, the preferred approach
> is updating the UEFI spec for hotplug memory ranges rather than parsing SRAT
> at the EFI stage. Pratik is already pushing this forward, I am currently waiting on
> his RFCs. If he hasn't taken over the entire implementation, I can rebase my
> remaining patches on top of his work.
> 
> Hi Pratik, have you sent your UEFI RFC out yet? Just wanted to make sure
> I didn't miss your thread.
> 
>>
>> I am still not sure whether we shouldn't perform acceptance from
>> move_pfn_range_to_zone() and from memory notifiers / generic_online_page.
> 
> My understanding is that we already have full support for lazy and eager acceptance
> in generic_online_page() for static memory. We should be able to reuse that for
> hotplug memory and avoid adding acceptance logic in other places.
> 
> All we need is extending unaccept_bitmap and adding new plugged_bitmap to support
> hotplug memory. I updated accept_memory() to check both bitmaps to determine
> which memory should be accepted.

The plugged bitmap is a very odd beast. I hate it, but I can see why it might
currently be required. I wonder if there is a better name for it because

	"plugged"

is an overloaded term. What are the real semantics we want to express?

IIUC, unplug for virtio-mem requires prior conversion to shared memory. We
should have an intuitive mechanism for virtio-mem to just do the right thing
when unplugging memory (IOW, preparing for handback to the hypervisor).

> 
>>
>> In particular, it's unclear to me how virtio-mem (which uses interfaces to
>> add/remove memory) interacts with unaccept_memory / coco bitmap.
> 
> It works the same way as a physical DIMM: when memory is plugged, the
> corresponding bits in plugged_bitmap are set, and vice versa.

Well, no. When adding a Linux memory block through add_memory_resource() you do
coco_set_plugged_bitmap(). And in virtio_mem_send_plug_request() you do
coco_set_plugged_bitmap().

That's just super inconsistent and messy.


(coco_set_plugged_bitmap() and memory acceptance should *definitely not* be
open-coded like that in virtio_mem. There must be a clear abstraction layer with
clear, well documented semantics that virito-mem can iuse)


> 
> Memory acceptance is already handled in generic_online_page(), so we 
> do not need to do it inside virtio-mem. However, we do need to call
> unaccept_memory() during a memory unplug event in virtio-mem.

Again, I think we really need an abstraction that can just naturally be extended
for platforms that have to perform some work when returning memory to the
hypervisor. Open-coding x86's unaccept_memory() is not the way to go.

> 
> Currently, tdx_unaccept_memory() can act as a no-op since QEMU handles 
> hole-punching the private memory. That said, we still need to invoke 
> unaccept_memory() to properly update the unaccept_bitmap bits.
> 
>>
>> Can we have an overall design view on what happens at which stage when adding
>> /
>> removing memory through virtio-mem?
> 
> I have put together a design view summary for virtio-mem below.
> Please let me know if this looks correct or if we should adjust the framing.
> 
> Design Overview
> ---------------
> We maintain system stability and state safety using two metadata tracking
> layers during dynamic memory resizing operations:
> 1. plugged_bitmap: Explicitly tracks blocks plugged into the guest.
>    This protects load_unaligned_zeropad() from reading omitted memory
>    holes, preventing catastrophic guest crashes.

I hate load_unaligned_zeropad() so much at this point. We should finally rip it
out. I wish I would have more spare time to look into that.

The plugged bitmap is a clear sign that load_unaligned_zeropad() just has to go
instead of us hacking around it.

> 2. unaccept_bitmap: Explicitly tracks the secure page initialization state.

I didn't fully grasp the level of hackery we have to apply to make
load_unaligned_zeropad() not do stupid things.

Am I correct that we have to accept more memory, possibly falling into unplugged
virtio-mem ranges? What is the effect of that?

> 
> Step-by-Step Lifecycle Stages
> -----------------------------
> Using sub-block hotplug of a new memory block with eager acceptance
> as an example:
> 
> 1. Memory Addition (Plug) Stage
>    a. Host notifies guest -> virtio-mem driver handles the plug event.
>    b. Driver marks the allocated memory ranges in 'plugged_bitmap' by
>       calling coco_set_plugged_bitmap(addr, size, true). During this stage,
>       a plug request is also sent to the VMM.
>    c. Driver adds memory blocks via add_memory_resource().

Assume you hotplug a single device block (e.g., 2M). virtio-mem will set the
plugged bitmap of that one block. But add_memory_resource() will set the plugged
bitmap of the entire Linux memory block.

That seems completely broken?


-- 
Cheers,

David