Re: [PATCH v2] numa: add 'spm' option for Specific Purpose Memory

["Huang, FangSheng (Jerry)" <FangSheng.Huang@amd.com>]

Hi David,

I hope this email finds you well. I wanted to follow up on the SPM
patch series we discussed back in October.

I'm reaching out to check on the current status and see if there's
anything else I should address or any additional information I can
provide.

Thank you for your time and guidance on this!

Best regards,
Jerry Huang

On 10/22/2025 6:28 PM, David Hildenbrand wrote:
> On 22.10.25 12:09, Huang, FangSheng (Jerry) wrote:
>>
>>
>> On 10/21/2025 4:10 AM, David Hildenbrand wrote:
>>> On 20.10.25 11:07, fanhuang wrote:
>>>> Hi David and Igor,
>>>>
>>>> I apologize for the delayed response. Thank you very much for your
>>>> thoughtful
>>>> questions and feedback on the SPM patch series.
>>>>
>>>> Before addressing your questions, I'd like to briefly mention what the
>>>> new
>>>> QEMU patch series additionally resolves:
>>>>
>>>> 1. **Corrected SPM terminology**: Fixed the description error from the
>>>> previous
>>>>      version. The correct acronym is "Specific Purpose Memory" (not
>>>> "special
>>>>      purpose memory" as previously stated).
>>>>
>>>> 2. **Fixed overlapping E820 entries**: Updated the implementation to
>>>> properly
>>>>      handle overlapping E820 RAM entries before adding 
>>>> E820_SOFT_RESERVED
>>>>      regions.
>>>>
>>>>      The previous implementation created overlapping E820 entries by
>>>> first adding
>>>>      a large E820_RAM entry covering the entire above-4GB memory range,
>>>> then
>>>>      adding E820_SOFT_RESERVED entries for SPM regions that overlapped
>>>> with the
>>>>      RAM entry. This violated the E820 specification and caused 
>>>> OVMF/UEFI
>>>>      firmware to receive conflicting memory type information for the 
>>>> same
>>>>      physical addresses.
>>>>
>>>>      The new implementation processes SPM regions first to identify
>>>> reserved
>>>>      areas, then adds RAM entries around the SPM regions, generating a
>>>> clean,
>>>>      non-overlapping E820 map.
>>>>
>>>> Now, regarding your questions:
>>>>
>>>> ========================================================================
>>>> Why SPM Must Be Boot Memory
>>>> ========================================================================
>>>>
>>>> SPM cannot be implemented as hotplug memory (DIMM/NVDIMM) because:
>>>>
>>>> The primary goal of SPM is to ensure that memory is managed by guest
>>>> device drivers, not the guest OS. This requires boot-time discovery
>>>> for three key reasons:
>>>>
>>>> 1. SPM regions must appear in the E820 memory map as 
>>>> `E820_SOFT_RESERVED`
>>>>      during firmware initialization, before the OS starts.
>>>>
>>>> 2. Hotplug memory is integrated into kernel memory management, making
>>>>      it unavailable for device-specific use.
>>>>
>>>> ========================================================================
>>>> Detailed Use Case
>>>> ========================================================================
>>>>
>>>> **Background**
>>>> Unified Address Space for CPU and GPU:
>>>>
>>>> Modern heterogeneous computing architectures implement a coherent and
>>>> unified address space shared between CPUs and GPUs. Unlike traditional
>>>> discrete GPU designs with dedicated frame buffer, these accelerators
>>>> connect CPU and GPU through high-speed interconnects (e.g., XGMI):
>>>>
>>>> - **HBM (High Bandwidth Memory)**: Physically attached to each GPU,
>>>>     reported to the OS as driver-managed system memory
>>>>
>>>> - **XGMI (eXternal Global Memory Interconnect, aka. Infinity Fabric)**:
>>>>     Maintains data coherence between CPU and GPU, enabling direct CPU
>>>>     access to GPU HBM without data copying
>>>>
>>>> In this architecture, GPU HBM is reported as system memory to the OS,
>>>> but it needs to be managed exclusively by the GPU driver rather than
>>>> the general OS memory allocator. This driver-managed memory provides
>>>> optimal performance for GPU workloads while enabling coherent CPU-GPU
>>>> data sharing through the XGMI. This is where SPM (Specific Purpose
>>>> Memory) becomes essential.
>>>>
>>>> **Virtualization Scenario**
>>>>
>>>> In virtualization, hypervisor need to expose this memory topology to
>>>> guest VMs while maintaining the same driver-managed vs OS-managed
>>>> distinction.
>>>
>>> Just wondering, could device hotplug in that model ever work? I guess we
>>> wouldn't expose the memory at all in e820 (after all, it gets hotplugged
>>> later) and instead the device driver in the guest would have to
>>> detect+hotplug that memoory.
>>>
>>> But that sounds weird, because the device driver in the VM shouldn't do
>>> something virt specific.
>>>
>>> Which raises the question: how is device hoplug of such gpus handled on
>>> bare metal? Or does it simply not work? :)
>>>
>> Hi David, Thank you for your thoughtful feedback.
>> To directly answer your question:
>> in our use case, GPU device hotplug does NOT work on bare metal,
>> and this is by design.
> 
> Cool, thanks for clarifying!
>