Re: [PATCH RFC v2 00/15] Add virtualization support for EGM

[Alex Williamson <alex@shazbot.org>]

On Wed, 11 Mar 2026 06:47:12 +0000
Ankit Agrawal <ankita@nvidia.com> wrote:

> Thanks Alex for the review.
> 
> >> The patch series introduce a new nvgrace-egm auxiliary driver module
> >> to manage and map the HI/EGM region in the Grace Blackwell systems.
> >> This binds to the auxiliary device created by the parent
> >> nvgrace-gpu (in-tree module for device assignment) / nvidia-vgpu-vfio
> >> (out-of-tree open source module for SRIOV vGPU) to manage the
> >> EGM region for the VM. Note that there is a unique EGM region per
> >> socket and the auxiliary device gets created for every region. The
> >> parent module fetches the EGM region information from the ACPI
> >> tables and populate to the data structures shared with the auxiliary
> >> nvgrace-egm module.
> >>
> >> nvgrace-egm module handles the following:
> >> 1. Fetch the EGM memory properties (base HPA, length, proximity domain)
> >> from the parent device shared EGM region structure.
> >> 2. Create a char device that can be used as memory-backend-file by Qemu
> >> for the VM and implement file operations. The char device is /dev/egmX,
> >> where X is the PXM node ID of the EGM being mapped fetched in 1.
> >> 3. Zero the EGM memory on first device open().
> >> 4. Map the QEMU VMA to the EGM region using remap_pfn_range.
> >> 5. Cleaning up state and destroying the chardev on device unbind.
> >> 6. Handle presence of retired poisoned pages on the EGM region.
> >>
> >> Since nvgrace-egm is an auxiliary module to the nvgrace-gpu, it is kept
> >> in the same directory.  
> >
> >
> > Pondering this series for a bit, is this auxiliary chardev approach
> > really the model we should be pursuing?
> >
> > I know we're trying to disassociate the EGM region from the GPU, and
> > de-duplicate it between GPUs on the same socket, but is there actually a
> > use case of the EGM chardev separate from the GPU?  
> 
> It is not just de-duplication. The EGM is a carveout of system memory
> logically and physically separate and disconnected from the GPU. The
> uniqueness here is that the information (SPA, size) of the region is present
> on the GPU ACPI tables.
> 
> >
> > The independent lifecycle of this aux device is troubling and it hasn't
> > been confirmed whether or not access to the EGM region has some
> >dependency on the state of the GPU.   
> 
> The EGM region is independent on the state of the GPU. One can plausibly
> bootup the VM with just the EGM memory chardev as the backend file and
> no GPU.

Seems like we have the wrong model then to base the lifecycle of the
aux devices on the state of the PCI driver if EGM is fully independent
of the state of the PCI device.

> > nvgrace-gpu is manipulating sysfs
> > on devices owned by nvgrace-egm, we don't have mechanisms to manage the
> > aux device relative to the state of the GPU, we're trying to add a
> > driver that can bind to device created by an out-of-tree driver, and
> > we're inventing new uAPIs on the chardev for things that already exist
> > for vfio regions.  
> 
> Sorry for the confusion. The nvgrace-egm would not bind to the device
> created by the out-of-tree driver. We would have a separate out-of-tree
> equivalent of nvgrace-egm to bind to the device by the out-of-tree vfio
> driver. Maybe we can consider exposing a register/unregister APIs from
> nvgrace-egm where a module (in-tree nvgrace / out-of-tree) can register
> a pdev and nvgrace-egm can use to fetch the region info.

Ok, this wasn't clear to me, but does that also mean that if some GPUs
are managed by nvgrace-gpu and others by out-of-tree drivers that the
in-kernel and out-of-tree equivalent drivers are both installing
chardevs as /dev/egmXX?  Playing in the same space is ugly, but what
happens when the 2 GPUs per socket are split between drivers and they
both try to added the same chardev?

> > Therefore, does it actually make more sense to expose EGM as a device
> > specific region on the vfio device fd?
> >
> > For example, nvgrace-gpu might manage the de-duplication by only
> > exposing this device specific region on the lowest BDF GPU per socket.
> > The existing REGION_INFO ioctl handles reporting the size to the user.
> > The direct association to the GPU device handles reporting the node
> > locality.  If necessary, a capability on the region could report the
> > associated PXM, and maybe even the retired page list.
> >
> > All of the lifecycle issues are automatically handled, there's no
> > separate aux device.  If necessary, zapping and faulting across reset
> > is handled just like a BAR mapping.  
> 
> The EGM memory (which becomes the system memory of the VM) cannot
> be connected to the GPU reset as it is unrelated to the GPU device. We would
> not want that to happen to system memory on GPU reset.

It's not the state of the EGM/system memory that I'm concerned about,
it's the fact that the routing to access that memory traverses two
GPUs and both the backplane and c2c NVLink connections.  If access
through that channel is 100% independent of the state of either GPU
then GPU resets are irrelevant.

However, I'd then ask the question why we're associating EGM to the GPU
PCI driver at all.  For instance, why should nvgrace-gpu spawn aux
devices to feed into an nvgrace-egm driver, and duplicate that whole
thing in an out-of-tree driver, when we could just have one in-kernel
platform(?) driver walk ACPI, find these ranges, and expose them as
chardev entirely independent of the PCI driver bound to the GPU?
 
> > If we need to expose the EGM size and GPU association via sysfs for
> > management tooling, nvgrace-gpu could add an "egm_size" attribute to the
> > PCI device's sysfs node.  This could also avoid the implicit
> >  implementation knowledge about which GPU exposes the EGM device
> > specific region.
> >
> > Was such a design considered?  It seems much, much simpler and could be
> > implemented by either nvgrace-gpu or identically by an out-of-tree
> > driver without references in an in-kernel ID table.
> >
> > I'd like to understand the pros and cons of such an approach vs the one
> > presented here.  Thanks,  
> 
> We didn't consider it as a separate BAR / region as the EGM memory (part of the
> system memory) is unrelated to the GPU device besides having its information
> in the GPU ACPI table and becomes the system memory of the VM. Considering
> it as part of the device BAR / region would connect the lifecyle of the EGM region
> on the GPU device. Also we cannot consider zapping/faulting across GPU reset
> as it is system memory of the VM.

It's curious why the EGM description is associated to the GPU ACPI
object if it really is fully independent.  It seems like perhaps it
should be a unique ACPI object in that case, which would make claiming
it via a platform driver easier.  Maybe we don't need to be tied to
that firmware decision in the design of the software driver though.
Thanks,

Alex