[PATCH 0/4] Introduce nvgrace-egm driver for Extended GPU Memory

[PATCH 0/4] Introduce nvgrace-egm driver for Extended GPU Memory

[Ankit Agrawal]

Background
==========
NVIDIA Grace-based systems (Grace Hopper, Grace Blackwell Superchip) have
a special bios-configurable mode called EGM (Extended GPU Memory) that
allows a carveout to be created from the system memory. This carveout is a
contiguous region of special range of system DRAM reserved by the UEFI
boot firmware.

This range does not appear in the EFI memory map and is therefore invisible
to the host kernel. i.e. there are no struct pages for it and it cannot
be managed by the normal page allocator. This range is described in ACPI
as vendor DSD properties on the GPU PCI device node (see ACPI Design
below). It is intended to be used by a VMM to back an entire virtual
machine's physical memory. The VMM maps the range into its address space
via mmap() on a char device and passes it to KVM as the backing store for
guest RAM similar to memfd or hugetlbfs. KVM builds Stage-2 page tables
mapping the guest's GPA range linearly to the EGM SPAs i.e. a guest offset
X maps to SPA (egm_base + X).

The range is special because the GPU on these systems has direct high-speed
access to it that bypasses the host system IOMMU. Normal PCI DMA is
gated by the IOMMU and requires an explicit IOMMU mapping. The GPU's path
to this memory goes via NVLink/NVSwitch directly to the memory controller
and does not involve the host IOMMU. Since KVM maps guest GPA to EGM SPA
linearly, the GPU can reach guest memory at those same SPAs without
needing any additional host IOMMU mapping. Normal system memory that the
kernel sees is protected by the IOMMU in the usual way and this range is
deliberately kept separate.

To make this work a driver is needed to parse the ACPI, capture the
physical memory range, expose it as a char device and allow it to be
mmap'd for KVM. A VMM like qemu can open this char dev and then use it
much like a hugetlbfs or memfd to back the VM. Due to the security
difference the memory has to be strictly separated from the rest of the
hypervisor memory. Exposing the range as a DMABUF for iommufd is planned
as future work and is not part of this series.

This hardware feature allows the system memory to be access across nodes
on a multi-node system such as Grace Blackwell NVL72. It can also provide
throughput improvements because the GPU can access VM system memory via
its high-bandwidth fabric rather than going through the IOMMU-gated PCIe
path. Moreover skipping IOMMU translations could also improve performance
for any use cases targeted towards high performance.

Due to the security difference, the EGM range is strictly isolated from
the rest of the hypervisor memory (which is covered through IOMMU).

A driver is needed to:
  - Parse ACPI to locate the physical range (base SPA, size, proximity
    domain)
  - Expose the range as a char device (/dev/egmX, X = NUMA proximity
    domain) that a VMM such as QEMU can open and mmap to back guest RAM
  - Zero the region on first open to isolate successive VM uses
  - Track and expose retired ECC pages to the VMM via ioctl
  - Register the PFN range with the memory_failure infrastructure for
    runtime uncorrectable error handling


ACPI Design
===========
The EGM region properties are exposed as NVIDIA vendor DSD properties on
the GPU's PCI ACPI device node:

  nvidia,egm-pxm                     NUMA proximity domain of the region
  nvidia,egm-base-pa                 Physical base address (SPA) of the
                                     region
  nvidia,egm-size                    Length of the region in bytes
  nvidia,egm-retired-pages-data-base Physical address of the firmware-
                                     provided retired ECC pages table

This ACPI construct helps with describing and discovering this physically
present memory range that is intentionally absent from the EFI memory map
and is tied to a specific device for its access properties.

On a multi-socket system there is one EGM region per socket. Multiple GPUs
on the same socket advertise identical region properties (same proximity
domain, base address and size). The driver deduplicates by proximity
domain: the first GPU probed for a given domain creates the char device
and subsequent GPUs on the same socket are linked to it via sysfs.


Implementation
==============
Patch 1 introduces the driver skeleton: Kconfig/Makefile/MAINTAINERS, the
egm device class, the char device minor number range (MAX_EGM_NODES=4
matching the largest 4-socket Grace configuration), ACPI property walking
(nvidia,egm-pxm / nvidia,egm-base-pa / nvidia,egm-size), per-region state
tracking with PXM-based deduplication, char device creation (/dev/egmX),
GPU-to-EGM sysfs symlinks and the egm_size sysfs attribute.

Patch 2 implements the file operations: memory scrubbing on the first
open() using memremap()/memset() in 1 GiB chunks with cond_resched() to
avoid RCU stalls and mmap() via remap_pfn_range().

Patch 3 adds retired ECC page handling: reads the firmware-provided
retired-pages table from the ACPI DSD property
nvidia,egm-retired-pages-data-base at module init, stores the page offsets
in a per-device xarray and exposes them to userspace via the
EGM_RETIRED_PAGES_LIST ioctl (UAPI header include/uapi/linux/egm.h).
Each firmware entry covers a 64 KiB retired region. The driver inserts one
xarray entry per kernel page so a lookup at the running page size hits the
right retired range on both 4K and 64K kernels.

Patch 4 registers the EGM PFN range with the memory_failure infrastructure
on the first open() and unregisters it on the last close(). The
pfn_to_vma_pgoff callback validates that a reported PFN falls within the
EGM region, converts it to an in-region file offset, and inserts it into
the retired-pages xarray so runtime uncorrectable errors are surfaced
alongside ACPI-reported retired pages via the same ioctl.


Enablement
==========
EGM mode is enabled through a flag in the platform firmware (SBIOS). The
size of the host-visible partition is separately configurable; all
remaining system DRAM is reserved as the EGM region invisible to the
hypervisor.


RFC Posting
===========
Link: https://lore.kernel.org/all/20260223155514.152435-1-ankita@nvidia.com/


Verification
============
Tested on a 2-socket (4 GPU) Grace Blackwell platform by booting a VM with
QEMU [1] and kernel branch [2] and confirming EGM capability is visible
inside the guest and basic cuda workloads. Note that this has dependency
on the generic dmabuf importer/exporter work in progress [3]. Until then, a
bypass PFNMAP workaround is used [4] (also part of the kernel branch [2].


Link: https://github.com/NVIDIA/QEMU/tree/nvidia_stable-10.1 [1]
Link: https://github.com/ankita-nv/linux/tree/v7.2-egm-01072026 [2]
Link: https://lore.kernel.org/all/0-v1-b5cab63049c0+191af-dmabuf_map_type_jgg@nvidia.com/ [3]
Link: https://github.com/ankita-nv/linux/commit/f2a915d00916c5ef5e25f7bc2ad5c10efd36fb4b [4]


Suggested-by: Alex Williamson <alex@shazbot.org>
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aniket Agashe <aniketa@nvidia.com>
Suggested-by: Vikram Sethi <vsethi@nvidia.com>
Suggested-by: Matthew R. Ochs <mochs@nvidia.com>

Ankit Agrawal (4):
  platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM
    regions
  platform/nvidia: Implement mmap and memory scrubbing for EGM chardev
  platform/nvidia: Handle retired ECC pages and expose via ioctl
  platform/nvidia: Register EGM PFNMAP range with memory_failure

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 MAINTAINERS                                   |   6 +
 drivers/platform/Kconfig                      |   1 +
 drivers/platform/Makefile                     |   1 +
 drivers/platform/nvidia/Kconfig               |  10 +
 drivers/platform/nvidia/Makefile              |   3 +
 drivers/platform/nvidia/egm.c                 | 874 ++++++++++++++++++
 include/uapi/linux/egm.h                      |  28 +
 8 files changed, 924 insertions(+)
 create mode 100644 drivers/platform/nvidia/Kconfig
 create mode 100644 drivers/platform/nvidia/Makefile
 create mode 100644 drivers/platform/nvidia/egm.c
 create mode 100644 include/uapi/linux/egm.h

-- 
2.34.1

[PATCH 1/4] platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM regions

[Ankit Agrawal]

The Extended GPU Memory (EGM) feature enables the GPU to access system
memory allocations within and across nodes through a high-bandwidth path
on Grace-based systems. The GPU can utilize system memory on the same
socket, a different socket or a different node in a multi-node setup.

When EGM mode is enabled through SBIOS, host system memory is partitioned
into two parts: a Hypervisor region for the host OS and a
Hypervisor-Invisible (HI) region for the VM. Only the hypervisor region
is part of the host EFI map and visible to the host OS on bootup. The
HI partition is interchangeably called the EGM region. Its base SPA and
size are exposed through ACPI DSDT properties.

Whilst the EGM region is accessible on the host, it is not
added/hot-plugged to the kernel. The HI region is assigned to a VM
by mapping the QEMU VMA to the SPA using remap_pfn_range(). So
it looks like the following:

  |---- Sysmem ----|                  VM view
  |                |
  |IPA <-> SPA map |
  |                |
  |--- HI / EGM ---|-- Host Mem --|   Host view

Introduce the nvgrace-egm platform driver module and implement full EGM
region discovery, char device exposure and sysfs metadata.

Build infrastructure:
- Kconfig, Makefile entries, MAINTAINERS entry.
- An 'egm' device class and a range of char device minor numbers.
  MAX_EGM_NODES is set to 4 to matches the largest 4-socket Grace
  configuration.

Discovery:
- Walk all PCI devices at module init time. For each one advertising
  "nvidia,egm-pxm", read the EGM region properties (base SPA, size and
  PXM domain) from the ACPI DSD table via the GPU device handle, using
  "nvidia,egm-base-pa" and "nvidia,egm-size" properties.
- Maintain a module-level list (egm_chardevs) of discovered regions,
  de-duplicating by PXM so each socket gets exactly one EGM device
  regardless of how many GPUs it has (2 GPUs per EGM device in GB200).

Char device creation:
- Each EGM region is exposed as /dev/egmX, where X is the PXM number.
  Embed struct device and struct cdev inside nvgrace_egm_dev so the
  device lifecycle follows the standard kernel device model.
- Implement setup_egm_chardev() and del_egm_chardev() for setup and
  teardown. The devnode callback sets device permissions to 0600.
- Stub file operations (open, release and mmap) are registered.

GPU tracking and sysfs:
- Track which PCI devices are associated with each EGM region in a
  per-device GPU list (struct gpu_node).
- Expose sysfs symlinks from each EGM device to its associated GPU PCI
  devices to allow the userspace to determine GPU-to-socket affinity and
  replicate the host EGM topology in the VM. Helps admin understand
  GPU association.
- Expose the EGM region size via an egm_size sysfs attribute so QEMU
  can determine the memory-backend-file size. On a 2-socket Grace
  Blackwell setup this appears as:
    Socket0: /sys/devices/virtual/egm/egm4/egm_size
    Socket1: /sys/devices/virtual/egm/egm5/egm_size

Introduce nvgrace_egm_create_pci_egm_devs() and its counterpart
nvgrace_egm_destroy_pci_devs() that are called from module init/exit.

Suggested-by: Alex Williamson <alex@shazbot.org>
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aniket Agashe <aniketa@nvidia.com>
Suggested-by: Matthew R. Ochs <mochs@nvidia.com>
Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 MAINTAINERS                      |   5 +
 drivers/platform/Kconfig         |   1 +
 drivers/platform/Makefile        |   1 +
 drivers/platform/nvidia/Kconfig  |  10 +
 drivers/platform/nvidia/Makefile |   3 +
 drivers/platform/nvidia/egm.c    | 459 +++++++++++++++++++++++++++++++
 6 files changed, 479 insertions(+)
 create mode 100644 drivers/platform/nvidia/Kconfig
 create mode 100644 drivers/platform/nvidia/Makefile
 create mode 100644 drivers/platform/nvidia/egm.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 15011f5752a9..2edf903a7746 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19344,6 +19344,11 @@ L:	linux-acpi@vger.kernel.org
 S:	Maintained
 F:	drivers/acpi/apei/ghes-nvidia.c
 
+NVIDIA GRACE EGM PLATFORM DRIVER
+M:	Ankit Agrawal <ankita@nvidia.com>
+S:	Supported
+F:	drivers/platform/nvidia/
+
 NVIDIA VRS RTC DRIVER
 M:	Shubhi Garg <shgarg@nvidia.com>
 L:	linux-tegra@vger.kernel.org
diff --git a/drivers/platform/Kconfig b/drivers/platform/Kconfig
index 312788f249c9..21f2dc901301 100644
--- a/drivers/platform/Kconfig
+++ b/drivers/platform/Kconfig
@@ -22,3 +22,4 @@ source "drivers/platform/arm64/Kconfig"
 source "drivers/platform/raspberrypi/Kconfig"
 
 source "drivers/platform/wmi/Kconfig"
+source "drivers/platform/nvidia/Kconfig"
diff --git a/drivers/platform/Makefile b/drivers/platform/Makefile
index fa322e7f8716..96de8fa26ed5 100644
--- a/drivers/platform/Makefile
+++ b/drivers/platform/Makefile
@@ -15,3 +15,4 @@ obj-$(CONFIG_SURFACE_PLATFORMS)	+= surface/
 obj-$(CONFIG_ARM64_PLATFORM_DEVICES)	+= arm64/
 obj-$(CONFIG_BCM2835_VCHIQ)	+= raspberrypi/
 obj-$(CONFIG_ACPI_WMI)		+= wmi/
+obj-$(CONFIG_NVGRACE_EGM)	+= nvidia/
diff --git a/drivers/platform/nvidia/Kconfig b/drivers/platform/nvidia/Kconfig
new file mode 100644
index 000000000000..8554a58b1141
--- /dev/null
+++ b/drivers/platform/nvidia/Kconfig
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config NVGRACE_EGM
+	tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip"
+	depends on ARM64 || (COMPILE_TEST && 64BIT)
+	help
+	  Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace
+	  based chips required to avail the CPU memory as additional
+	  cross-node/cross-socket memory for GPU using KVM/qemu.
+
+	  If you don't know what to do here, say N.
diff --git a/drivers/platform/nvidia/Makefile b/drivers/platform/nvidia/Makefile
new file mode 100644
index 000000000000..a89f820908fb
--- /dev/null
+++ b/drivers/platform/nvidia/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_NVGRACE_EGM) += nvgrace-egm.o
+nvgrace-egm-y := egm.o
diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c
new file mode 100644
index 000000000000..a340c4fcbc7a
--- /dev/null
+++ b/drivers/platform/nvidia/egm.c
@@ -0,0 +1,459 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#include <linux/vfio_pci_core.h>
+
+#define NVGRACE_EGM_DEV_NAME "egm"
+#define MAX_EGM_NODES 4
+
+static dev_t dev;
+static struct class *class;
+
+struct gpu_node {
+	struct list_head list;
+	struct pci_dev *pdev;
+};
+
+struct nvgrace_egm_dev {
+	struct device device;
+	struct cdev cdev;
+	phys_addr_t egmphys;
+	size_t egmlength;
+	u64 egmpxm;
+	struct list_head gpus;
+};
+
+struct nvgrace_egm_dev_entry {
+	struct list_head list;
+	struct nvgrace_egm_dev *egm_dev;
+};
+
+/*
+ * Track egm device lists. Note that there is one device per socket.
+ * All the GPUs belonging to the same sockets are associated with
+ * the EGM device for that socket.
+ */
+static LIST_HEAD(egm_chardevs);
+
+static int nvgrace_egm_open(struct inode *inode, struct file *file)
+{
+	return 0;
+}
+
+static int nvgrace_egm_release(struct inode *inode, struct file *file)
+{
+	return 0;
+}
+
+static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	/*
+	 * Mapping the EGM region into userspace is implemented by a later
+	 * patch. Until then refuse the mmap.
+	 */
+	return -EOPNOTSUPP;
+}
+
+static const struct file_operations file_ops = {
+	.owner = THIS_MODULE,
+	.open = nvgrace_egm_open,
+	.release = nvgrace_egm_release,
+	.mmap = nvgrace_egm_mmap,
+};
+
+static int nvgrace_egm_create_gpu_links(struct nvgrace_egm_dev *egm_dev,
+					struct pci_dev *pdev)
+{
+	int ret;
+
+	ret = sysfs_create_link(&egm_dev->device.kobj,
+				&pdev->dev.kobj,
+				dev_name(&pdev->dev));
+
+	if (ret && ret != -EEXIST)
+		return ret;
+
+	return 0;
+}
+
+static void remove_egm_symlinks(struct nvgrace_egm_dev *egm_dev,
+				struct pci_dev *pdev)
+{
+	sysfs_remove_link(&egm_dev->device.kobj, dev_name(&pdev->dev));
+}
+
+static int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev)
+{
+	struct gpu_node *node;
+
+	node = kzalloc_obj(*node, GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	/*
+	 * The pdev is stored for the lifetime of the EGM device but it is
+	 * obtained from a for_each_pci_dev() walk that drops its reference as
+	 * it advances. Take a reference here so the stored pointer stays valid
+	 * even if the GPU is later unbound; remove_gpus() drops it.
+	 */
+	node->pdev = pci_dev_get(pdev);
+
+	list_add_tail(&node->list, &egm_dev->gpus);
+
+	return nvgrace_egm_create_gpu_links(egm_dev, pdev);
+}
+
+static void remove_gpus(struct nvgrace_egm_dev *egm_dev)
+{
+	struct gpu_node *node, *tmp;
+
+	list_for_each_entry_safe(node, tmp, &egm_dev->gpus, list) {
+		remove_egm_symlinks(egm_dev, node->pdev);
+		list_del(&node->list);
+		pci_dev_put(node->pdev);
+		kfree(node);
+	}
+}
+
+static void egm_chardev_release(struct device *dev)
+{
+	struct nvgrace_egm_dev *egm_chardev = container_of(dev, struct nvgrace_egm_dev, device);
+
+	remove_gpus(egm_chardev);
+	kfree(egm_chardev);
+}
+
+static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
+						 u64 egmpxm)
+{
+	struct nvgrace_egm_dev *egm_chardev;
+	unsigned int baseminor = MINOR(dev);
+	int ret;
+
+	/*
+	 * Only MAX_EGM_NODES minors from baseminor were reserved. Reject a PXM
+	 * outside that window.
+	 */
+	if (egmpxm < baseminor || egmpxm - baseminor >= MAX_EGM_NODES) {
+		pr_err("nvgrace-egm: EGM proximity domain %llu outside reserved minor window [%u, %u)\n",
+		       egmpxm, baseminor, baseminor + MAX_EGM_NODES);
+		goto create_err;
+	}
+
+	egm_chardev = kzalloc_obj(*egm_chardev, GFP_KERNEL);
+	if (!egm_chardev)
+		goto create_err;
+
+	device_initialize(&egm_chardev->device);
+
+	/*
+	 * Use the proximity domain number as the device minor number.
+	 * So the EGM corresponding to node X would be /dev/egmX.
+	 */
+	egm_chardev->egmphys = egmphys;
+	egm_chardev->egmlength = egmlength;
+	egm_chardev->egmpxm = egmpxm;
+	INIT_LIST_HEAD(&egm_chardev->gpus);
+
+	egm_chardev->device.devt = MKDEV(MAJOR(dev), egm_chardev->egmpxm);
+	egm_chardev->device.class = class;
+	egm_chardev->device.release = egm_chardev_release;
+	cdev_init(&egm_chardev->cdev, &file_ops);
+	egm_chardev->cdev.owner = THIS_MODULE;
+
+	ret = dev_set_name(&egm_chardev->device, "egm%llu", egm_chardev->egmpxm);
+	if (ret)
+		goto error_exit;
+
+	ret = cdev_device_add(&egm_chardev->cdev, &egm_chardev->device);
+	if (ret)
+		goto error_exit;
+
+	return egm_chardev;
+
+error_exit:
+	put_device(&egm_chardev->device);
+create_err:
+	return NULL;
+}
+
+static void del_egm_chardev(struct nvgrace_egm_dev *egm_chardev)
+{
+	cdev_device_del(&egm_chardev->cdev, &egm_chardev->device);
+	put_device(&egm_chardev->device);
+}
+
+static char *egm_devnode(const struct device *device, umode_t *mode)
+{
+	if (mode)
+		*mode = 0600;
+
+	return NULL;
+}
+
+static ssize_t egm_size_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct nvgrace_egm_dev *egm_dev =
+		container_of(dev, struct nvgrace_egm_dev, device);
+
+	return sysfs_emit(buf, "0x%zx\n", egm_dev->egmlength);
+}
+
+static DEVICE_ATTR_RO(egm_size);
+
+static struct attribute *attrs[] = {
+	&dev_attr_egm_size.attr,
+	NULL,
+};
+
+static const struct attribute_group attr_group = {
+	.attrs = attrs,
+};
+
+static const struct attribute_group *attr_groups[] = {
+	&attr_group,
+	NULL
+};
+
+/*
+ * Determine if the EGM feature is enabled. If disabled, there
+ * will be no EGM properties populated in the ACPI tables and this
+ * fetch would fail.
+ */
+static int has_egm_property(struct pci_dev *pdev, u64 *pegmpxm)
+{
+	return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm",
+					pegmpxm);
+}
+
+static int fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys,
+			      u64 *pegmlength)
+{
+	int ret;
+
+	/*
+	 * The memory information is present in the system ACPI tables as DSD
+	 * properties nvidia,egm-base-pa and nvidia,egm-size.
+	 */
+	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size",
+				       pegmlength);
+	if (ret)
+		return ret;
+
+	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa",
+				       pegmphys);
+
+	return ret;
+}
+
+static struct nvgrace_egm_dev *get_egm_dev(u64 egmpxm)
+{
+	struct nvgrace_egm_dev_entry *egm_entry;
+
+	list_for_each_entry(egm_entry, &egm_chardevs, list) {
+		/*
+		 * A system could have multiple GPUs associated with an
+		 * EGM region and will have the same set of EGM region
+		 * information. Return the existing EGM device if already
+		 * registered through a different GPU on the same socket.
+		 */
+		if (egm_entry->egm_dev->egmpxm == egmpxm)
+			return egm_entry->egm_dev;
+	}
+
+	return NULL;
+}
+
+static void nvgrace_egm_destroy_pci_devs(void)
+{
+	struct nvgrace_egm_dev_entry *entry, *tmp;
+
+	list_for_each_entry_safe(entry, tmp, &egm_chardevs, list) {
+		list_del(&entry->list);
+		del_egm_chardev(entry->egm_dev);
+		kfree(entry);
+	}
+}
+
+/*
+ * Walk all PCI devices and for each one that has a companion ACPI object
+ * advertising EGM properties, create an auxiliary device for that EGM range.
+ * Multiple GPUs on the same socket share a single EGM region (same proximity
+ * domain). So de-duplicate by skipping a pxm that is already registered.
+ */
+static int nvgrace_egm_create_pci_egm_devs(void)
+{
+	struct nvgrace_egm_dev_entry *egm_entry;
+	struct nvgrace_egm_dev *egm_dev;
+	struct pci_dev *pdev = NULL;
+	int ret;
+
+	for_each_pci_dev(pdev) {
+		u64 egmphys, egmlength, egmpxm;
+
+		if (has_egm_property(pdev, &egmpxm))
+			continue;
+
+		ret = fetch_egm_property(pdev, &egmphys, &egmlength);
+		if (ret)
+			continue;
+
+		egm_dev = get_egm_dev(egmpxm);
+		if (egm_dev) {
+			ret = add_gpu(egm_dev, pdev);
+			if (ret)
+				goto free_dev;
+			continue;
+		}
+
+		egm_entry = kzalloc_obj(*egm_entry, GFP_KERNEL);
+		if (!egm_entry) {
+			pci_dev_put(pdev);
+			return -ENOMEM;
+		}
+
+		egm_dev = setup_egm_chardev(egmphys, egmlength, egmpxm);
+		if (!egm_dev) {
+			kfree(egm_entry);
+			pci_dev_put(pdev);
+			return -EINVAL;
+		}
+
+		egm_entry->egm_dev = egm_dev;
+
+		ret = add_gpu(egm_entry->egm_dev, pdev);
+		if (ret) {
+			del_egm_chardev(egm_dev);
+			kfree(egm_entry);
+			goto free_dev;
+		}
+
+		list_add_tail(&egm_entry->list, &egm_chardevs);
+	}
+
+	return 0;
+
+free_dev:
+	/*
+	 * Reached only via goto from inside the loop where the pdev still
+	 * holds the reference from the last pci_get_device(). So drop it here.
+	 */
+	pci_dev_put(pdev);
+	nvgrace_egm_destroy_pci_devs();
+	return ret;
+}
+
+/*
+ * The char device minor is the EGM proximity domain number which is sparse
+ * and need not start at 0. Find the lowest proximity domain among the EGM
+ * devices so the minor range can be reserved starting there. Returns -ENODEV
+ * if the system has no EGM devices.
+ *
+ * EGM PXMs are assumed contiguous within [lowest, lowest + MAX_EGM_NODES),
+ * which holds for the supported Grace configurations.
+ */
+static int nvgrace_egm_lowest_pxm(u64 *pmin_pxm)
+{
+	struct pci_dev *pdev = NULL;
+	bool found = false;
+
+	for_each_pci_dev(pdev) {
+		u64 egmphys, egmlength, egmpxm;
+
+		if (has_egm_property(pdev, &egmpxm))
+			continue;
+
+		if (fetch_egm_property(pdev, &egmphys, &egmlength))
+			continue;
+
+		if (!found || egmpxm < *pmin_pxm) {
+			*pmin_pxm = egmpxm;
+			found = true;
+		}
+	}
+
+	if (!found)
+		return -ENODEV;
+
+	/*
+	 * The PXM is used verbatim as the minor (only MINORBITS wide). Validate
+	 * the window would not overflow MINORMASK and be truncated.
+	 */
+	if (*pmin_pxm + (MAX_EGM_NODES - 1) > MINORMASK) {
+		pr_err("nvgrace-egm: lowest EGM proximity domain %llu\n"
+		       "exceeds the minor range\n", *pmin_pxm);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
+static int __init nvgrace_egm_init(void)
+{
+	u64 min_pxm;
+	int ret;
+
+	/*
+	 * Each EGM region is exposed as /dev/egmX with the proximity domain X
+	 * as its minor number. Reserve MAX_EGM_NODES minors starting at
+	 * the lowest proximity domain so that every egmX device falls within
+	 * the reserved range. With no EGM devices present, load without
+	 * reserving any resources.
+	 */
+	ret = nvgrace_egm_lowest_pxm(&min_pxm);
+	if (ret == -ENODEV)
+		return 0;
+	else if (ret)
+		return ret;
+
+	ret = alloc_chrdev_region(&dev, min_pxm, MAX_EGM_NODES,
+				  NVGRACE_EGM_DEV_NAME);
+	if (ret < 0)
+		return ret;
+
+	class = class_create(NVGRACE_EGM_DEV_NAME);
+	if (IS_ERR(class)) {
+		unregister_chrdev_region(dev, MAX_EGM_NODES);
+		return PTR_ERR(class);
+	}
+
+	class->devnode = egm_devnode;
+	class->dev_groups = attr_groups;
+
+	ret = nvgrace_egm_create_pci_egm_devs();
+	if (ret)
+		goto cleanup_pci_devs;
+
+	return 0;
+
+cleanup_pci_devs:
+	nvgrace_egm_destroy_pci_devs();
+	class_destroy(class);
+	unregister_chrdev_region(dev, MAX_EGM_NODES);
+
+	return ret;
+}
+
+static void __exit nvgrace_egm_cleanup(void)
+{
+	/*
+	 * When no EGM devices were present, the module loaded inertly and
+	 * reserved nothing. @class stays NULL and there is nothing to undo.
+	 */
+	if (!class)
+		return;
+
+	nvgrace_egm_destroy_pci_devs();
+	class_destroy(class);
+	unregister_chrdev_region(dev, MAX_EGM_NODES);
+}
+
+module_init(nvgrace_egm_init);
+module_exit(nvgrace_egm_cleanup);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Ankit Agrawal <ankita@nvidia.com>");
+MODULE_DESCRIPTION("NVGRACE EGM - Module to support Extended GPU Memory on NVIDIA Grace Based systems");
-- 
2.34.1

[PATCH 2/4] platform/nvidia: Implement mmap and memory scrubbing for EGM chardev

[Ankit Agrawal]

Setup the file operations (open, close and mmap) for the EGM char device.

Memory scrubbing:
- The EGM region is invisible to the host Linux kernel and is not
  managed by the page allocator. So the driver is responsible for
  zeroing it before handing it to a VM.
- Clear the entire EGM region on the first open() call using memremap()
  and memset() in 1 GiB chunks with cond_resched() between iterations
  to avoid RCU stalls on large regions.
- The region is handed to a single VM at a time and must be scrubbed
  before each handover. Serialise openers with a mutex guarding open_count
  and refuse a second opener with -EBUSY while the region is still held.

mmap:
- Implement nvgrace_egm_mmap() using remap_pfn_range(). The EGM region
  is not part of the host EFI map and has no struct pages making
  remap_pfn_range() the correct interface for mapping it directly into
  QEMU's virtual address space.
- Validate that the requested range fits within the EGM region. Reject a
  vm_pgoff that already lies outside the region before it is shifted into
  a byte count so that the page offset cannot overflow and slip past the
  range check.

Suggested-by: Aniket Agashe <aniketa@nvidia.com>
Suggested-by: Vikram Sethi <vsethi@nvidia.com>
Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 drivers/platform/nvidia/egm.c | 125 ++++++++++++++++++++++++++++++++--
 1 file changed, 121 insertions(+), 4 deletions(-)

diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c
index a340c4fcbc7a..09ab00213de2 100644
--- a/drivers/platform/nvidia/egm.c
+++ b/drivers/platform/nvidia/egm.c
@@ -3,6 +3,9 @@
  * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
  */
 
+#include <linux/mutex.h>
+#include <linux/sched/signal.h>
+#include <linux/sizes.h>
 #include <linux/vfio_pci_core.h>
 
 #define NVGRACE_EGM_DEV_NAME "egm"
@@ -19,6 +22,10 @@ struct gpu_node {
 struct nvgrace_egm_dev {
 	struct device device;
 	struct cdev cdev;
+	/* serialises the first-open scrub */
+	struct mutex open_lock;
+	/* protected by open_lock */
+	unsigned int open_count;
 	phys_addr_t egmphys;
 	size_t egmlength;
 	u64 egmpxm;
@@ -39,21 +46,129 @@ static LIST_HEAD(egm_chardevs);
 
 static int nvgrace_egm_open(struct inode *inode, struct file *file)
 {
-	return 0;
+	struct nvgrace_egm_dev *egm_dev =
+		container_of(inode->i_cdev, struct nvgrace_egm_dev, cdev);
+	phys_addr_t phys;
+	size_t remaining, chunk_size;
+	void *chunk_addr;
+	int ret;
+
+	file->private_data = egm_dev;
+
+	/*
+	 * The EGM region is a physical carveout handed to a single VM at a
+	 * time and must be scrubbed before each handover. Take open_lock
+	 * killably around the scrub which can run for long time, so that
+	 * the waiters stay killable.
+	 */
+	if (mutex_lock_killable(&egm_dev->open_lock))
+		return -EINTR;
+
+	/*
+	 * Refuse a second opener while the region is still held. release()
+	 * runs only when the last reference to the struct file drops but an
+	 * mmap keeps that reference (and hence open_count) alive past
+	 * close for the lifetime of the VMA. So open_count returns to zero
+	 * only when every mapping is gone. A concurrent opener cannot be
+	 * handed the region: it cannot be re-scrubbed without destroying the
+	 * current consumer's live data.
+	 */
+	if (egm_dev->open_count) {
+		ret = -EBUSY;
+		goto unlock;
+	}
+
+	/*
+	 * nvgrace-egm module is responsible to manage the EGM memory as
+	 * the host kernel has no knowledge of it. Clear the region before
+	 * handing over to userspace.
+	 *
+	 * The EGM region can be very large (hundreds of GiB). So Map and zero
+	 * one chunk at a time rather than mapping the whole region at once.
+	 */
+	phys = egm_dev->egmphys;
+	remaining = egm_dev->egmlength;
+
+	while (remaining > 0) {
+		/*
+		 * The scrub holds the lock for a long time; let it be
+		 * SIGKILL'd.
+		 */
+		if (fatal_signal_pending(current)) {
+			ret = -EINTR;
+			goto unlock;
+		}
+
+		chunk_size = min(remaining, SZ_1G);
+
+		chunk_addr = memremap(phys, chunk_size, MEMREMAP_WB);
+		if (!chunk_addr) {
+			ret = -ENOMEM;
+			goto unlock;
+		}
+
+		memset(chunk_addr, 0, chunk_size);
+		memunmap(chunk_addr);
+		cond_resched();
+
+		phys += chunk_size;
+		remaining -= chunk_size;
+	}
+
+	/*
+	 * Mark the device open only after the scrub completes so that a
+	 * concurrent opener cannot observe a non-zero count and proceed
+	 * before the region has been cleared.
+	 */
+	egm_dev->open_count = 1;
+	ret = 0;
+
+unlock:
+	mutex_unlock(&egm_dev->open_lock);
+	return ret;
 }
 
 static int nvgrace_egm_release(struct inode *inode, struct file *file)
 {
+	struct nvgrace_egm_dev *egm_dev =
+		container_of(inode->i_cdev, struct nvgrace_egm_dev, cdev);
+
+	guard(mutex)(&egm_dev->open_lock);
+
+	if (!--egm_dev->open_count)
+		file->private_data = NULL;
+
 	return 0;
 }
 
 static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma)
 {
+	struct nvgrace_egm_dev *egm_dev = file->private_data;
+	u64 req_len, pgoff, end;
+	unsigned long start_pfn, num_pages;
+
+	pgoff = vma->vm_pgoff;
+	num_pages = egm_dev->egmlength >> PAGE_SHIFT;
+
+	/* Reject a page offset that already lies outside the EGM region. */
+	if (pgoff >= num_pages)
+		return -EINVAL;
+
+	if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
+	    check_add_overflow(PHYS_PFN(egm_dev->egmphys), pgoff, &start_pfn) ||
+	    check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
+		return -EOVERFLOW;
+
+	if (end > egm_dev->egmlength)
+		return -EINVAL;
+
 	/*
-	 * Mapping the EGM region into userspace is implemented by a later
-	 * patch. Until then refuse the mmap.
+	 * EGM memory is invisible to the host kernel and is not managed
+	 * by it. Map the usermode VMA to the EGM region.
 	 */
-	return -EOPNOTSUPP;
+	return remap_pfn_range(vma, vma->vm_start,
+			       start_pfn, req_len,
+			       vma->vm_page_prot);
 }
 
 static const struct file_operations file_ops = {
@@ -122,6 +237,7 @@ static void egm_chardev_release(struct device *dev)
 	struct nvgrace_egm_dev *egm_chardev = container_of(dev, struct nvgrace_egm_dev, device);
 
 	remove_gpus(egm_chardev);
+	mutex_destroy(&egm_chardev->open_lock);
 	kfree(egm_chardev);
 }
 
@@ -155,6 +271,7 @@ static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
 	egm_chardev->egmphys = egmphys;
 	egm_chardev->egmlength = egmlength;
 	egm_chardev->egmpxm = egmpxm;
+	mutex_init(&egm_chardev->open_lock);
 	INIT_LIST_HEAD(&egm_chardev->gpus);
 
 	egm_chardev->device.devt = MKDEV(MAJOR(dev), egm_chardev->egmpxm);
-- 
2.34.1

[PATCH 3/4] platform/nvidia: Handle retired ECC pages and expose via ioctl

[Ankit Agrawal]

EGM regions may contain pages retired due to ECC/poison errors that
are known to the system firmware beforehand. SBIOS records the list
in a dedicated carveout region whose SPA is advertised via the ACPI
DSD property "nvidia,egm-retired-pages-data-base". The carveout begins
with a u64 count followed by that many u64 system physical addresses
each the base SPA of a retired 64K page within the EGM region.

The EGM region is linear, so the SPA - EGM_base calculation can be
used to determine the offset within the EGM region. Moreover, the region
is assigned linearly to the VM. So offset within EGM region on the host
would be same as on the VM. Hence the retired pages offset is calculated
and stored by the nvgrace-egm. An xarray is used to track the retired
page offsets. These are exposed to the userspace (Qemu) which may use the
information to skip the retired pages while forming the memory mapping
for the VM.

Note that a retired page SPA SBIOS entry is for 64k size, but the
xarray list tracks offsets in PAGE_SIZE. So SZ_64K/PAGE_SIZE entries
are added to the xarray to ensure comaptibility with 4k OS.

Patch implements the following:
Retired page tracking:
- Map the carveout region with memremap() and validate the count.
- convert each physical address to an in-EGM-region offset and
  populate a per-EGM-device hashtable (keyed by that offset).

Userspace ioctl:
- Introduce EGM_RETIRED_PAGES_LIST ioctl and add a new UAPI header
  Userspace (QEMU) calls this ioctl to retrieve the list of retired
  page offsets. QEMU then communicates the list to the VM so the
  guest can take appropriate action.

Update MAINTAINERS to list include/uapi/linux/egm.h

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 MAINTAINERS                                   |   1 +
 drivers/platform/nvidia/egm.c                 | 211 +++++++++++++++++-
 include/uapi/linux/egm.h                      |  28 +++
 4 files changed, 231 insertions(+), 10 deletions(-)
 create mode 100644 include/uapi/linux/egm.h

diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index 3f0ef1e27eb0..94c9ca4a89ff 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -306,6 +306,7 @@ Code  Seq#    Include File                                             Comments
 'v'   20-27  arch/powerpc/include/uapi/asm/vas-api.h                   VAS API
 'v'   C0-FF  linux/meye.h                                              conflict!
 'w'   all                                                              CERN SCI driver
+'x'   00-1F  include/uapi/linux/egm.h                                  NVIDIA nvgrace-egm driver
 'y'   00-1F                                                            packet based user level communications
                                                                        <mailto:zapman@interlan.net>
 'z'   00-3F                                                            CAN bus card conflict!
diff --git a/MAINTAINERS b/MAINTAINERS
index 2edf903a7746..e77c22a9e598 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19348,6 +19348,7 @@ NVIDIA GRACE EGM PLATFORM DRIVER
 M:	Ankit Agrawal <ankita@nvidia.com>
 S:	Supported
 F:	drivers/platform/nvidia/
+F:	include/uapi/linux/egm.h
 
 NVIDIA VRS RTC DRIVER
 M:	Shubhi Garg <shgarg@nvidia.com>
diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c
index 09ab00213de2..df963caedaa8 100644
--- a/drivers/platform/nvidia/egm.c
+++ b/drivers/platform/nvidia/egm.c
@@ -7,10 +7,28 @@
 #include <linux/sched/signal.h>
 #include <linux/sizes.h>
 #include <linux/vfio_pci_core.h>
+#include <linux/xarray.h>
+#include <uapi/linux/egm.h>
 
 #define NVGRACE_EGM_DEV_NAME "egm"
 #define MAX_EGM_NODES 4
 
+/*
+ * Presence marker stored in nvgrace_egm_dev.retired_pages. Only the
+ * page offset within the region carries information and the value
+ * just needs to be a non-NULL xa entry.
+ */
+#define EGM_RETIRED_MARK xa_mk_value(1)
+
+/*
+ * The structure match the format of the retired pages information populated
+ * by the system firmware.
+ */
+struct fw_egm_retired_pages {
+	u64 num_retired_pages;
+	__u64 retired_page_addr[4096];
+};
+
 static dev_t dev;
 static struct class *class;
 
@@ -26,8 +44,14 @@ struct nvgrace_egm_dev {
 	struct mutex open_lock;
 	/* protected by open_lock */
 	unsigned int open_count;
+	/*
+	 * Set of retired page offsets within the
+	 * EGM region and keyed by page index.
+	 */
+	struct xarray retired_pages;
 	phys_addr_t egmphys;
 	size_t egmlength;
+	phys_addr_t retiredpagesphys;
 	u64 egmpxm;
 	struct list_head gpus;
 };
@@ -44,6 +68,9 @@ struct nvgrace_egm_dev_entry {
  */
 static LIST_HEAD(egm_chardevs);
 
+static void cleanup_retired_pages(struct nvgrace_egm_dev *egm_dev);
+static int nvgrace_egm_fetch_retired_pages(struct nvgrace_egm_dev *egm_dev);
+
 static int nvgrace_egm_open(struct inode *inode, struct file *file)
 {
 	struct nvgrace_egm_dev *egm_dev =
@@ -171,11 +198,68 @@ static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma)
 			       vma->vm_page_prot);
 }
 
+static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+	unsigned long minsz = offsetofend(struct egm_retired_pages_list, count);
+	struct egm_retired_pages_list info;
+	void __user *uarg = (void __user *)arg;
+	struct nvgrace_egm_dev *egm_dev = file->private_data;
+
+	if (copy_from_user(&info, uarg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz || !egm_dev)
+		return -EINVAL;
+
+	switch (cmd) {
+	case EGM_RETIRED_PAGES_LIST: {
+		unsigned long retired_page_struct_size = sizeof(struct egm_retired_pages_info);
+		struct egm_retired_pages_info tmp;
+		unsigned long page_index;
+		void *entry;
+		int count = 0, fill = 0;
+
+		xa_for_each(&egm_dev->retired_pages, page_index, entry)
+			count++;
+
+		if (info.argsz < (minsz + count * retired_page_struct_size)) {
+			info.argsz = minsz + count * retired_page_struct_size;
+			info.count = 0;
+			goto done;
+		}
+
+		xa_for_each(&egm_dev->retired_pages, page_index, entry) {
+			if (fill >= count)
+				break;
+
+			tmp.offset = (u64)page_index << PAGE_SHIFT;
+			tmp.size = PAGE_SIZE;
+
+			if (copy_to_user((u8 __user *)uarg + minsz +
+					 fill * retired_page_struct_size,
+					 &tmp, retired_page_struct_size))
+				return -EFAULT;
+
+			fill++;
+		}
+
+		info.count = fill;
+		break;
+	}
+	default:
+		return -EINVAL;
+	}
+
+done:
+	return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0;
+}
+
 static const struct file_operations file_ops = {
 	.owner = THIS_MODULE,
 	.open = nvgrace_egm_open,
 	.release = nvgrace_egm_release,
 	.mmap = nvgrace_egm_mmap,
+	.unlocked_ioctl = nvgrace_egm_ioctl,
 };
 
 static int nvgrace_egm_create_gpu_links(struct nvgrace_egm_dev *egm_dev,
@@ -236,13 +320,15 @@ static void egm_chardev_release(struct device *dev)
 {
 	struct nvgrace_egm_dev *egm_chardev = container_of(dev, struct nvgrace_egm_dev, device);
 
+	cleanup_retired_pages(egm_chardev);
 	remove_gpus(egm_chardev);
 	mutex_destroy(&egm_chardev->open_lock);
 	kfree(egm_chardev);
 }
 
 static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
-						 u64 egmpxm)
+						 u64 egmpxm,
+						 u64 retiredpagesphys)
 {
 	struct nvgrace_egm_dev *egm_chardev;
 	unsigned int baseminor = MINOR(dev);
@@ -271,7 +357,9 @@ static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
 	egm_chardev->egmphys = egmphys;
 	egm_chardev->egmlength = egmlength;
 	egm_chardev->egmpxm = egmpxm;
+	egm_chardev->retiredpagesphys = retiredpagesphys;
 	mutex_init(&egm_chardev->open_lock);
+	xa_init(&egm_chardev->retired_pages);
 	INIT_LIST_HEAD(&egm_chardev->gpus);
 
 	egm_chardev->device.devt = MKDEV(MAJOR(dev), egm_chardev->egmpxm);
@@ -284,6 +372,10 @@ static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
 	if (ret)
 		goto error_exit;
 
+	ret = nvgrace_egm_fetch_retired_pages(egm_chardev);
+	if (ret)
+		goto error_exit;
+
 	ret = cdev_device_add(&egm_chardev->cdev, &egm_chardev->device);
 	if (ret)
 		goto error_exit;
@@ -302,6 +394,85 @@ static void del_egm_chardev(struct nvgrace_egm_dev *egm_chardev)
 	put_device(&egm_chardev->device);
 }
 
+static void cleanup_retired_pages(struct nvgrace_egm_dev *egm_dev)
+{
+	/* Entries are value marks, so xa_destroy() frees the table itself. */
+	xa_destroy(&egm_dev->retired_pages);
+}
+
+static int nvgrace_egm_fetch_retired_pages(struct nvgrace_egm_dev *egm_dev)
+{
+	struct fw_egm_retired_pages *egm_retired;
+	u64 count;
+	int index, ret = 0;
+
+	/* No retired-pages region was advertised; nothing to populate. */
+	if (!egm_dev->retiredpagesphys)
+		return 0;
+
+	egm_retired = memremap(egm_dev->retiredpagesphys,
+			       sizeof(*egm_retired), MEMREMAP_WB);
+	if (!egm_retired)
+		return -ENOMEM;
+
+	count = egm_retired->num_retired_pages;
+	if (count > ARRAY_SIZE(egm_retired->retired_page_addr)) {
+		memunmap(egm_retired);
+		return -EINVAL;
+	}
+
+	for (index = 0; index < count && !ret; index++) {
+		phys_addr_t base = egm_retired->retired_page_addr[index];
+		int sub;
+
+		/*
+		 * Entries come from SBIOS. Skip any base outside the EGM
+		 * region [egmphys, egmphys + egmlength).
+		 */
+		if (base < egm_dev->egmphys ||
+		    base >= egm_dev->egmphys + egm_dev->egmlength) {
+			dev_warn_ratelimited(&egm_dev->device,
+					     "Ignoring retired page %pa outside EGM region\n",
+					     &base);
+			continue;
+		}
+
+		/*
+		 * Since the EGM is linearly mapped, the offset in the
+		 * carveout is the same offset in the VM system memory.
+		 *
+		 * Calculate the offset to communicate to the usermode
+		 * apps.
+		 *
+		 * The retired page entry represent a retired region of
+		 * size 64K. So on a 4K kernel, each entry spans 0x10
+		 * 4K sub-pages; add one entry per sub-page so that any
+		 * lookup at 4K granularity hits the right retired range.
+		 */
+		for (sub = 0; sub < SZ_64K / PAGE_SIZE; sub++) {
+			unsigned long offset = base +
+				(phys_addr_t)sub * PAGE_SIZE - egm_dev->egmphys;
+			void *old;
+
+			old = xa_store(&egm_dev->retired_pages,
+				       offset >> PAGE_SHIFT, EGM_RETIRED_MARK,
+				       GFP_KERNEL);
+			if (xa_is_err(old)) {
+				ret = xa_err(old);
+				break;
+			}
+		}
+	}
+
+	memunmap(egm_retired);
+
+	/*
+	 * On failure the partially populated table is freed by
+	 * egm_chardev_release() via the caller's error path.
+	 */
+	return ret;
+}
+
 static char *egm_devnode(const struct device *device, umode_t *mode)
 {
 	if (mode)
@@ -347,22 +518,39 @@ static int has_egm_property(struct pci_dev *pdev, u64 *pegmpxm)
 }
 
 static int fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys,
-			      u64 *pegmlength)
+			      u64 *pegmlength, u64 *pretiredpagesphys)
 {
 	int ret;
 
 	/*
-	 * The memory information is present in the system ACPI tables as DSD
-	 * properties nvidia,egm-base-pa and nvidia,egm-size.
+	 * The EGM memory information is present in the system ACPI tables
+	 * as DSD properties nvidia,egm-base-pa and nvidia,egm-size.
 	 */
 	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size",
 				       pegmlength);
 	if (ret)
-		return ret;
+		goto error_exit;
 
 	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa",
 				       pegmphys);
+	if (ret)
+		goto error_exit;
+
+	/*
+	 * SBIOS puts the list of retired pages on a region exposed as
+	 * "nvidia,egm-retired-pages-data-base". Older firmware may not expose
+	 * it. Treat an absent or zero property as 0 retired pages so the
+	 * EGM device is still created (with an empty retired-pages table).
+	 */
+	ret = device_property_read_u64(&pdev->dev,
+				       "nvidia,egm-retired-pages-data-base",
+				       pretiredpagesphys);
+	if (ret) {
+		*pretiredpagesphys = 0;
+		ret = 0;
+	}
 
+error_exit:
 	return ret;
 }
 
@@ -409,12 +597,13 @@ static int nvgrace_egm_create_pci_egm_devs(void)
 	int ret;
 
 	for_each_pci_dev(pdev) {
-		u64 egmphys, egmlength, egmpxm;
+		u64 egmphys, egmlength, egmpxm, retiredpagesphys;
 
 		if (has_egm_property(pdev, &egmpxm))
 			continue;
 
-		ret = fetch_egm_property(pdev, &egmphys, &egmlength);
+		ret = fetch_egm_property(pdev, &egmphys, &egmlength,
+					 &retiredpagesphys);
 		if (ret)
 			continue;
 
@@ -432,7 +621,8 @@ static int nvgrace_egm_create_pci_egm_devs(void)
 			return -ENOMEM;
 		}
 
-		egm_dev = setup_egm_chardev(egmphys, egmlength, egmpxm);
+		egm_dev = setup_egm_chardev(egmphys, egmlength, egmpxm,
+					    retiredpagesphys);
 		if (!egm_dev) {
 			kfree(egm_entry);
 			pci_dev_put(pdev);
@@ -478,12 +668,13 @@ static int nvgrace_egm_lowest_pxm(u64 *pmin_pxm)
 	bool found = false;
 
 	for_each_pci_dev(pdev) {
-		u64 egmphys, egmlength, egmpxm;
+		u64 egmphys, egmlength, egmpxm, retiredpagesphys;
 
 		if (has_egm_property(pdev, &egmpxm))
 			continue;
 
-		if (fetch_egm_property(pdev, &egmphys, &egmlength))
+		if (fetch_egm_property(pdev, &egmphys, &egmlength,
+				       &retiredpagesphys))
 			continue;
 
 		if (!found || egmpxm < *pmin_pxm) {
diff --git a/include/uapi/linux/egm.h b/include/uapi/linux/egm.h
new file mode 100644
index 000000000000..ee165056c2f6
--- /dev/null
+++ b/include/uapi/linux/egm.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
+ */
+
+#ifndef _UAPI_LINUX_EGM_H
+#define _UAPI_LINUX_EGM_H
+
+#include <linux/types.h>
+
+#define EGM_TYPE ('x')
+
+struct egm_retired_pages_info {
+	__aligned_u64 offset;
+	__aligned_u64 size;
+};
+
+struct egm_retired_pages_list {
+	__u32 argsz;
+	/* out */
+	__u32 count;
+	/* out */
+	struct egm_retired_pages_info retired_pages[];
+};
+
+#define EGM_RETIRED_PAGES_LIST     _IOWR(EGM_TYPE, 0x01, struct egm_retired_pages_list)
+
+#endif /* _UAPI_LINUX_EGM_H */
-- 
2.34.1

[PATCH 4/4] platform/nvidia: Register EGM PFNMAP range with memory_failure

[Ankit Agrawal]

EGM carveout memory is mapped directly into userspace (QEMU) and is not
added to the kernel. It is not managed by the kernel page allocator and
has no struct pages. The module can thus utilize the Linux memory manager's
memory_failure mechanism for regions with no struct pages. The Linux MM
code exposes register/unregister APIs allowing modules to register such
memory regions for memory_failure handling. On the occurrence of the
memory_failure, Linux MM calls the registered function if the PFN is
within the range.

These dynamically occurring memory_failure pages are tracked alongside
the statically populated list by the system firmware in the xarray. So
the userspace can get both the information through the common
EGM_RETIRED_PAGES_LIST ioctl.

Register the EGM PFN range with the memory_failure infrastructure on
first open() and unregister it on the last close(). Provide a
pfn_to_vma_pgoff callback that:
- Validates that the reported PFN falls within the EGM region and the
  current VMA.
- Converts the PFN to a file offset within the EGM region.
- Records the poisoned offset in the existing retired-pages hashtable
  so it is reported to userspace via EGM_RETIRED_PAGES_LIST alongside
  statically configured retired pages from SBIOS.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
---
 drivers/platform/nvidia/egm.c | 123 +++++++++++++++++++++++++++++++---
 1 file changed, 115 insertions(+), 8 deletions(-)

diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c
index df963caedaa8..39c4be768f75 100644
--- a/drivers/platform/nvidia/egm.c
+++ b/drivers/platform/nvidia/egm.c
@@ -3,6 +3,7 @@
  * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
  */
 
+#include <linux/memory-failure.h>
 #include <linux/mutex.h>
 #include <linux/sched/signal.h>
 #include <linux/sizes.h>
@@ -40,7 +41,7 @@ struct gpu_node {
 struct nvgrace_egm_dev {
 	struct device device;
 	struct cdev cdev;
-	/* serialises the first-open scrub */
+	/* serialises the first-open scrub + PFN registration */
 	struct mutex open_lock;
 	/* protected by open_lock */
 	unsigned int open_count;
@@ -49,6 +50,7 @@ struct nvgrace_egm_dev {
 	 * EGM region and keyed by page index.
 	 */
 	struct xarray retired_pages;
+	struct pfn_address_space pfn_address_space;
 	phys_addr_t egmphys;
 	size_t egmlength;
 	phys_addr_t retiredpagesphys;
@@ -71,6 +73,104 @@ static LIST_HEAD(egm_chardevs);
 static void cleanup_retired_pages(struct nvgrace_egm_dev *egm_dev);
 static int nvgrace_egm_fetch_retired_pages(struct nvgrace_egm_dev *egm_dev);
 
+static int pfn_memregion_offset(struct nvgrace_egm_dev *egm_dev,
+				unsigned long pfn,
+				pgoff_t *pfn_offset_in_region)
+{
+	unsigned long start_pfn, num_pages;
+
+	start_pfn = PHYS_PFN(egm_dev->egmphys);
+	num_pages = egm_dev->egmlength >> PAGE_SHIFT;
+
+	if (pfn < start_pfn || pfn >= start_pfn + num_pages)
+		return -EFAULT;
+
+	*pfn_offset_in_region = pfn - start_pfn;
+
+	return 0;
+}
+
+static int track_ecc_offset(struct nvgrace_egm_dev *egm_dev,
+			    unsigned long mem_offset)
+{
+	void *old;
+
+	/*
+	 * This runs from the pfn_to_vma_pgoff callback, which the
+	 * memory_failure path invokes under rcu_read_lock() (see
+	 * collect_procs_pfn()). Sleeping is not allowed in that atomic
+	 * context and so use GFP_ATOMIC rather than GFP_KERNEL.
+	 *
+	 * Storing by page index makes a repeated report at the same offset a
+	 * harmless overwrite, so no explicit de-duplication is needed.
+	 */
+	old = xa_store(&egm_dev->retired_pages, mem_offset >> PAGE_SHIFT,
+		       EGM_RETIRED_MARK, GFP_ATOMIC);
+	if (xa_is_err(old))
+		return xa_err(old);
+
+	return 0;
+}
+
+static int nvgrace_egm_pfn_to_vma_pgoff(struct vm_area_struct *vma,
+					unsigned long pfn,
+					pgoff_t *pgoff)
+{
+	struct nvgrace_egm_dev *egm_dev = vma->vm_file->private_data;
+	pgoff_t vma_offset_in_region = vma->vm_pgoff;
+	pgoff_t pfn_offset_in_region;
+	int ret;
+
+	ret = pfn_memregion_offset(egm_dev, pfn, &pfn_offset_in_region);
+	if (ret)
+		return ret;
+
+	/* Ensure PFN is not before VMA's start within the region */
+	if (pfn_offset_in_region < vma_offset_in_region)
+		return -EFAULT;
+
+	/* Ensure PFN is not past the VMA's end within the region */
+	if (pfn_offset_in_region >= vma_offset_in_region + vma_pages(vma))
+		return -EFAULT;
+
+	/*
+	 * The VMA's vm_pgoff is the page offset of its start within the EGM
+	 * region, so the region offset of the pfn is already the pgoff to
+	 * report.
+	 */
+	*pgoff = pfn_offset_in_region;
+
+	/*
+	 * Record the poisoned offset for reporting via
+	 * EGM_RETIRED_PAGES_LIST. This bookkeeping is best-effort: its
+	 * failure must not be reported as "vma does not map pfn", or the
+	 * owning process would not be signalled for the poisoned page.
+	 */
+	track_ecc_offset(egm_dev, *pgoff << PAGE_SHIFT);
+
+	return 0;
+}
+
+static int
+nvgrace_egm_register_pfn_range(struct inode *inode,
+			       struct nvgrace_egm_dev *egm_dev)
+{
+	unsigned long pfn, nr_pages;
+	int ret;
+
+	pfn = PHYS_PFN(egm_dev->egmphys);
+	nr_pages = egm_dev->egmlength >> PAGE_SHIFT;
+
+	egm_dev->pfn_address_space.node.start = pfn;
+	egm_dev->pfn_address_space.node.last = pfn + nr_pages - 1;
+	egm_dev->pfn_address_space.mapping = inode->i_mapping;
+	egm_dev->pfn_address_space.pfn_to_vma_pgoff = nvgrace_egm_pfn_to_vma_pgoff;
+
+	ret = register_pfn_address_space(&egm_dev->pfn_address_space);
+
+	return ret;
+}
+
 static int nvgrace_egm_open(struct inode *inode, struct file *file)
 {
 	struct nvgrace_egm_dev *egm_dev =
@@ -84,9 +184,9 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file)
 
 	/*
 	 * The EGM region is a physical carveout handed to a single VM at a
-	 * time and must be scrubbed before each handover. Take open_lock
-	 * killably around the scrub which can run for long time, so that
-	 * the waiters stay killable.
+	 * time and must be scrubbed and registered with memory_failure before
+	 * each handover. Take open_lock killably around the scrub which can
+	 * run for long time, so that the waiters stay killable.
 	 */
 	if (mutex_lock_killable(&egm_dev->open_lock))
 		return -EINTR;
@@ -142,10 +242,15 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file)
 		remaining -= chunk_size;
 	}
 
+	ret = nvgrace_egm_register_pfn_range(inode, egm_dev);
+	if (ret && ret != -EOPNOTSUPP)
+		goto unlock;
+
 	/*
-	 * Mark the device open only after the scrub completes so that a
-	 * concurrent opener cannot observe a non-zero count and proceed
-	 * before the region has been cleared.
+	 * Only mark the device open once the scrub and registration have
+	 * succeeded. On failure open_count stays 0, so this opener returns
+	 * an error (release() never runs for it) and the next opener
+	 * retries the scrub and registration from a clean state.
 	 */
 	egm_dev->open_count = 1;
 	ret = 0;
@@ -162,8 +267,10 @@ static int nvgrace_egm_release(struct inode *inode, struct file *file)
 
 	guard(mutex)(&egm_dev->open_lock);
 
-	if (!--egm_dev->open_count)
+	if (!--egm_dev->open_count) {
+		unregister_pfn_address_space(&egm_dev->pfn_address_space);
 		file->private_data = NULL;
+	}
 
 	return 0;
 }
-- 
2.34.1

Re: [PATCH 1/4] platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM regions

[Armin Wolf]

Am 02.07.26 um 21:25 schrieb Ankit Agrawal:
> The Extended GPU Memory (EGM) feature enables the GPU to access system
> memory allocations within and across nodes through a high-bandwidth path
> on Grace-based systems. The GPU can utilize system memory on the same
> socket, a different socket or a different node in a multi-node setup.
>
> When EGM mode is enabled through SBIOS, host system memory is partitioned
> into two parts: a Hypervisor region for the host OS and a
> Hypervisor-Invisible (HI) region for the VM. Only the hypervisor region
> is part of the host EFI map and visible to the host OS on bootup. The
> HI partition is interchangeably called the EGM region. Its base SPA and
> size are exposed through ACPI DSDT properties.
>
> Whilst the EGM region is accessible on the host, it is not
> added/hot-plugged to the kernel. The HI region is assigned to a VM
> by mapping the QEMU VMA to the SPA using remap_pfn_range(). So
> it looks like the following:
>
>    |---- Sysmem ----|                  VM view
>    |                |
>    |IPA <-> SPA map |
>    |                |
>    |--- HI / EGM ---|-- Host Mem --|   Host view
>
> Introduce the nvgrace-egm platform driver module and implement full EGM
> region discovery, char device exposure and sysfs metadata.
>
> Build infrastructure:
> - Kconfig, Makefile entries, MAINTAINERS entry.
> - An 'egm' device class and a range of char device minor numbers.
>    MAX_EGM_NODES is set to 4 to matches the largest 4-socket Grace
>    configuration.
>
> Discovery:
> - Walk all PCI devices at module init time. For each one advertising
>    "nvidia,egm-pxm", read the EGM region properties (base SPA, size and
>    PXM domain) from the ACPI DSD table via the GPU device handle, using
>    "nvidia,egm-base-pa" and "nvidia,egm-size" properties.
> - Maintain a module-level list (egm_chardevs) of discovered regions,
>    de-duplicating by PXM so each socket gets exactly one EGM device
>    regardless of how many GPUs it has (2 GPUs per EGM device in GB200).

I assume that the PCI device itself is being handled by the GPU driver, right?
In this case i suggest that you do the discovery inside the GPU driver during probing,
otherwise you might run intro problems once PCI hotplug comes into play.

Thanks,
Armin Wolf

>
> Char device creation:
> - Each EGM region is exposed as /dev/egmX, where X is the PXM number.
>    Embed struct device and struct cdev inside nvgrace_egm_dev so the
>    device lifecycle follows the standard kernel device model.
> - Implement setup_egm_chardev() and del_egm_chardev() for setup and
>    teardown. The devnode callback sets device permissions to 0600.
> - Stub file operations (open, release and mmap) are registered.
>
> GPU tracking and sysfs:
> - Track which PCI devices are associated with each EGM region in a
>    per-device GPU list (struct gpu_node).
> - Expose sysfs symlinks from each EGM device to its associated GPU PCI
>    devices to allow the userspace to determine GPU-to-socket affinity and
>    replicate the host EGM topology in the VM. Helps admin understand
>    GPU association.
> - Expose the EGM region size via an egm_size sysfs attribute so QEMU
>    can determine the memory-backend-file size. On a 2-socket Grace
>    Blackwell setup this appears as:
>      Socket0: /sys/devices/virtual/egm/egm4/egm_size
>      Socket1: /sys/devices/virtual/egm/egm5/egm_size
>
> Introduce nvgrace_egm_create_pci_egm_devs() and its counterpart
> nvgrace_egm_destroy_pci_devs() that are called from module init/exit.
>
> Suggested-by: Alex Williamson <alex@shazbot.org>
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Suggested-by: Aniket Agashe <aniketa@nvidia.com>
> Suggested-by: Matthew R. Ochs <mochs@nvidia.com>
> Assisted-by: Claude:claude-opus-4.8
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> ---
>   MAINTAINERS                      |   5 +
>   drivers/platform/Kconfig         |   1 +
>   drivers/platform/Makefile        |   1 +
>   drivers/platform/nvidia/Kconfig  |  10 +
>   drivers/platform/nvidia/Makefile |   3 +
>   drivers/platform/nvidia/egm.c    | 459 +++++++++++++++++++++++++++++++
>   6 files changed, 479 insertions(+)
>   create mode 100644 drivers/platform/nvidia/Kconfig
>   create mode 100644 drivers/platform/nvidia/Makefile
>   create mode 100644 drivers/platform/nvidia/egm.c
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 15011f5752a9..2edf903a7746 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -19344,6 +19344,11 @@ L:	linux-acpi@vger.kernel.org
>   S:	Maintained
>   F:	drivers/acpi/apei/ghes-nvidia.c
>   
> +NVIDIA GRACE EGM PLATFORM DRIVER
> +M:	Ankit Agrawal <ankita@nvidia.com>
> +S:	Supported
> +F:	drivers/platform/nvidia/
> +
>   NVIDIA VRS RTC DRIVER
>   M:	Shubhi Garg <shgarg@nvidia.com>
>   L:	linux-tegra@vger.kernel.org
> diff --git a/drivers/platform/Kconfig b/drivers/platform/Kconfig
> index 312788f249c9..21f2dc901301 100644
> --- a/drivers/platform/Kconfig
> +++ b/drivers/platform/Kconfig
> @@ -22,3 +22,4 @@ source "drivers/platform/arm64/Kconfig"
>   source "drivers/platform/raspberrypi/Kconfig"
>   
>   source "drivers/platform/wmi/Kconfig"
> +source "drivers/platform/nvidia/Kconfig"
> diff --git a/drivers/platform/Makefile b/drivers/platform/Makefile
> index fa322e7f8716..96de8fa26ed5 100644
> --- a/drivers/platform/Makefile
> +++ b/drivers/platform/Makefile
> @@ -15,3 +15,4 @@ obj-$(CONFIG_SURFACE_PLATFORMS)	+= surface/
>   obj-$(CONFIG_ARM64_PLATFORM_DEVICES)	+= arm64/
>   obj-$(CONFIG_BCM2835_VCHIQ)	+= raspberrypi/
>   obj-$(CONFIG_ACPI_WMI)		+= wmi/
> +obj-$(CONFIG_NVGRACE_EGM)	+= nvidia/
> diff --git a/drivers/platform/nvidia/Kconfig b/drivers/platform/nvidia/Kconfig
> new file mode 100644
> index 000000000000..8554a58b1141
> --- /dev/null
> +++ b/drivers/platform/nvidia/Kconfig
> @@ -0,0 +1,10 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +config NVGRACE_EGM
> +	tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip"
> +	depends on ARM64 || (COMPILE_TEST && 64BIT)
> +	help
> +	  Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace
> +	  based chips required to avail the CPU memory as additional
> +	  cross-node/cross-socket memory for GPU using KVM/qemu.
> +
> +	  If you don't know what to do here, say N.
> diff --git a/drivers/platform/nvidia/Makefile b/drivers/platform/nvidia/Makefile
> new file mode 100644
> index 000000000000..a89f820908fb
> --- /dev/null
> +++ b/drivers/platform/nvidia/Makefile
> @@ -0,0 +1,3 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +obj-$(CONFIG_NVGRACE_EGM) += nvgrace-egm.o
> +nvgrace-egm-y := egm.o
> diff --git a/drivers/platform/nvidia/egm.c b/drivers/platform/nvidia/egm.c
> new file mode 100644
> index 000000000000..a340c4fcbc7a
> --- /dev/null
> +++ b/drivers/platform/nvidia/egm.c
> @@ -0,0 +1,459 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved
> + */
> +
> +#include <linux/vfio_pci_core.h>
> +
> +#define NVGRACE_EGM_DEV_NAME "egm"
> +#define MAX_EGM_NODES 4
> +
> +static dev_t dev;
> +static struct class *class;
> +
> +struct gpu_node {
> +	struct list_head list;
> +	struct pci_dev *pdev;
> +};
> +
> +struct nvgrace_egm_dev {
> +	struct device device;
> +	struct cdev cdev;
> +	phys_addr_t egmphys;
> +	size_t egmlength;
> +	u64 egmpxm;
> +	struct list_head gpus;
> +};
> +
> +struct nvgrace_egm_dev_entry {
> +	struct list_head list;
> +	struct nvgrace_egm_dev *egm_dev;
> +};
> +
> +/*
> + * Track egm device lists. Note that there is one device per socket.
> + * All the GPUs belonging to the same sockets are associated with
> + * the EGM device for that socket.
> + */
> +static LIST_HEAD(egm_chardevs);
> +
> +static int nvgrace_egm_open(struct inode *inode, struct file *file)
> +{
> +	return 0;
> +}
> +
> +static int nvgrace_egm_release(struct inode *inode, struct file *file)
> +{
> +	return 0;
> +}
> +
> +static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma)
> +{
> +	/*
> +	 * Mapping the EGM region into userspace is implemented by a later
> +	 * patch. Until then refuse the mmap.
> +	 */
> +	return -EOPNOTSUPP;
> +}
> +
> +static const struct file_operations file_ops = {
> +	.owner = THIS_MODULE,
> +	.open = nvgrace_egm_open,
> +	.release = nvgrace_egm_release,
> +	.mmap = nvgrace_egm_mmap,
> +};
> +
> +static int nvgrace_egm_create_gpu_links(struct nvgrace_egm_dev *egm_dev,
> +					struct pci_dev *pdev)
> +{
> +	int ret;
> +
> +	ret = sysfs_create_link(&egm_dev->device.kobj,
> +				&pdev->dev.kobj,
> +				dev_name(&pdev->dev));
> +
> +	if (ret && ret != -EEXIST)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static void remove_egm_symlinks(struct nvgrace_egm_dev *egm_dev,
> +				struct pci_dev *pdev)
> +{
> +	sysfs_remove_link(&egm_dev->device.kobj, dev_name(&pdev->dev));
> +}
> +
> +static int add_gpu(struct nvgrace_egm_dev *egm_dev, struct pci_dev *pdev)
> +{
> +	struct gpu_node *node;
> +
> +	node = kzalloc_obj(*node, GFP_KERNEL);
> +	if (!node)
> +		return -ENOMEM;
> +
> +	/*
> +	 * The pdev is stored for the lifetime of the EGM device but it is
> +	 * obtained from a for_each_pci_dev() walk that drops its reference as
> +	 * it advances. Take a reference here so the stored pointer stays valid
> +	 * even if the GPU is later unbound; remove_gpus() drops it.
> +	 */
> +	node->pdev = pci_dev_get(pdev);
> +
> +	list_add_tail(&node->list, &egm_dev->gpus);
> +
> +	return nvgrace_egm_create_gpu_links(egm_dev, pdev);
> +}
> +
> +static void remove_gpus(struct nvgrace_egm_dev *egm_dev)
> +{
> +	struct gpu_node *node, *tmp;
> +
> +	list_for_each_entry_safe(node, tmp, &egm_dev->gpus, list) {
> +		remove_egm_symlinks(egm_dev, node->pdev);
> +		list_del(&node->list);
> +		pci_dev_put(node->pdev);
> +		kfree(node);
> +	}
> +}
> +
> +static void egm_chardev_release(struct device *dev)
> +{
> +	struct nvgrace_egm_dev *egm_chardev = container_of(dev, struct nvgrace_egm_dev, device);
> +
> +	remove_gpus(egm_chardev);
> +	kfree(egm_chardev);
> +}
> +
> +static struct nvgrace_egm_dev *setup_egm_chardev(u64 egmphys, u64 egmlength,
> +						 u64 egmpxm)
> +{
> +	struct nvgrace_egm_dev *egm_chardev;
> +	unsigned int baseminor = MINOR(dev);
> +	int ret;
> +
> +	/*
> +	 * Only MAX_EGM_NODES minors from baseminor were reserved. Reject a PXM
> +	 * outside that window.
> +	 */
> +	if (egmpxm < baseminor || egmpxm - baseminor >= MAX_EGM_NODES) {
> +		pr_err("nvgrace-egm: EGM proximity domain %llu outside reserved minor window [%u, %u)\n",
> +		       egmpxm, baseminor, baseminor + MAX_EGM_NODES);
> +		goto create_err;
> +	}
> +
> +	egm_chardev = kzalloc_obj(*egm_chardev, GFP_KERNEL);
> +	if (!egm_chardev)
> +		goto create_err;
> +
> +	device_initialize(&egm_chardev->device);
> +
> +	/*
> +	 * Use the proximity domain number as the device minor number.
> +	 * So the EGM corresponding to node X would be /dev/egmX.
> +	 */
> +	egm_chardev->egmphys = egmphys;
> +	egm_chardev->egmlength = egmlength;
> +	egm_chardev->egmpxm = egmpxm;
> +	INIT_LIST_HEAD(&egm_chardev->gpus);
> +
> +	egm_chardev->device.devt = MKDEV(MAJOR(dev), egm_chardev->egmpxm);
> +	egm_chardev->device.class = class;
> +	egm_chardev->device.release = egm_chardev_release;
> +	cdev_init(&egm_chardev->cdev, &file_ops);
> +	egm_chardev->cdev.owner = THIS_MODULE;
> +
> +	ret = dev_set_name(&egm_chardev->device, "egm%llu", egm_chardev->egmpxm);
> +	if (ret)
> +		goto error_exit;
> +
> +	ret = cdev_device_add(&egm_chardev->cdev, &egm_chardev->device);
> +	if (ret)
> +		goto error_exit;
> +
> +	return egm_chardev;
> +
> +error_exit:
> +	put_device(&egm_chardev->device);
> +create_err:
> +	return NULL;
> +}
> +
> +static void del_egm_chardev(struct nvgrace_egm_dev *egm_chardev)
> +{
> +	cdev_device_del(&egm_chardev->cdev, &egm_chardev->device);
> +	put_device(&egm_chardev->device);
> +}
> +
> +static char *egm_devnode(const struct device *device, umode_t *mode)
> +{
> +	if (mode)
> +		*mode = 0600;
> +
> +	return NULL;
> +}
> +
> +static ssize_t egm_size_show(struct device *dev, struct device_attribute *attr,
> +			     char *buf)
> +{
> +	struct nvgrace_egm_dev *egm_dev =
> +		container_of(dev, struct nvgrace_egm_dev, device);
> +
> +	return sysfs_emit(buf, "0x%zx\n", egm_dev->egmlength);
> +}
> +
> +static DEVICE_ATTR_RO(egm_size);
> +
> +static struct attribute *attrs[] = {
> +	&dev_attr_egm_size.attr,
> +	NULL,
> +};
> +
> +static const struct attribute_group attr_group = {
> +	.attrs = attrs,
> +};
> +
> +static const struct attribute_group *attr_groups[] = {
> +	&attr_group,
> +	NULL
> +};
> +
> +/*
> + * Determine if the EGM feature is enabled. If disabled, there
> + * will be no EGM properties populated in the ACPI tables and this
> + * fetch would fail.
> + */
> +static int has_egm_property(struct pci_dev *pdev, u64 *pegmpxm)
> +{
> +	return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm",
> +					pegmpxm);
> +}
> +
> +static int fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys,
> +			      u64 *pegmlength)
> +{
> +	int ret;
> +
> +	/*
> +	 * The memory information is present in the system ACPI tables as DSD
> +	 * properties nvidia,egm-base-pa and nvidia,egm-size.
> +	 */
> +	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size",
> +				       pegmlength);
> +	if (ret)
> +		return ret;
> +
> +	ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa",
> +				       pegmphys);
> +
> +	return ret;
> +}
> +
> +static struct nvgrace_egm_dev *get_egm_dev(u64 egmpxm)
> +{
> +	struct nvgrace_egm_dev_entry *egm_entry;
> +
> +	list_for_each_entry(egm_entry, &egm_chardevs, list) {
> +		/*
> +		 * A system could have multiple GPUs associated with an
> +		 * EGM region and will have the same set of EGM region
> +		 * information. Return the existing EGM device if already
> +		 * registered through a different GPU on the same socket.
> +		 */
> +		if (egm_entry->egm_dev->egmpxm == egmpxm)
> +			return egm_entry->egm_dev;
> +	}
> +
> +	return NULL;
> +}
> +
> +static void nvgrace_egm_destroy_pci_devs(void)
> +{
> +	struct nvgrace_egm_dev_entry *entry, *tmp;
> +
> +	list_for_each_entry_safe(entry, tmp, &egm_chardevs, list) {
> +		list_del(&entry->list);
> +		del_egm_chardev(entry->egm_dev);
> +		kfree(entry);
> +	}
> +}
> +
> +/*
> + * Walk all PCI devices and for each one that has a companion ACPI object
> + * advertising EGM properties, create an auxiliary device for that EGM range.
> + * Multiple GPUs on the same socket share a single EGM region (same proximity
> + * domain). So de-duplicate by skipping a pxm that is already registered.
> + */
> +static int nvgrace_egm_create_pci_egm_devs(void)
> +{
> +	struct nvgrace_egm_dev_entry *egm_entry;
> +	struct nvgrace_egm_dev *egm_dev;
> +	struct pci_dev *pdev = NULL;
> +	int ret;
> +
> +	for_each_pci_dev(pdev) {
> +		u64 egmphys, egmlength, egmpxm;
> +
> +		if (has_egm_property(pdev, &egmpxm))
> +			continue;
> +
> +		ret = fetch_egm_property(pdev, &egmphys, &egmlength);
> +		if (ret)
> +			continue;
> +
> +		egm_dev = get_egm_dev(egmpxm);
> +		if (egm_dev) {
> +			ret = add_gpu(egm_dev, pdev);
> +			if (ret)
> +				goto free_dev;
> +			continue;
> +		}
> +
> +		egm_entry = kzalloc_obj(*egm_entry, GFP_KERNEL);
> +		if (!egm_entry) {
> +			pci_dev_put(pdev);
> +			return -ENOMEM;
> +		}
> +
> +		egm_dev = setup_egm_chardev(egmphys, egmlength, egmpxm);
> +		if (!egm_dev) {
> +			kfree(egm_entry);
> +			pci_dev_put(pdev);
> +			return -EINVAL;
> +		}
> +
> +		egm_entry->egm_dev = egm_dev;
> +
> +		ret = add_gpu(egm_entry->egm_dev, pdev);
> +		if (ret) {
> +			del_egm_chardev(egm_dev);
> +			kfree(egm_entry);
> +			goto free_dev;
> +		}
> +
> +		list_add_tail(&egm_entry->list, &egm_chardevs);
> +	}
> +
> +	return 0;
> +
> +free_dev:
> +	/*
> +	 * Reached only via goto from inside the loop where the pdev still
> +	 * holds the reference from the last pci_get_device(). So drop it here.
> +	 */
> +	pci_dev_put(pdev);
> +	nvgrace_egm_destroy_pci_devs();
> +	return ret;
> +}
> +
> +/*
> + * The char device minor is the EGM proximity domain number which is sparse
> + * and need not start at 0. Find the lowest proximity domain among the EGM
> + * devices so the minor range can be reserved starting there. Returns -ENODEV
> + * if the system has no EGM devices.
> + *
> + * EGM PXMs are assumed contiguous within [lowest, lowest + MAX_EGM_NODES),
> + * which holds for the supported Grace configurations.
> + */
> +static int nvgrace_egm_lowest_pxm(u64 *pmin_pxm)
> +{
> +	struct pci_dev *pdev = NULL;
> +	bool found = false;
> +
> +	for_each_pci_dev(pdev) {
> +		u64 egmphys, egmlength, egmpxm;
> +
> +		if (has_egm_property(pdev, &egmpxm))
> +			continue;
> +
> +		if (fetch_egm_property(pdev, &egmphys, &egmlength))
> +			continue;
> +
> +		if (!found || egmpxm < *pmin_pxm) {
> +			*pmin_pxm = egmpxm;
> +			found = true;
> +		}
> +	}
> +
> +	if (!found)
> +		return -ENODEV;
> +
> +	/*
> +	 * The PXM is used verbatim as the minor (only MINORBITS wide). Validate
> +	 * the window would not overflow MINORMASK and be truncated.
> +	 */
> +	if (*pmin_pxm + (MAX_EGM_NODES - 1) > MINORMASK) {
> +		pr_err("nvgrace-egm: lowest EGM proximity domain %llu\n"
> +		       "exceeds the minor range\n", *pmin_pxm);
> +		return -ERANGE;
> +	}
> +
> +	return 0;
> +}
> +
> +static int __init nvgrace_egm_init(void)
> +{
> +	u64 min_pxm;
> +	int ret;
> +
> +	/*
> +	 * Each EGM region is exposed as /dev/egmX with the proximity domain X
> +	 * as its minor number. Reserve MAX_EGM_NODES minors starting at
> +	 * the lowest proximity domain so that every egmX device falls within
> +	 * the reserved range. With no EGM devices present, load without
> +	 * reserving any resources.
> +	 */
> +	ret = nvgrace_egm_lowest_pxm(&min_pxm);
> +	if (ret == -ENODEV)
> +		return 0;
> +	else if (ret)
> +		return ret;
> +
> +	ret = alloc_chrdev_region(&dev, min_pxm, MAX_EGM_NODES,
> +				  NVGRACE_EGM_DEV_NAME);
> +	if (ret < 0)
> +		return ret;
> +
> +	class = class_create(NVGRACE_EGM_DEV_NAME);
> +	if (IS_ERR(class)) {
> +		unregister_chrdev_region(dev, MAX_EGM_NODES);
> +		return PTR_ERR(class);
> +	}
> +
> +	class->devnode = egm_devnode;
> +	class->dev_groups = attr_groups;
> +
> +	ret = nvgrace_egm_create_pci_egm_devs();
> +	if (ret)
> +		goto cleanup_pci_devs;
> +
> +	return 0;
> +
> +cleanup_pci_devs:
> +	nvgrace_egm_destroy_pci_devs();
> +	class_destroy(class);
> +	unregister_chrdev_region(dev, MAX_EGM_NODES);
> +
> +	return ret;
> +}
> +
> +static void __exit nvgrace_egm_cleanup(void)
> +{
> +	/*
> +	 * When no EGM devices were present, the module loaded inertly and
> +	 * reserved nothing. @class stays NULL and there is nothing to undo.
> +	 */
> +	if (!class)
> +		return;
> +
> +	nvgrace_egm_destroy_pci_devs();
> +	class_destroy(class);
> +	unregister_chrdev_region(dev, MAX_EGM_NODES);
> +}
> +
> +module_init(nvgrace_egm_init);
> +module_exit(nvgrace_egm_cleanup);
> +
> +MODULE_LICENSE("GPL");
> +MODULE_AUTHOR("Ankit Agrawal <ankita@nvidia.com>");
> +MODULE_DESCRIPTION("NVGRACE EGM - Module to support Extended GPU Memory on NVIDIA Grace Based systems");

Re: [PATCH 1/4] platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM regions

[Jason Gunthorpe]

On Fri, Jul 03, 2026 at 12:22:21AM +0200, Armin Wolf wrote:
> > - Walk all PCI devices at module init time. For each one advertising
> >    "nvidia,egm-pxm", read the EGM region properties (base SPA, size and
> >    PXM domain) from the ACPI DSD table via the GPU device handle, using
> >    "nvidia,egm-base-pa" and "nvidia,egm-size" properties.
> > - Maintain a module-level list (egm_chardevs) of discovered regions,
> >    de-duplicating by PXM so each socket gets exactly one EGM device
> >    regardless of how many GPUs it has (2 GPUs per EGM device in GB200).
> 
> I assume that the PCI device itself is being handled by the GPU driver, right?
> In this case i suggest that you do the discovery inside the GPU driver during probing,
> otherwise you might run intro problems once PCI hotplug comes into play.

IMHO the ACPI modeling placing the egm-base-pa under the PCI device is
not really very good (but we are stuck with it). The EGM is an
entirely distinct object from the PCI device. The ACPI is just saying
that the PCI device has efficient access to the EGM.

It wouldn't make any sense to involve the GPU driver since the GPU
driver doesn't even run on the hypervisor.

Jason