RE: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory

Shameer Kolothum
2025-11-24T17:09:35Z

> -----Original Message-----
> From: Ankit Agrawal <ankita@nvidia.com>
> Sent: 24 November 2025 11:59
> To: Ankit Agrawal <ankita@nvidia.com>; jgg@ziepe.ca; Yishai Hadas
> <yishaih@nvidia.com>; Shameer Kolothum <skolothumtho@nvidia.com>;
> kevin.tian@intel.com; alex@shazbot.org; Aniket Agashe
> <aniketa@nvidia.com>; Vikram Sethi <vsethi@nvidia.com>; Matt Ochs
> <mochs@nvidia.com>
> Cc: Yunxiang.Li@amd.com; yi.l.liu@intel.com;
> zhangdongdong@eswincomputing.com; Avihai Horon <avihaih@nvidia.com>;
> bhelgaas@google.com; peterx@redhat.com; pstanner@redhat.com; Alistair
> Popple <apopple@nvidia.com>; kvm@vger.kernel.org; linux-
> kernel@vger.kernel.org; Neo Jia <cjia@nvidia.com>; Kirti Wankhede
> <kwankhede@nvidia.com>; Tarun Gupta (SW-GPU) <targupta@nvidia.com>;
> Zhi Wang <zhiw@nvidia.com>; Dan Williams <danw@nvidia.com>; Dheeraj
> Nigam <dnigam@nvidia.com>; Krishnakant Jaju <kjaju@nvidia.com>
> Subject: [PATCH v5 1/7] vfio/nvgrace-gpu: Use faults to map device memory
>
> From: Ankit Agrawal <ankita@nvidia.com>
>
> To make use of the huge pfnmap support and to support zap/remap
> sequence, fault/huge_fault ops based mapping mechanism needs to
> be implemented.
>
> Currently nvgrace-gpu module relies on remap_pfn_range to do
> the mapping during VM bootup. Replace it to instead rely on fault
> and use vmf_insert_pfn to setup the mapping.
>
> Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
> ---
> drivers/vfio/pci/nvgrace-gpu/main.c | 50 +++++++++++++++++------------
> 1 file changed, 30 insertions(+), 20 deletions(-)
>
> diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-
> gpu/main.c
> index e346392b72f6..f74f3d8e1ebe 100644
> --- a/drivers/vfio/pci/nvgrace-gpu/main.c
> +++ b/drivers/vfio/pci/nvgrace-gpu/main.c
> @@ -130,6 +130,32 @@ static void nvgrace_gpu_close_device(struct
> vfio_device *core_vdev)
> vfio_pci_core_close_device(core_vdev);
> }
>
> +static vm_fault_t nvgrace_gpu_vfio_pci_fault(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *vma = vmf->vma;
> + struct nvgrace_gpu_pci_core_device *nvdev = vma->vm_private_data;
> + int index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -
> PAGE_SHIFT);
> + vm_fault_t ret = VM_FAULT_SIGBUS;
> + struct mem_region *memregion;
> + unsigned long pgoff, pfn;
> +
> + memregion = nvgrace_gpu_memregion(index, nvdev);
> + if (!memregion)
> + return ret;
> +
> + pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
> + pfn = PHYS_PFN(memregion->memphys) + pgoff;

The core fault code seems to calculate the BAR offset in vma_to_pfn()
which is missing here.

pgoff = vma->vm_pgoff &
((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);

Is the assumption here is user space will always map at BAR offset 0?

> +
> + scoped_guard(rwsem_read, &nvdev->core_device.memory_lock)
> + ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn);
> +
> + return ret;

Could do return vmf_insert_pfn(vmf->vma, vmf->address, pfn); if
you don't need it later.

Thanks,
Shameer