Re: [RFC v2 07/15] vfio/cxl: expose CXL region to the userspace via a new VFIO device region

Dave Jiang
2025-12-11T18:02:00Z

On 12/11/25 10:31 AM, Manish Honap wrote:
>
>
>> -----Original Message-----
>> From: Dave Jiang <dave.jiang@intel.com>
>> Sent: 11 December 2025 21:36
>> To: Manish Honap <mhonap@nvidia.com>; Aniket Agashe
>> <aniketa@nvidia.com>; Ankit Agrawal <ankita@nvidia.com>; Alex Williamson
>> <alwilliamson@nvidia.com>; Vikram Sethi <vsethi@nvidia.com>; Jason
>> Gunthorpe <jgg@nvidia.com>; Matt Ochs <mochs@nvidia.com>; Shameer
>> Kolothum <skolothumtho@nvidia.com>; alejandro.lucero-palau@amd.com;
>> dave@stgolabs.net; jonathan.cameron@huawei.com;
>> alison.schofield@intel.com; vishal.l.verma@intel.com;
>> ira.weiny@intel.com; dan.j.williams@intel.com; jgg@ziepe.ca; Yishai
>> Hadas <yishaih@nvidia.com>; kevin.tian@intel.com
>> Cc: Neo Jia <cjia@nvidia.com>; Kirti Wankhede <kwankhede@nvidia.com>;
>> Tarun Gupta (SW-GPU) <targupta@nvidia.com>; Zhi Wang <zhiw@nvidia.com>;
>> Krishnakant Jaju <kjaju@nvidia.com>; linux-kernel@vger.kernel.org;
>> linux-cxl@vger.kernel.org; kvm@vger.kernel.org
>> Subject: Re: [RFC v2 07/15] vfio/cxl: expose CXL region to the userspace
>> via a new VFIO device region
>>
>> External email: Use caution opening links or attachments
>>
>>
>> On 12/9/25 9:50 AM, mhonap@nvidia.com wrote:
>>> From: Manish Honap <mhonap@nvidia.com>
>>>
>>> To directly access the device memory, a CXL region is required.
>>> Creating a CXL region requires to configure HDM decoders on the path
>>> to map the access of HPA level by level and evetually hit the DPA in
>>> the CXL topology.
>>>
>>> For the userspace, e.g. QEMU, to access the CXL region, the region is
>>> required to be exposed via VFIO interfaces.
>>>
>>> Introduce a new VFIO device region and region ops to expose the
>>> created CXL region when initialize the device in the vfio-cxl-core.
>>> Introduce a new sub region type for the userspace to identify a CXL
>> region.
>>>
>>> Co-developed-by: Zhi Wang <zhiw@nvidia.com>
>>> Signed-off-by: Zhi Wang <zhiw@nvidia.com>
>>> Signed-off-by: Manish Honap <mhonap@nvidia.com>
>>> ---
>>> drivers/vfio/pci/vfio_cxl_core.c | 122
>> +++++++++++++++++++++++++++++++
>>> drivers/vfio/pci/vfio_pci_core.c | 3 +-
>>> include/linux/vfio_pci_core.h | 5 ++
>>> include/uapi/linux/vfio.h | 4 +
>>> 4 files changed, 133 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/vfio/pci/vfio_cxl_core.c
>>> b/drivers/vfio/pci/vfio_cxl_core.c
>>> index cf53720c0cb7..35d95de47fa8 100644
>>> --- a/drivers/vfio/pci/vfio_cxl_core.c
>>> +++ b/drivers/vfio/pci/vfio_cxl_core.c
>>> @@ -231,6 +231,128 @@ void vfio_cxl_core_destroy_cxl_region(struct
>>> vfio_cxl_core_device *cxl) }
>>> EXPORT_SYMBOL_GPL(vfio_cxl_core_destroy_cxl_region);
>>>
>>> +static int vfio_cxl_region_mmap(struct vfio_pci_core_device *pci,
>>> + struct vfio_pci_region *region,
>>> + struct vm_area_struct *vma) {
>>> + struct vfio_cxl_region *cxl_region = region->data;
>>> + u64 req_len, pgoff, req_start, end;
>>> + int ret;
>>> +
>>> + if (!(region->flags & VFIO_REGION_INFO_FLAG_MMAP))
>>> + return -EINVAL;
>>> +
>>> + if (!(region->flags & VFIO_REGION_INFO_FLAG_READ) &&
>>> + (vma->vm_flags & VM_READ))
>>> + return -EPERM;
>>> +
>>> + if (!(region->flags & VFIO_REGION_INFO_FLAG_WRITE) &&
>>> + (vma->vm_flags & VM_WRITE))
>>> + return -EPERM;
>>> +
>>> + pgoff = vma->vm_pgoff &
>>> + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
>>> +
>>> + if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) ||
>>> + check_add_overflow(PHYS_PFN(cxl_region->addr), pgoff,
>> &req_start) ||
>>> + check_add_overflow(PFN_PHYS(pgoff), req_len, &end))
>>> + return -EOVERFLOW;
>>> +
>>> + if (end > cxl_region->size)
>>> + return -EINVAL;
>>> +
>>> + if (cxl_region->noncached)
>>> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>>> + vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
>>> +
>>> + vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED | VM_IO | VM_PFNMAP |
>>> + VM_DONTEXPAND | VM_DONTDUMP);
>>> +
>>> + ret = remap_pfn_range(vma, vma->vm_start, req_start,
>>> + req_len, vma->vm_page_prot);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + vma->vm_pgoff = req_start;
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device
>> *core_dev,
>>> + char __user *buf, size_t count, loff_t
>> *ppos,
>>> + bool iswrite) {
>>> + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) -
>> VFIO_PCI_NUM_REGIONS;
>>> + struct vfio_cxl_region *cxl_region = core_dev->region[i].data;
>>> + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
>>> +
>>> + if (!count)
>>> + return 0;
>>> +
>>> + return vfio_pci_core_do_io_rw(core_dev, false,
>>> + cxl_region->vaddr,
>>> + (char __user *)buf, pos, count,
>>> + 0, 0, iswrite); }
>>> +
>>> +static void vfio_cxl_region_release(struct vfio_pci_core_device
>> *vdev,
>>> + struct vfio_pci_region *region) { }
>>> +
>>> +static const struct vfio_pci_regops vfio_cxl_regops = {
>>> + .rw = vfio_cxl_region_rw,
>>> + .mmap = vfio_cxl_region_mmap,
>>> + .release = vfio_cxl_region_release,
>>> +};
>>> +
>>> +int vfio_cxl_core_register_cxl_region(struct vfio_cxl_core_device
>>> +*cxl) {
>>> + struct vfio_pci_core_device *pci = &cxl->pci_core;
>>> + struct vfio_cxl *cxl_core = cxl->cxl_core;
>>> + u32 flags;
>>> + int ret;
>>> +
>>> + if (WARN_ON(!cxl_core->region.region || cxl_core->region.vaddr))
>>> + return -EEXIST;
>>> +
>>> + cxl_core->region.vaddr = ioremap(cxl_core->region.addr,
>> cxl_core->region.size);
>>> + if (!cxl_core->region.addr)
>>
>> I think you are wanting to check cxl_core->region.vaddr here right?
>
> Yes, you are correct. I will update this check.
>
>>
>> Also, what is the ioremap'd region for?
>
> It is to handle read/write operations when QEMU performs I/O on the VFIO CXL device region via the read()/write() syscalls.

For the CXL device region, for the most part the operations are done via the region being mmap()'d by qemu right? I understand read/write to BAR0 MMIO. What specific operations are done via read/write to the region? It may be worth mentioning in the commit log.

>
>>
>> DJ