[RFC PATCH 0/3] vfio-user: support multiple FDs backing region mmaps

[RFC PATCH 0/3] vfio-user: support multiple FDs backing region mmaps

[Naman Gulati]

This series extends the vfio-user protocol specification to allow
servers to pass multiple file descriptors for different sparse memory
areas to be mmap'ed within a single device region.

This extension is intended for exposing paravirtualized PCI devices via
a vfio-user server that may have multiple unique sub-regions in a BAR
backed by some mmap'able buffers sourced from distinct file descriptors.

This change introduces a new capability:
  VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS

Unlike VFIO_REGION_INFO_CAP_SPARSE_MMAP (which maps all sparse areas
from a single open file descriptor), this new capability defines an
array of sparse mmap areas where each area contains an index into the
array of file descriptors.

When all sparse areas share a single file descriptor, servers may
continue to emit VFIO_REGION_INFO_CAP_SPARSE_MMAP and send exactly one
file descriptor in SCM_RIGHTS for backwards compatibility with older
clients.

The extensions in this series also lays the groundwork to make use of
the in-development series [1] extending vfio-pci to return distinct
dmabuf FDs for discrete regions of a device's BAR. A vfio-user server
may choose to return dmabuf FDs for various regions of a device's BAR to
map a vfio passthrough device into the guest, in order to retain control
of the device and be able to revoke mappings passed to the VM.


[1] https://lore.kernel.org/all/20260715174737.15287-1-matt@ozlabs.org

Naman Gulati (3):
  vfio-user: add multi-fd region mmap capability
  vfio: mmap sparse regions backed by multiple FDs
  vfio-user: implement client support for multi-fd region mmap

 docs/interop/vfio-user.rst    | 71 +++++++++++++++++++++++++++++++++--
 hw/vfio-user/device.c         | 63 +++++++++++++++++++++++++++++--
 hw/vfio-user/protocol.h       | 20 ++++++++++
 hw/vfio/device.c              | 50 ++++++++++++------------
 hw/vfio/region.c              | 41 ++++++++++++++++----
 hw/vfio/vfio-region.h         |  8 ++++
 include/hw/vfio/vfio-device.h | 29 ++++++++++++--
 7 files changed, 239 insertions(+), 43 deletions(-)

-- 
2.55.0.679.g6767b8d81c-goog

[RFC PATCH 1/3] vfio-user: add multi-fd region mmap capability

[Naman Gulati]

Extend the vfio-user protocol specification to allow servers to pass
multiple file descriptors for different sparse memory areas within a
single device region with capability
VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS.

VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS defines an array of sparse mmap
areas where each area contains:
- offset: Start of the area within the device region (BAR).
- fd_offset: Start of the mapping within the backing file descriptor.
- size: Size of the mappable area.
- fd_index: Zero-based index into the array of file descriptors
  received in the SCM_RIGHTS ancillary data of the
  VFIO_USER_DEVICE_GET_REGION_INFO reply message.

Signed-off-by: Naman Gulati <namangulati@google.com>
---
 docs/interop/vfio-user.rst | 71 +++++++++++++++++++++++++++++++++++---
 1 file changed, 67 insertions(+), 4 deletions(-)

diff --git a/docs/interop/vfio-user.rst b/docs/interop/vfio-user.rst
index 1c92f442d2..5744f58e31 100644
--- a/docs/interop/vfio-user.rst
+++ b/docs/interop/vfio-user.rst
@@ -125,7 +125,10 @@ Sparse Regions
 A region can be memory-mappable in whole or in part. When only a subset of a
 region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP``
 capability is included in the region info reply. This capability describes
-which portions can be mapped by the client.
+which portions can be mapped by the client. When sparse mmap areas are backed
+by multiple distinct file descriptors, a
+``VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS`` capability is included in the region
+info reply instead.
 
 .. Note::
    For example, in a virtual NVMe controller, sparse regions can be used so
@@ -859,9 +862,9 @@ Reply
   * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region.
   * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region.
   * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region.
-    When this flag is set, the reply will include a file descriptor in its
-    meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as
-    ``SCM_RIGHTS`` type ancillary data.
+    When this flag is set, the reply will include one or more file descriptors
+    in its meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed
+    as ``SCM_RIGHTS`` type ancillary data.
   * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the
     reply.
 
@@ -950,6 +953,66 @@ The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
 vfio_region_info_cap_sparse_mmap``).
 
 
+VFIO sparse mmap fds cap header
+"""""""""""""""""""""""""""""""
+
++----------------------+--------------------------------------+
+| Name                 | Value                                |
++======================+======================================+
+| id                   | VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS |
++----------------------+--------------------------------------+
+| version              | 0x1                                  |
++----------------------+--------------------------------------+
+| next                 | <next>                               |
++----------------------+--------------------------------------+
+| sparse mmap fds info | VFIO region info sparse mmap fds     |
++----------------------+--------------------------------------+
+
+This capability is defined when a region supports direct access by the client
+via mmap() where different sparse mmap areas are backed by distinct file
+descriptors in the ``SCM_RIGHTS`` ancillary data array. Capability ID
+``VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS`` is defined as 16 (0x10) to avoid
+clashing with kernel VFIO capabilities.
+
+VFIO region info cap sparse mmap fds
+""""""""""""""""""""""""""""""""""""
+
++-----------+--------+------+
+| Name      | Offset | Size |
++===========+========+======+
+| nr_areas  | 0      | 4    |
++-----------+--------+------+
+| reserved  | 4      | 4    |
++-----------+--------+------+
+| offset    | 8      | 8    |
++-----------+--------+------+
+| fd_offset | 16     | 8    |
++-----------+--------+------+
+| size      | 24     | 8    |
++-----------+--------+------+
+| fd_index  | 32     | 4    |
++-----------+--------+------+
+| pad       | 36     | 4    |
++-----------+--------+------+
+| ...       |        |      |
++-----------+--------+------+
+
+* *nr_areas* is the number of sparse mmap areas in the region.
+* *offset*, *fd_offset*, *size*, and *fd_index* describe a single area that can be
+  mapped by the client. There will be *nr_areas* tuples of offset, fd_offset, size,
+  fd_index, and pad.
+* *offset* specifies the start of the area within the device region (BAR).
+* *fd_offset* specifies the start of the mapping within the backing file descriptor
+  referenced by *fd_index*.
+* *fd_index* is the zero-based index into the array of file descriptors
+  received in the ``SCM_RIGHTS`` ancillary data of the
+  ``VFIO_USER_DEVICE_GET_REGION_INFO`` reply message.
+
+If all sparse mmap areas are backed by a single file descriptor, the server
+may instead send ``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` and exactly one file
+descriptor in ``SCM_RIGHTS`` for compatibility with older clients.
+
+
 ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``
 --------------------------------------
 
-- 
2.55.0.679.g6767b8d81c-goog

[RFC PATCH 2/3] vfio: mmap sparse regions backed by multiple FDs

[Naman Gulati]

Generalize QEMU's VFIO core layer to support mapping sparse memory
subregions from distinct file descriptors within a single device
region.

Replace list of region_fds in VFIODevice with a collection of struct
VFIORegionFDs that internally holds a collection of FDs to mmap per
region.

Add setup_sparse_mmaps hook to VFIODeviceIOOps to allow transport
backends (such as vfio-user) to handle custom sparse mmap capabilities,
falling back to default kernel VFIO sparse parsing.

Store per-subregion fd_offset in struct VFIOMmap to properly address
the mmap call for a subregion.

Signed-off-by: Naman Gulati <namangulati@google.com>
---
 hw/vfio-user/device.c         | 12 ++++++---
 hw/vfio/device.c              | 50 +++++++++++++++++++----------------
 hw/vfio/region.c              | 41 ++++++++++++++++++++++------
 hw/vfio/vfio-region.h         |  8 ++++++
 include/hw/vfio/vfio-device.h | 29 +++++++++++++++++---
 5 files changed, 102 insertions(+), 38 deletions(-)

diff --git a/hw/vfio-user/device.c b/hw/vfio-user/device.c
index b8d2b7c1a8..9c4e10c1fd 100644
--- a/hw/vfio-user/device.c
+++ b/hw/vfio-user/device.c
@@ -169,10 +169,10 @@ static int vfio_user_get_region_info(VFIOUserProxy *proxy,
 }
 
 static int vfio_user_device_io_get_region_info(VFIODevice *vbasedev,
-                                               struct vfio_region_info *info,
-                                               int *fd)
+                                               struct vfio_region_info *info)
 {
-    VFIOUserFDs fds = { 0, 1, fd};
+    int fd = -1;
+    VFIOUserFDs fds = { 0, 1, &fd };
     int ret;
 
     if (info->index > vbasedev->num_initial_regions) {
@@ -184,6 +184,12 @@ static int vfio_user_device_io_get_region_info(VFIODevice *vbasedev,
         return ret;
     }
 
+    if (fds.recv_fds > 0 && info->index < vbasedev->num_initial_regions) {
+        vbasedev->region_fds[info->index].fds = g_new0(int, 1);
+        vbasedev->region_fds[info->index].nr_fds = 1;
+        vbasedev->region_fds[info->index].fds[0] = fd;
+    }
+
     /* cap_offset in valid area */
     if ((info->flags & VFIO_REGION_INFO_FLAG_CAPS) &&
         (info->cap_offset < sizeof(*info) || info->cap_offset > info->argsz)) {
diff --git a/hw/vfio/device.c b/hw/vfio/device.c
index 1a7f8088aa..ea571e33ba 100644
--- a/hw/vfio/device.c
+++ b/hw/vfio/device.c
@@ -202,7 +202,6 @@ int vfio_device_get_region_info(VFIODevice *vbasedev, int index,
                                 struct vfio_region_info **info)
 {
     size_t argsz = sizeof(struct vfio_region_info);
-    int fd = -1;
     int ret;
 
     /*
@@ -226,7 +225,7 @@ int vfio_device_get_region_info(VFIODevice *vbasedev, int index,
 retry:
     (*info)->argsz = argsz;
 
-    ret = vbasedev->io_ops->get_region_info(vbasedev, *info, &fd);
+    ret = vbasedev->io_ops->get_region_info(vbasedev, *info);
     if (ret != 0) {
         g_free(*info);
         *info = NULL;
@@ -237,9 +236,14 @@ retry:
         argsz = (*info)->argsz;
         *info = g_realloc(*info, argsz);
 
-        if (fd != -1) {
-            close(fd);
-            fd = -1;
+        if (index < vbasedev->num_initial_regions &&
+            vbasedev->region_fds != NULL) {
+            for (int j = 0; j < vbasedev->region_fds[index].nr_fds; j++) {
+                close(vbasedev->region_fds[index].fds[j]);
+            }
+            g_free(vbasedev->region_fds[index].fds);
+            vbasedev->region_fds[index].fds = NULL;
+            vbasedev->region_fds[index].nr_fds = 0;
         }
 
         goto retry;
@@ -248,19 +252,22 @@ retry:
     if (index < vbasedev->num_initial_regions) {
         /* fill cache */
         vbasedev->reginfo[index] = *info;
-        if (vbasedev->region_fds != NULL) {
-            vbasedev->region_fds[index] = fd;
-        }
     }
 
     return 0;
 }
 
-int vfio_device_get_region_fd(VFIODevice *vbasedev, int index)
+int vfio_device_get_region_fd(VFIODevice *vbasedev, int index,
+                              uint32_t fd_index)
 {
-        return vbasedev->region_fds ?
-               vbasedev->region_fds[index] :
-               vbasedev->fd;
+    if (!vbasedev->region_fds) {
+        return vbasedev->fd;
+    }
+    if (index < 0 || index >= vbasedev->num_initial_regions ||
+        fd_index >= vbasedev->region_fds[index].nr_fds) {
+        return -1;
+    }
+    return vbasedev->region_fds[index].fds[fd_index];
 }
 
 int vfio_device_get_region_info_type(VFIODevice *vbasedev, uint32_t type,
@@ -480,7 +487,6 @@ void vfio_device_detach(VFIODevice *vbasedev)
 void vfio_device_prepare(VFIODevice *vbasedev, VFIOContainer *bcontainer,
                          struct vfio_device_info *info)
 {
-    int i;
 
     vbasedev->num_irqs = info->num_irqs;
     vbasedev->num_initial_regions = info->num_regions;
@@ -495,10 +501,8 @@ void vfio_device_prepare(VFIODevice *vbasedev, VFIOContainer *bcontainer,
     vbasedev->reginfo = g_new0(struct vfio_region_info *,
                                vbasedev->num_initial_regions);
     if (vbasedev->use_region_fds) {
-        vbasedev->region_fds = g_new0(int, vbasedev->num_initial_regions);
-        for (i = 0; i < vbasedev->num_initial_regions; i++) {
-            vbasedev->region_fds[i] = -1;
-        }
+        vbasedev->region_fds = g_new0(VFIORegionFDs,
+                                      vbasedev->num_initial_regions);
     }
 }
 
@@ -508,8 +512,11 @@ void vfio_device_unprepare(VFIODevice *vbasedev)
 
     for (i = 0; i < vbasedev->num_initial_regions; i++) {
         g_free(vbasedev->reginfo[i]);
-        if (vbasedev->region_fds != NULL && vbasedev->region_fds[i] != -1) {
-            close(vbasedev->region_fds[i]);
+        if (vbasedev->region_fds != NULL) {
+            for (int j = 0; j < vbasedev->region_fds[i].nr_fds; j++) {
+                close(vbasedev->region_fds[i].fds[j]);
+            }
+            g_free(vbasedev->region_fds[i].fds);
         }
     }
 
@@ -592,13 +599,10 @@ static int vfio_device_io_device_feature(VFIODevice *vbasedev,
 }
 
 static int vfio_device_io_get_region_info(VFIODevice *vbasedev,
-                                          struct vfio_region_info *info,
-                                          int *fd)
+                                          struct vfio_region_info *info)
 {
     int ret;
 
-    *fd = -1;
-
     ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, info);
 
     return ret < 0 ? -errno : ret;
diff --git a/hw/vfio/region.c b/hw/vfio/region.c
index dbde339180..3a555e574c 100644
--- a/hw/vfio/region.c
+++ b/hw/vfio/region.c
@@ -162,9 +162,9 @@ static int vfio_mmap_compare_offset(const void *a, const void *b)
     return 0;
 }
 
-static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
-                                          struct vfio_region_info *info,
-                                          Error **errp)
+int vfio_default_setup_sparse_mmaps(VFIORegion *region,
+                                    struct vfio_region_info *info,
+                                    Error **errp)
 {
     struct vfio_info_cap_header *hdr;
     struct vfio_region_info_cap_sparse_mmap *sparse;
@@ -188,6 +188,8 @@ static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
                                             sparse->areas[i].offset +
                                             sparse->areas[i].size - 1);
             region->mmaps[j].offset = sparse->areas[i].offset;
+            region->mmaps[j].fd_offset = region->fd_offset +
+                                         sparse->areas[i].offset;
             region->mmaps[j].size = sparse->areas[i].size;
             j++;
         }
@@ -195,6 +197,25 @@ static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
 
     region->nr_mmaps = j;
     region->mmaps = g_realloc(region->mmaps, j * sizeof(VFIOMmap));
+    return 0;
+}
+
+static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
+                                          struct vfio_region_info *info,
+                                          Error **errp)
+{
+    int ret, i;
+
+    if (region->vbasedev->io_ops &&
+        region->vbasedev->io_ops->setup_sparse_mmaps) {
+        ret = region->vbasedev->io_ops->setup_sparse_mmaps(region, info, errp);
+    } else {
+        ret = vfio_default_setup_sparse_mmaps(region, info, errp);
+    }
+
+    if (ret) {
+        return ret;
+    }
 
     /*
      * Sort sparse mmaps by offset to ensure proper handling of gaps
@@ -261,6 +282,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region,
                 region->nr_mmaps = 1;
                 region->mmaps = g_new0(VFIOMmap, region->nr_mmaps);
                 region->mmaps[0].offset = 0;
+                region->mmaps[0].fd_offset = region->fd_offset;
                 region->mmaps[0].size = region->size;
             } else if (ret) {
                 return ret;
@@ -355,7 +377,6 @@ int vfio_region_mmap(VFIORegion *region)
     off_t map_offset = 0;
     size_t align;
     char *name;
-    int fd;
 
     if (!region->mem || !region->nr_mmaps) {
         return 0;
@@ -392,8 +413,6 @@ int vfio_region_mmap(VFIORegion *region)
         return ret;
     }
 
-    fd = vfio_device_get_region_fd(region->vbasedev, region->nr);
-
     map_align = (void *)ROUND_UP((uintptr_t)map_base, (uintptr_t)align);
     munmap(map_base, map_align - map_base);
     munmap(map_align + region->size,
@@ -405,12 +424,18 @@ int vfio_region_mmap(VFIORegion *region)
      * offsets being in ascending order.
      */
     for (i = 0; i < region->nr_mmaps; i++) {
+        int fd = vfio_device_get_region_fd(region->vbasedev, region->nr,
+                                           region->mmaps[i].fd_index);
+        if (fd < 0) {
+            ret = -EINVAL;
+            goto no_mmap;
+        }
+
         munmap(map_align + map_offset, region->mmaps[i].offset - map_offset);
         region->mmaps[i].mmap = mmap(map_align + region->mmaps[i].offset,
                                      region->mmaps[i].size, prot,
                                      MAP_SHARED | MAP_FIXED, fd,
-                                     region->fd_offset +
-                                     region->mmaps[i].offset);
+                                     region->mmaps[i].fd_offset);
         if (region->mmaps[i].mmap == MAP_FAILED) {
             ret = -errno;
             /*
diff --git a/hw/vfio/vfio-region.h b/hw/vfio/vfio-region.h
index 58b236f113..0bb082e9a6 100644
--- a/hw/vfio/vfio-region.h
+++ b/hw/vfio/vfio-region.h
@@ -10,12 +10,17 @@
 #define HW_VFIO_REGION_H
 
 #include "system/memory.h"
+#ifdef CONFIG_LINUX
+#include <linux/vfio.h>
+#endif
 
 typedef struct VFIOMmap {
     MemoryRegion mem;
     void *mmap;
     off_t offset;
     size_t size;
+    uint32_t fd_index;
+    uint64_t fd_offset;
 } VFIOMmap;
 
 typedef struct VFIODevice VFIODevice;
@@ -43,5 +48,8 @@ int vfio_region_mmap(VFIORegion *region);
 void vfio_region_mmaps_set_enabled(VFIORegion *region, bool enabled);
 void vfio_region_exit(VFIORegion *region);
 void vfio_region_finalize(VFIORegion *region);
+int vfio_default_setup_sparse_mmaps(VFIORegion *region,
+                                    struct vfio_region_info *info,
+                                    Error **errp);
 
 #endif /* HW_VFIO_REGION_H */
diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h
index 8472420d3f..c97a35fa80 100644
--- a/include/hw/vfio/vfio-device.h
+++ b/include/hw/vfio/vfio-device.h
@@ -43,12 +43,18 @@ enum {
 
 typedef struct VFIODeviceOps VFIODeviceOps;
 typedef struct VFIODeviceIOOps VFIODeviceIOOps;
+typedef struct VFIORegion VFIORegion;
 typedef struct VFIOMigration VFIOMigration;
 
 typedef struct IOMMUFDBackend IOMMUFDBackend;
 typedef struct VFIOIOASHwpt VFIOIOASHwpt;
 typedef struct VFIOUserProxy VFIOUserProxy;
 
+typedef struct VFIORegionFDs {
+    int *fds;
+    int nr_fds;
+} VFIORegionFDs;
+
 typedef struct VFIODevice {
     QLIST_ENTRY(VFIODevice) next;
     QLIST_ENTRY(VFIODevice) container_next;
@@ -89,7 +95,7 @@ typedef struct VFIODevice {
     VFIOIOASHwpt *hwpt;
     QLIST_ENTRY(VFIODevice) hwpt_next;
     struct vfio_region_info **reginfo;
-    int *region_fds;
+    struct VFIORegionFDs *region_fds;
     VFIODeviceCPR cpr;
     VFIOUserProxy *proxy;
 } VFIODevice;
@@ -211,12 +217,26 @@ struct VFIODeviceIOOps {
      * @vdev: #VFIODevice to use
      * @info: set @info->index to the region index to look up; the rest of the
      *        struct will be filled in on success
-     * @fd: pointer to the fd for the region; will be -1 if not found
      *
      * Returns 0 on success or -errno.
      */
     int (*get_region_info)(VFIODevice *vdev,
-                           struct vfio_region_info *info, int *fd);
+                           struct vfio_region_info *info);
+
+    /**
+     * @setup_sparse_mmaps
+     *
+     * Parse sparse mmap capabilities for a region and initialize region->mmaps.
+     *
+     * @region: #VFIORegion to set up
+     * @info: region info struct containing capabilities
+     * @errp: pointer to Error*, to store an error if it happens
+     *
+     * Returns 0 on success, -ENODEV if not sparse, or negative error code.
+     */
+    int (*setup_sparse_mmaps)(VFIORegion *region,
+                              struct vfio_region_info *info,
+                              Error **errp);
 
     /**
      * @get_irq_info
@@ -317,7 +337,8 @@ int vfio_device_get_region_info_type(VFIODevice *vbasedev, uint32_t type,
  *
  * Returns the fd.
  */
-int vfio_device_get_region_fd(VFIODevice *vbasedev, int index);
+int vfio_device_get_region_fd(VFIODevice *vbasedev, int index,
+                              uint32_t fd_index);
 
 bool vfio_device_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_type);
 
-- 
2.55.0.679.g6767b8d81c-goog

Re: [RFC PATCH 2/3] vfio: mmap sparse regions backed by multiple FDs

[John Levon]

On Tue, Aug 11, 2026 at 11:39:29PM +0000, Naman Gulati wrote:

> Generalize QEMU's VFIO core layer to support mapping sparse memory
> subregions from distinct file descriptors within a single device
> region.
> 
> Replace list of region_fds in VFIODevice with a collection of struct
> VFIORegionFDs that internally holds a collection of FDs to mmap per
> region.

This is getting a bit non-obvious to me, we now have a region_fds array where
each member is also an array.

Given that you've introduced a dependency on VFIORegion in the
include/hw/vfio/vfio-device.h for the other callback, I'm wondering why we don't
just store an array of VFIORegions in VFIODevice instead of reginfo+region_fds ?

And I'm wondering if we really have to have two separate callbacks for getting
region info.
 

> @@ -184,6 +184,12 @@ static int vfio_user_device_io_get_region_info(VFIODevice *vbasedev,
>          return ret;
>      }
>  
> +    if (fds.recv_fds > 0 && info->index < vbasedev->num_initial_regions) {
> +        vbasedev->region_fds[info->index].fds = g_new0(int, 1);
> +        vbasedev->region_fds[info->index].nr_fds = 1;
> +        vbasedev->region_fds[info->index].fds[0] = fd;
> +    }
> +

This seems quite surprising behaviour in particular - if we happen to be asking
about a "standard" region, then we'll store its fd info in vbasedev. But we
might not be setting device->reginfo[index] at the same time.

I'd expect vbasedev to be const in these callbacks.

And what about the special regions > num_initial_regions?

I feel like this callback should be passing in a VFIORegion to be filled in,
then the caller can do as they wish with it. We could even embed the
vfio_region_info into VFIORegion for convenience for the callers that want caps
etc.

Alternatively, if we don't want to do this, then at least get_region_info()
should take a VFIORegionFds out parameter, that the caller can then do what they
like with.

Thoughts Cédric ?

I'm also not sure why we have a sparse mmaps callback - why can't that be
generic code that handles both capabilities? Are you concerned about the cap
number clashing in future vfio kernel versions perhaps?

regards
john

[RFC PATCH 3/3] vfio-user: implement client support for multi-fd region mmap

[Naman Gulati]

Add client support for capability VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS
(capability ID 16) in vfio-user.

Update vfio_user_device_io_get_region_info() to receive multiple file
descriptors passed in SCM_RIGHTS ancillary data for a region, and
implement setup_sparse_mmaps in vfio_user_device_io_ops_sock to parse
per-area file descriptor indices and fd_offset values.

Signed-off-by: Naman Gulati <namangulati@google.com>
---
 hw/vfio-user/device.c   | 63 ++++++++++++++++++++++++++++++++++++-----
 hw/vfio-user/protocol.h | 20 +++++++++++++
 2 files changed, 76 insertions(+), 7 deletions(-)

diff --git a/hw/vfio-user/device.c b/hw/vfio-user/device.c
index 9c4e10c1fd..48b1cd8f1e 100644
--- a/hw/vfio-user/device.c
+++ b/hw/vfio-user/device.c
@@ -14,6 +14,9 @@
 
 #include "hw/vfio-user/device.h"
 #include "hw/vfio-user/trace.h"
+#include "hw/vfio/vfio-region.h"
+#include "hw/vfio/vfio-helpers.h"
+#include "hw/vfio/trace.h"
 
 /*
  * These are to defend against a malign server trying
@@ -171,23 +174,26 @@ static int vfio_user_get_region_info(VFIOUserProxy *proxy,
 static int vfio_user_device_io_get_region_info(VFIODevice *vbasedev,
                                                struct vfio_region_info *info)
 {
-    int fd = -1;
-    VFIOUserFDs fds = { 0, 1, &fd };
+    int fds[VFIO_USER_MAX_MAX_FDS];
+    int num_fds = VFIO_USER_MAX_MAX_FDS;
+    VFIOUserFDs user_fds = { 0, num_fds, fds };
     int ret;
 
     if (info->index > vbasedev->num_initial_regions) {
         return -EINVAL;
     }
 
-    ret = vfio_user_get_region_info(vbasedev->proxy, info, &fds);
+    ret = vfio_user_get_region_info(vbasedev->proxy, info, &user_fds);
     if (ret) {
         return ret;
     }
 
-    if (fds.recv_fds > 0 && info->index < vbasedev->num_initial_regions) {
-        vbasedev->region_fds[info->index].fds = g_new0(int, 1);
-        vbasedev->region_fds[info->index].nr_fds = 1;
-        vbasedev->region_fds[info->index].fds[0] = fd;
+    if (user_fds.recv_fds > 0 && info->index < vbasedev->num_initial_regions) {
+        vbasedev->region_fds[info->index].fds = g_new0(int, user_fds.recv_fds);
+        vbasedev->region_fds[info->index].nr_fds = user_fds.recv_fds;
+        for (int i = 0; i < user_fds.recv_fds; i++) {
+            vbasedev->region_fds[info->index].fds[i] = fds[i];
+        }
     }
 
     /* cap_offset in valid area */
@@ -478,9 +484,52 @@ static int vfio_user_device_io_region_write(VFIODevice *vbasedev, uint8_t index,
 /*
  * Socket-based io_ops
  */
+static int vfio_user_device_io_setup_sparse_mmaps(VFIORegion *region,
+                                                  struct vfio_region_info *info,
+                                                  Error **errp)
+{
+    struct vfio_info_cap_header *hdr;
+    int i, j = 0;
+
+    hdr = vfio_get_region_info_cap(info, VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS);
+    if (hdr) {
+        struct vfio_region_info_cap_sparse_mmap_fds *sparse_fds =
+            container_of(hdr, struct vfio_region_info_cap_sparse_mmap_fds,
+                         header);
+
+        trace_vfio_region_sparse_mmap_header(region->vbasedev->name,
+                                             region->nr, sparse_fds->nr_areas);
+
+        region->mmaps = g_new0(VFIOMmap, sparse_fds->nr_areas);
+
+        for (i = 0; i < sparse_fds->nr_areas; i++) {
+            if (sparse_fds->areas[i].size) {
+                uint64_t end = sparse_fds->areas[i].offset +
+                               sparse_fds->areas[i].size - 1;
+
+                trace_vfio_region_sparse_mmap_entry(i,
+                                                    sparse_fds->areas[i].offset,
+                                                    end);
+                region->mmaps[j].offset = sparse_fds->areas[i].offset;
+                region->mmaps[j].fd_offset = sparse_fds->areas[i].fd_offset;
+                region->mmaps[j].size = sparse_fds->areas[i].size;
+                region->mmaps[j].fd_index = sparse_fds->areas[i].fd_index;
+                j++;
+            }
+        }
+
+        region->nr_mmaps = j;
+        region->mmaps = g_realloc(region->mmaps, j * sizeof(VFIOMmap));
+        return 0;
+    }
+
+    return vfio_default_setup_sparse_mmaps(region, info, errp);
+}
+
 VFIODeviceIOOps vfio_user_device_io_ops_sock = {
     .device_feature = vfio_user_device_io_device_feature,
     .get_region_info = vfio_user_device_io_get_region_info,
+    .setup_sparse_mmaps = vfio_user_device_io_setup_sparse_mmaps,
     .get_irq_info = vfio_user_device_io_get_irq_info,
     .set_irqs = vfio_user_device_io_set_irqs,
     .region_read = vfio_user_device_io_region_read,
diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h
index c478d1353f..135b965d99 100644
--- a/hw/vfio-user/protocol.h
+++ b/hw/vfio-user/protocol.h
@@ -166,6 +166,26 @@ typedef struct {
     uint64_t offset;
 } VFIOUserRegionInfo;
 
+/*
+ * VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS
+ */
+#define VFIO_REGION_INFO_CAP_SPARSE_MMAP_FDS 16
+
+struct vfio_region_sparse_mmap_fd_area {
+    uint64_t offset;
+    uint64_t fd_offset;
+    uint64_t size;
+    uint32_t fd_index;
+    uint32_t pad;
+};
+
+struct vfio_region_info_cap_sparse_mmap_fds {
+    struct vfio_info_cap_header header;
+    uint32_t nr_areas;
+    uint32_t reserved;
+    struct vfio_region_sparse_mmap_fd_area areas[];
+};
+
 /*
  * VFIO_USER_DEVICE_GET_IRQ_INFO
  * imported from struct vfio_irq_info
-- 
2.55.0.679.g6767b8d81c-goog