* [PATCH v3 00/20] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 01/20] doc: add deprecation notice for VFIO API Anatoly Burakov
` (19 more replies)
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (7 subsequent siblings)
8 siblings, 20 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, etc.).
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, which is why this will be kept for compatibility for these
use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind`
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (20):
doc: add deprecation notice for VFIO API
doc: add deprecation notice for vDPA driver API
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vhost: remove group-related API from drivers
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 7 +
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 10 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 33 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 2 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal_vfio.c | 2433 ++++++++-------------
lib/eal/linux/eal_vfio.h | 167 +-
lib/eal/linux/eal_vfio_cdev.c | 389 ++++
lib/eal/linux/eal_vfio_group.c | 983 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
35 files changed, 4266 insertions(+), 1868 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.47.3
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v3 01/20] doc: add deprecation notice for VFIO API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 02/20] doc: add deprecation notice for vDPA driver API Anatoly Burakov
` (18 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev
The entirety of VFIO API will be either made internal and only available to
drivers, or will be removed. Add upcoming changes to deprecation notices.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 1e60ac4e82..5e3df71048 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -158,3 +158,7 @@ Deprecation Notices
without relying on flow API.
Currently there is no alternative API
providing the same functionality as with ``repr_matching_en`` set to 0.
+
+* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only, and will only be available to EAL and drivers.
+ Group-based API (``rte_vfio_*_group_*``) will be removed and replaced with unified container device assignment API.
+ This change will be made in 26.11 release.
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 02/20] doc: add deprecation notice for vDPA driver API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 01/20] doc: add deprecation notice for VFIO API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (17 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev
Currently, vDPA driver API is partly built around VFIO group API, which
will be removed in a future release and replaced with a device container
assignment API. Make this know in deprecation notice.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 5e3df71048..e2c08387fe 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -162,3 +162,6 @@ Deprecation Notices
* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only, and will only be available to EAL and drivers.
Group-based API (``rte_vfio_*_group_*``) will be removed and replaced with unified container device assignment API.
This change will be made in 26.11 release.
+
+* vdpa: vDPA driver API will no longer offer ``get_vfio_group_fd`` as part of its internal API.
+ All drivers will be adjusted to use the new unified VFIO container device assignment API.
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 01/20] doc: add deprecation notice for VFIO API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 02/20] doc: add deprecation notice for vDPA driver API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 17:36 ` Stephen Hemminger
2025-11-18 16:29 ` [PATCH v3 04/20] vfio: make all functions internal Anatoly Burakov
` (16 subsequent siblings)
19 siblings, 1 reply; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 04/20] vfio: make all functions internal
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (2 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 05/20] vfio: split get device info from setup Anatoly Burakov
` (15 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Bruce Richardson, Tyler Retzlaff, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 30 ++++++++++++------------
lib/eal/include/rte_vfio.h | 47 +++++++++++++++++++++++++++++++++-----
lib/eal/linux/eal_vfio.c | 32 +++++++++++++-------------
lib/eal/windows/eal.c | 4 ++--
4 files changed, 74 insertions(+), 39 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6215245ad5..3142ccd4bd 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -811,7 +811,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -821,7 +821,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -830,33 +830,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -866,7 +866,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -874,7 +874,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -882,7 +882,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -890,7 +890,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -898,7 +898,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -907,7 +907,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -916,7 +916,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -927,7 +927,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 21fe7cb1d9..33854025da 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -445,7 +445,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -456,7 +456,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 05/20] vfio: split get device info from setup
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (3 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 04/20] vfio: make all functions internal Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 06/20] vfio: add container device assignment API Anatoly Burakov
` (14 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson, Tyler Retzlaff
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 8562fdcc6b..57ec9d7066 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1194,12 +1202,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index f6673cf181..9500b8aff5 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -328,12 +328,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 3142ccd4bd..dea32ae428 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -814,8 +814,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 06/20] vfio: add container device assignment API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (4 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 05/20] vfio: split get device info from setup Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 07/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (13 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Bruce Richardson, Tyler Retzlaff
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index dea32ae428..a7360db7a7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -936,3 +936,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..e7e2ee950b 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 07/20] net/nbl: do not use VFIO group bind API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (5 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 06/20] vfio: add container device assignment API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 08/20] net/ntnic: use container device assignment API Anatoly Burakov
` (12 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Dimon Zhao, Kyo Liu, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 18 +++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 566f3a11ee..d641ecd9a8 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,13 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +431,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +543,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +556,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 08/20] net/ntnic: use container device assignment API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (6 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 07/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 09/20] vdpa/ifc: " Anatoly Burakov
` (11 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 09/20] vdpa/ifc: use container device assignment API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (7 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 08/20] net/ntnic: use container device assignment API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 10/20] vdpa/nfp: " Anatoly Burakov
` (10 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 15 +++------------
1 file changed, 3 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..6f1c050787 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -174,28 +174,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 10/20] vdpa/nfp: use container device assignment API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (8 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 09/20] vdpa/ifc: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 11/20] vdpa/sfc: " Anatoly Burakov
` (9 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index f4fd5c92ec..4885fa5cbc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -122,33 +122,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
rte_pci_unmap_device(pci_dev);
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 11/20] vdpa/sfc: use container device assignment API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (9 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 10/20] vdpa/nfp: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 12/20] vhost: remove group-related API from drivers Anatoly Burakov
` (8 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 12/20] vhost: remove group-related API from drivers
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (10 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 11/20] vdpa/sfc: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 13/20] vfio: remove group-based API Anatoly Burakov
` (7 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia, Matan Azrad,
Viacheslav Ovsiienko, Chaoyong He
Some vDPA drivers have "get_vfio_group_fd" call in their internal driver
API structure, but it is not used for anything beyond device assignment
to containers which can now be achieved via other means, so remove this
API and all its usages.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
drivers/vdpa/ifc/ifcvf_vdpa.c | 19 -------------------
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 20 --------------------
lib/vhost/vdpa_driver.h | 3 ---
5 files changed, 47 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 0c2b4d020a..2f80cf4072 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to response to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 6f1c050787..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -1204,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1465,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1596,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4885fa5cbc..efd7ee7d95 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -152,7 +150,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1018,22 +1015,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1185,7 +1166,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 13/20] vfio: remove group-based API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (11 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 12/20] vhost: remove group-related API from drivers Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 14/20] vfio: cleanup and refactor Anatoly Burakov
` (6 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Bruce Richardson, Tyler Retzlaff
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index a7360db7a7..f66ac01ba2 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -848,13 +848,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -873,14 +866,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -897,24 +882,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index e7e2ee950b..941b7d0541 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 14/20] vfio: cleanup and refactor
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (12 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 13/20] vfio: remove group-based API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
` (5 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Kyo Liu, Leon Yu,
Sam Chen, Tyler Retzlaff
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 250 ++-
lib/eal/linux/eal_vfio.c | 2285 ++++++++--------------
lib/eal/linux/eal_vfio.h | 142 +-
lib/eal/linux/eal_vfio_group.c | 983 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2129 insertions(+), 1636 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index c0aa21b57d..cd9ccccefd 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -148,6 +148,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index 0cb074ab95..6752584a7b 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -375,6 +375,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if dpdk_conf.get('RTE_ARCH_64')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 550d4e0e8d..4b11b38322 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 57ec9d7066..03a61e7765 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index d641ecd9a8..d71d4fc035 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -413,7 +413,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f66ac01ba2..7b3ce875a8 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -913,3 +913,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 941b7d0541..b6d7b4591f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -41,26 +41,47 @@ struct vfio_device_info;
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Enumeration of VFIO operational modes.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
+/**
+ * @internal
+ * Set up a device managed by VFIO driver.
+ *
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -255,32 +328,45 @@ rte_vfio_container_destroy(int container_fd);
* @return
* 0 on success.
* <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..8545552308 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -9,6 +9,7 @@
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
+#include <sys/stat.h>
#include <dirent.h>
#include <rte_errno.h>
@@ -24,80 +25,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +313,127 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+/*
+ * we will rely on kernel to not allow user to assign the same device to different containers, but
+ * kernel will not prevent mapping the same device twice using two different fd's, so we need to
+ * deduplicate our internal config to make sure we only store unique device fd's.
+ */
+static bool
+fd_is_same(int fd1, int fd2)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ struct stat st1, st2;
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
+ if (fd1 < 0 || fd2 < 0)
+ return false;
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
+ if (fstat(fd1, &st1) < 0)
+ return false;
+ if (fstat(fd2, &st2) < 0)
+ return false;
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
+ return st1.st_dev == st2.st_dev && st1.st_ino == st2.st_ino;
}
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
+bool
+vfio_container_is_default(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = (struct container){0};
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +448,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +468,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +478,546 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *idev, *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we want to prevent user from assigning devices twice to prevent resource leaks, but for
+ * group mode this is not trivial, as there is no direct way to know which fd belongs to
+ * which group/device, except for directly comparing fd's with stat. so, that's what we're
+ * going to do. we do not need to look in other configs as if we were to attempt to use a
+ * different container, the kernel wouldn't have allowed us to bind the group to the
+ * container in the first place.
+ */
+ DEVICE_FOREACH_ACTIVE(cfg, idev) {
+ if (idev != dev && fd_is_same(idev->fd, dev->fd)) {
+ EAL_LOG(ERR, "Device %s already assigned to this container",
+ dev_addr);
+ res = VFIO_EXISTS;
+ *out_dev = idev;
+ goto dev_remove;
+ }
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+dev_remove:
+ /* device will be closed, but we still need to keep the group consistent */
+ grp->n_devices--;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1037,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1055,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1074,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1103,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1158,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1173,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1194,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1211,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1247,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1268,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1285,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1314,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..68d3a3ec6e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,161 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we only need to initialize the lock in mem maps */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +176,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..6123a83aec
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,983 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ close(vfio_container_fd);
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (!n_extensions)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char dev[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(dev, sizeof(dev),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index e99ebed256..75a9afdd03 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 15/20] bus/pci: use the new VFIO mode API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (13 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 14/20] vfio: cleanup and refactor Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 16/20] bus/fslmc: " Anatoly Burakov
` (4 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 66d7e09a6e..6f7b67e140 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -598,7 +598,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 16/20] bus/fslmc: use the new VFIO mode API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (14 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 17/20] net/hinic3: " Anatoly Burakov
` (3 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/fslmc/fslmc_bus.c | 10 +++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 4529ec5085..82b53bcc54 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -335,6 +335,13 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -582,7 +589,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 4b11b38322..afde967e34 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 17/20] net/hinic3: use the new VFIO mode API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (15 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 16/20] bus/fslmc: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 18/20] net/ntnic: " Anatoly Burakov
` (2 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 668bbf4a0e..c5bf2bc8a4 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -77,7 +77,7 @@ bool
hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *rte_dev)
{
return ((RTE_ETH_DEV_TO_PCI(rte_dev)->kdrv == RTE_PCI_KDRV_VFIO) &&
- (rte_vfio_noiommu_is_enabled() != 1));
+ (rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU));
}
int
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 18/20] net/ntnic: use the new VFIO mode API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (16 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 17/20] net/hinic3: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 20/20] vfio: introduce cdev mode Anatoly Burakov
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 23c3452d2d..c1195e5fbe 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2692,7 +2692,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 19/20] vfio: remove no-IOMMU check API
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (17 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 18/20] net/ntnic: " Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 20/20] vfio: introduce cdev mode Anatoly Burakov
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Bruce Richardson, Tyler Retzlaff
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 7b3ce875a8..d850ef2c4f 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -842,12 +842,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index b6d7b4591f..2730cfa67d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 8545552308..2e9ad7dbb4 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1310,13 +1310,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v3 20/20] vfio: introduce cdev mode
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
` (18 preceding siblings ...)
2025-11-18 16:29 ` [PATCH v3 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2025-11-18 16:29 ` Anatoly Burakov
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2025-11-18 16:29 UTC (permalink / raw)
To: dev, Bruce Richardson, Tyler Retzlaff
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 210 +++++++++++++++++
lib/eal/linux/eal_vfio.h | 29 ++-
lib/eal/linux/eal_vfio_cdev.c | 389 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 710 insertions(+), 2 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index d850ef2c4f..c993d383bf 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -923,3 +923,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 2730cfa67d..385e09b5ba 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 2e9ad7dbb4..e1d1de494b 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -369,6 +369,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -538,6 +552,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -695,6 +758,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -715,6 +821,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -787,6 +896,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -906,6 +1033,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -971,6 +1104,9 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
+ /* if primary is in cdev mode, we need to sync ioas as well */
+ if (mode == RTE_VFIO_MODE_CDEV && vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
/* primary handles DMA setup for default containers */
group_cfg->dma_setup_done = true;
@@ -990,6 +1126,19 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
@@ -1004,6 +1153,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1143,6 +1293,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1342,6 +1526,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1400,6 +1603,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 68d3a3ec6e..52cb3a0e08 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -160,12 +171,24 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -176,6 +199,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..d2d48faa8e
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,389 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
+#include <limits.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ /* cdev mode does not have different IOMMU ops */
+ vfio_cfg.ops = &iommufd_ops;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0)
+ return -1;
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 75a9afdd03..adf043d24c 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h
2025-11-18 16:29 ` [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2025-11-18 17:36 ` Stephen Hemminger
0 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2025-11-18 17:36 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev, Maxime Coquelin
On Tue, 18 Nov 2025 16:29:04 +0000
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
> required. Update internal headers to version v6.17 and include iommufd.h.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v7 00/18] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (18 more replies)
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (6 subsequent siblings)
8 siblings, 19 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, etc.).
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, which is why this will be kept for compatibility for these
use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (18):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vhost: remove group-related API from drivers
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 7 -
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 10 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 33 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 2 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal_vfio.c | 2433 ++++++++-------------
lib/eal/linux/eal_vfio.h | 167 +-
lib/eal/linux/eal_vfio_cdev.c | 390 ++++
lib/eal/linux/eal_vfio_group.c | 984 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
35 files changed, 4261 insertions(+), 1875 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.47.3
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v7 01/18] uapi: update to v6.17 and add iommufd.h
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 02/18] vfio: make all functions internal Anatoly Burakov
` (17 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
Depends-on: series-37486 ("doc: add deprecation notice for VFIO API")
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 02/18] vfio: make all functions internal
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 03/18] vfio: split get device info from setup Anatoly Burakov
` (16 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 4 ---
lib/eal/freebsd/eal.c | 30 +++++++++---------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++----------
lib/eal/windows/eal.c | 4 +--
5 files changed, 74 insertions(+), 43 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 4bfbb7468e..b137ed0445 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -140,9 +140,5 @@ Deprecation Notices
``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
Those API functions are used internally by DPDK core and netvsc PMD.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only, and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: vDPA driver API will no longer offer ``get_vfio_group_fd`` as part of its internal API.
All drivers will be adjusted to use the new unified VFIO container device assignment API.
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 60f5e676a8..4cc80a2107 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -813,7 +813,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -823,7 +823,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -832,33 +832,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -868,7 +868,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -876,7 +876,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -884,7 +884,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -892,7 +892,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -900,7 +900,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -909,7 +909,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -918,7 +918,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -929,7 +929,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index f06375a624..d2179a9569 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -447,7 +447,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -458,7 +458,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 03/18] vfio: split get device info from setup
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 02/18] vfio: make all functions internal Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 04/18] vfio: add container device assignment API Anatoly Burakov
` (15 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..54e9506058 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index f6673cf181..9500b8aff5 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -328,12 +328,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 4cc80a2107..63eaa734eb 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -816,8 +816,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 04/18] vfio: add container device assignment API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (2 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 03/18] vfio: split get device info from setup Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (14 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 63eaa734eb..0adb01032d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -938,3 +938,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..e7e2ee950b 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 05/18] net/nbl: do not use VFIO group bind API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (3 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 04/18] vfio: add container device assignment API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 06/18] net/ntnic: use container device assignment API Anatoly Burakov
` (13 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Dimon Zhao, Kyo Liu, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 18 +++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..fb256e543f 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,13 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +431,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +543,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +556,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 06/18] net/ntnic: use container device assignment API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (4 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 07/18] vdpa/ifc: " Anatoly Burakov
` (12 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 07/18] vdpa/ifc: use container device assignment API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (5 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 06/18] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 08/18] vdpa/nfp: " Anatoly Burakov
` (11 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 15 +++------------
1 file changed, 3 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..6f1c050787 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -174,28 +174,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 08/18] vdpa/nfp: use container device assignment API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (6 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 07/18] vdpa/ifc: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 09/18] vdpa/sfc: " Anatoly Burakov
` (10 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index f4fd5c92ec..4885fa5cbc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -122,33 +122,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
rte_pci_unmap_device(pci_dev);
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 09/18] vdpa/sfc: use container device assignment API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (7 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 08/18] vdpa/nfp: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 10/18] vhost: remove group-related API from drivers Anatoly Burakov
` (9 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 10/18] vhost: remove group-related API from drivers
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (8 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 09/18] vdpa/sfc: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 11/18] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia, Matan Azrad,
Viacheslav Ovsiienko, Chaoyong He
Some vDPA drivers have "get_vfio_group_fd" call in their internal driver
API structure, but it is not used for anything beyond device assignment
to containers which can now be achieved via other means, so remove this
API and all its usages.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 3 ---
drivers/vdpa/ifc/ifcvf_vdpa.c | 19 -------------------
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 20 --------------------
lib/vhost/vdpa_driver.h | 3 ---
6 files changed, 50 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 0c2b4d020a..2f80cf4072 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to response to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index b137ed0445..ac667e91a6 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -139,6 +139,3 @@ Deprecation Notices
* bus/vmbus: Starting DPDK 25.11, all the vmbus API defined in
``drivers/bus/vmbus/rte_bus_vmbus.h`` will become internal to DPDK.
Those API functions are used internally by DPDK core and netvsc PMD.
-
-* vdpa: vDPA driver API will no longer offer ``get_vfio_group_fd`` as part of its internal API.
- All drivers will be adjusted to use the new unified VFIO container device assignment API.
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 6f1c050787..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -1204,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1465,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1596,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4885fa5cbc..efd7ee7d95 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -152,7 +150,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1018,22 +1015,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1185,7 +1166,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 11/18] vfio: remove group-based API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (9 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 10/18] vhost: remove group-related API from drivers Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 12/18] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0adb01032d..da59e3d5cf 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,13 +850,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -875,14 +868,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -899,24 +884,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index e7e2ee950b..941b7d0541 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 12/18] vfio: cleanup and refactor
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (10 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 11/18] vfio: remove group-based API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Kyo Liu, Leon Yu,
Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 250 ++-
lib/eal/linux/eal_vfio.c | 2285 ++++++++--------------
lib/eal/linux/eal_vfio.h | 142 +-
lib/eal/linux/eal_vfio_group.c | 984 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2130 insertions(+), 1636 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 523b0fc0ed..0f7206a31d 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -148,6 +148,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index 02e2798cca..070a6badf9 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -381,6 +381,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if dpdk_conf.get('RTE_ARCH_64')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 550d4e0e8d..4b11b38322 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 54e9506058..0d30b1cdf1 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index fb256e543f..9aacf7438c 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -413,7 +413,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index da59e3d5cf..86ac148c88 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -915,3 +915,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 941b7d0541..0af41c3610 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -255,32 +328,45 @@ rte_vfio_container_destroy(int container_fd);
* @return
* 0 on success.
* <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..8545552308 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -9,6 +9,7 @@
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
+#include <sys/stat.h>
#include <dirent.h>
#include <rte_errno.h>
@@ -24,80 +25,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +313,127 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+/*
+ * we will rely on kernel to not allow user to assign the same device to different containers, but
+ * kernel will not prevent mapping the same device twice using two different fd's, so we need to
+ * deduplicate our internal config to make sure we only store unique device fd's.
+ */
+static bool
+fd_is_same(int fd1, int fd2)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ struct stat st1, st2;
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
+ if (fd1 < 0 || fd2 < 0)
+ return false;
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
+ if (fstat(fd1, &st1) < 0)
+ return false;
+ if (fstat(fd2, &st2) < 0)
+ return false;
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
+ return st1.st_dev == st2.st_dev && st1.st_ino == st2.st_ino;
}
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
+bool
+vfio_container_is_default(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = (struct container){0};
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +448,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +468,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +478,546 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *idev, *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we want to prevent user from assigning devices twice to prevent resource leaks, but for
+ * group mode this is not trivial, as there is no direct way to know which fd belongs to
+ * which group/device, except for directly comparing fd's with stat. so, that's what we're
+ * going to do. we do not need to look in other configs as if we were to attempt to use a
+ * different container, the kernel wouldn't have allowed us to bind the group to the
+ * container in the first place.
+ */
+ DEVICE_FOREACH_ACTIVE(cfg, idev) {
+ if (idev != dev && fd_is_same(idev->fd, dev->fd)) {
+ EAL_LOG(ERR, "Device %s already assigned to this container",
+ dev_addr);
+ res = VFIO_EXISTS;
+ *out_dev = idev;
+ goto dev_remove;
+ }
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+dev_remove:
+ /* device will be closed, but we still need to keep the group consistent */
+ grp->n_devices--;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1037,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1055,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1074,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1103,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1158,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1173,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1194,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1211,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1247,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1268,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1285,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1314,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..68d3a3ec6e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,161 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we only need to initialize the lock in mem maps */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +176,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..1f1b3084c2
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,984 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ close(vfio_container_fd);
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char dev[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(dev, sizeof(dev),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index e99ebed256..75a9afdd03 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 13/18] bus/pci: use the new VFIO mode API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (11 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 12/18] vfio: cleanup and refactor Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 14/18] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 2ffac82e94..6a5035ac62 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -597,7 +597,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 14/18] bus/fslmc: use the new VFIO mode API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (12 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 15/18] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 10 +++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index abdb0ad50d..d486c41ef1 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -335,6 +335,13 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -582,7 +589,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 4b11b38322..afde967e34 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 15/18] net/hinic3: use the new VFIO mode API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (13 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 14/18] bus/fslmc: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 16/18] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 668bbf4a0e..c5bf2bc8a4 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -77,7 +77,7 @@ bool
hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *rte_dev)
{
return ((RTE_ETH_DEV_TO_PCI(rte_dev)->kdrv == RTE_PCI_KDRV_VFIO) &&
- (rte_vfio_noiommu_is_enabled() != 1));
+ (rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU));
}
int
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 16/18] net/ntnic: use the new VFIO mode API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (14 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 15/18] net/hinic3: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 89e751e7e0..7fda61d3ff 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2691,7 +2691,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 17/18] vfio: remove no-IOMMU check API
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (15 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 16/18] net/ntnic: " Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 18/18] vfio: introduce cdev mode Anatoly Burakov
2026-05-01 22:45 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 86ac148c88..84bb618484 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -844,12 +844,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0af41c3610..c4ba0e5cda 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 8545552308..2e9ad7dbb4 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1310,13 +1310,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v7 18/18] vfio: introduce cdev mode
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (16 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-02-26 14:17 ` Anatoly Burakov
2026-05-01 22:45 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-02-26 14:17 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 210 +++++++++++++++++
lib/eal/linux/eal_vfio.h | 29 ++-
lib/eal/linux/eal_vfio_cdev.c | 390 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 711 insertions(+), 2 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 84bb618484..a49a8a2b4c 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -925,3 +925,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index c4ba0e5cda..502a68c948 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 2e9ad7dbb4..e1d1de494b 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -369,6 +369,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -538,6 +552,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -695,6 +758,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -715,6 +821,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -787,6 +896,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -906,6 +1033,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -971,6 +1104,9 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
+ /* if primary is in cdev mode, we need to sync ioas as well */
+ if (mode == RTE_VFIO_MODE_CDEV && vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
/* primary handles DMA setup for default containers */
group_cfg->dma_setup_done = true;
@@ -990,6 +1126,19 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
@@ -1004,6 +1153,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1143,6 +1293,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1342,6 +1526,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1400,6 +1603,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 68d3a3ec6e..52cb3a0e08 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -160,12 +171,24 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -176,6 +199,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..ce61a97853
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,390 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ /* cdev mode does not have different IOMMU ops */
+ vfio_cfg.ops = &iommufd_ops;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0)
+ return -1;
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 75a9afdd03..adf043d24c 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v7 00/18] Support VFIO cdev API in DPDK
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
` (17 preceding siblings ...)
2026-02-26 14:17 ` [PATCH v7 18/18] vfio: introduce cdev mode Anatoly Burakov
@ 2026-05-01 22:45 ` Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-05-01 22:45 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Thu, 26 Feb 2026 14:17:23 +0000
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> This patchset introduces a major refactor of the VFIO subsystem in DPDK to
> support character device (cdev) interface introduced in Linux kernel, as well as
> make the API more streamlined and useful. The goal is to simplify device
> management, improve compatibility, and clarify API responsibilities.
>
> The following sections outline the key issues addressed by this patchset and the
> corresponding changes introduced.
The patch set is still interesting but lots of build failures.
You need to rebase and resubmit.
Look for shadow variable warnings etc.
ccache clang -Ilib/librte_eal.a.p -Ilib -I../lib -Ilib/eal/common -I../lib/eal/common -I. -I.. -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -I../kernel/linux -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -Ilib/argparse -I../lib/argparse -Xclang -fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wvla -Wcast-qual -Wcomma -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=corei7 -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API '-DABI_VERSION="26.1"' -DRTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP -DRTE_LOG_DEFAULT_LOGTYPE=lib.eal -DRTE_ANNOTATE_LOCKS -Wthread-safety -MD -MQ lib/librte_eal.a.p/eal_linux_eal_vfio_group.c.o -MF lib/librte_eal.a.p/eal_linux_eal_vfio_group.c.o.d -o lib/librte_eal.a.p/eal_linux_eal_vfio_group.c.o -c ../lib/eal/linux/eal_vfio_group.c
../lib/eal/linux/eal_vfio_group.c:914:8: error: declaration shadows a local variable [-Werror,-Wshadow]
914 | char dev[PATH_MAX];
| ^
../lib/eal/linux/eal_vfio_group.c:905:94: note: previous declaration is here
905 | vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
| ^
1 error generated.
[97/7766] Generating
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v8 00/18] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (18 more replies)
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (5 subsequent siblings)
8 siblings, 19 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.).
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (18):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vhost: remove group-related API from drivers
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 10 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 33 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal_vfio.c | 2437 ++++++++-------------
lib/eal/linux/eal_vfio.h | 167 +-
lib/eal/linux/eal_vfio_cdev.c | 390 ++++
lib/eal/linux/eal_vfio_group.c | 984 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
35 files changed, 4248 insertions(+), 1896 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.47.3
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v8 01/18] uapi: update to v6.17 and add iommufd.h
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 02/18] vfio: make all functions internal Anatoly Burakov
` (17 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 02/18] vfio: make all functions internal
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 03/18] vfio: split get device info from setup Anatoly Burakov
` (16 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ----
lib/eal/freebsd/eal.c | 30 +++++++++---------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++----------
lib/eal/windows/eal.c | 4 +--
5 files changed, 74 insertions(+), 45 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index c7f8230278..f2901064f5 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -30,12 +30,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 03/18] vfio: split get device info from setup
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 02/18] vfio: make all functions internal Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 04/18] vfio: add container device assignment API Anatoly Burakov
` (15 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..54e9506058 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 170a2e03d0..3ee4b76781 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..f8ab932962 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 04/18] vfio: add container device assignment API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (2 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 03/18] vfio: split get device info from setup Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (14 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f8ab932962..0c64a62c5a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -944,3 +944,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..e7e2ee950b 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 05/18] net/nbl: do not use VFIO group bind API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (3 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 04/18] vfio: add container device assignment API Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 06/18] net/ntnic: use container device assignment API Anatoly Burakov
` (13 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 18 +++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..fb256e543f 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,13 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +431,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +543,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +556,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 06/18] net/ntnic: use container device assignment API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (4 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 07/18] vdpa/ifc: " Anatoly Burakov
` (12 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 07/18] vdpa/ifc: use container device assignment API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (5 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 06/18] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-06-11 15:08 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 08/18] vdpa/nfp: " Anatoly Burakov
` (11 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:08 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 15 +++------------
1 file changed, 3 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..6f1c050787 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -174,28 +174,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 08/18] vdpa/nfp: use container device assignment API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (6 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 07/18] vdpa/ifc: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 09/18] vdpa/sfc: " Anatoly Burakov
` (10 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index f4fd5c92ec..4885fa5cbc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -122,33 +122,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
rte_pci_unmap_device(pci_dev);
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 09/18] vdpa/sfc: use container device assignment API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (7 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 08/18] vdpa/nfp: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 10/18] vhost: remove group-related API from drivers Anatoly Burakov
` (9 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 10/18] vhost: remove group-related API from drivers
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (8 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 09/18] vdpa/sfc: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 11/18] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia, Matan Azrad,
Viacheslav Ovsiienko, Chaoyong He
Some vDPA drivers have "get_vfio_group_fd" call in their internal driver
API structure, but it is not used for anything beyond device assignment
to containers which can now be achieved via other means, so remove this
API and all its usages.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
drivers/vdpa/ifc/ifcvf_vdpa.c | 19 -------------------
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 20 --------------------
lib/vhost/vdpa_driver.h | 3 ---
6 files changed, 51 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 0c2b4d020a..2f80cf4072 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to response to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index f2901064f5..c520129ac3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -30,10 +30,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 6f1c050787..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -1204,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1465,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1596,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4885fa5cbc..efd7ee7d95 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -152,7 +150,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1018,22 +1015,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1185,7 +1166,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 11/18] vfio: remove group-based API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (9 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 10/18] vhost: remove group-related API from drivers Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 12/18] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0c64a62c5a..cda72dfd1d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,13 +856,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -881,14 +874,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -905,24 +890,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index e7e2ee950b..941b7d0541 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 12/18] vfio: cleanup and refactor
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (10 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 11/18] vfio: remove group-based API Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 250 ++-
lib/eal/linux/eal_vfio.c | 2265 +++++++---------------
lib/eal/linux/eal_vfio.h | 142 +-
lib/eal/linux/eal_vfio_group.c | 984 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2104 insertions(+), 1642 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..3ca68ccf24 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 54e9506058..0d30b1cdf1 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index fb256e543f..9aacf7438c 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -413,7 +413,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index cda72dfd1d..6c1d1e3751 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -921,3 +921,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 941b7d0541..0af41c3610 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -255,32 +328,45 @@ rte_vfio_container_destroy(int container_fd);
* @return
* 0 on success.
* <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..708d14ad51 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -9,6 +9,7 @@
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
+#include <sys/stat.h>
#include <dirent.h>
#include <rte_errno.h>
@@ -24,80 +25,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +313,106 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = (struct container){0};
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +427,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +447,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +457,535 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we would've liked to prevent user from assigning devices twice to
+ * prevent resource leaks, but for group mode this is not possible, as
+ * there is no way to know which fd belongs to which group/device.
+ *
+ * we also do not need to look in other configs as if we were to attempt
+ * to use a different container, the kernel wouldn't have allowed us to
+ * bind the group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1005,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1023,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1042,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1071,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1126,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1141,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1162,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1179,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1215,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1236,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1253,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1282,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..68d3a3ec6e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,161 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we only need to initialize the lock in mem maps */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +176,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..520e61610c
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,984 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ close(vfio_container_fd);
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 13/18] bus/pci: use the new VFIO mode API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (11 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 12/18] vfio: cleanup and refactor Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 14/18] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..a1575b84e2 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 14/18] bus/fslmc: use the new VFIO mode API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (12 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 15/18] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 10 +++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index c7549a361a..a225c88b86 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,13 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +507,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 3ca68ccf24..15273fcd57 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 15/18] net/hinic3: use the new VFIO mode API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (13 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 14/18] bus/fslmc: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 16/18] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..224be6f81c 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU;
}
int
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 16/18] net/ntnic: use the new VFIO mode API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (14 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 15/18] net/hinic3: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..8b6bca974c 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 17/18] vfio: remove no-IOMMU check API
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (15 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 16/18] net/ntnic: " Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 18/18] vfio: introduce cdev mode Anatoly Burakov
2026-06-11 17:49 ` [PATCH v8 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6c1d1e3751..bb05a969a9 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,12 +850,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0af41c3610..c4ba0e5cda 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 708d14ad51..c104008a43 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1278,13 +1278,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v8 18/18] vfio: introduce cdev mode
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (16 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-06-11 15:09 ` Anatoly Burakov
2026-06-11 17:49 ` [PATCH v8 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-06-11 15:09 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 210 +++++++++++++++++
lib/eal/linux/eal_vfio.h | 29 ++-
lib/eal/linux/eal_vfio_cdev.c | 390 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 711 insertions(+), 2 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index bb05a969a9..a84280a66c 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -931,3 +931,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index c4ba0e5cda..502a68c948 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index c104008a43..004ee48cf5 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -348,6 +348,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -517,6 +531,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -663,6 +726,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -683,6 +789,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -755,6 +864,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -874,6 +1001,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -939,6 +1072,9 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
+ /* if primary is in cdev mode, we need to sync ioas as well */
+ if (mode == RTE_VFIO_MODE_CDEV && vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
/* primary handles DMA setup for default containers */
group_cfg->dma_setup_done = true;
@@ -958,6 +1094,19 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
@@ -972,6 +1121,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1111,6 +1261,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1310,6 +1494,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1368,6 +1571,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 68d3a3ec6e..52cb3a0e08 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -160,12 +171,24 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -176,6 +199,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..ce61a97853
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,390 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ /* cdev mode does not have different IOMMU ops */
+ vfio_cfg.ops = &iommufd_ops;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0)
+ return -1;
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.47.3
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v8 00/18] Support VFIO cdev API in DPDK
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
` (17 preceding siblings ...)
2026-06-11 15:09 ` [PATCH v8 18/18] vfio: introduce cdev mode Anatoly Burakov
@ 2026-06-11 17:49 ` Stephen Hemminger
18 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-06-11 17:49 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Thu, 11 Jun 2026 16:08:52 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> This patchset introduces a major refactor of the VFIO subsystem in DPDK to
> support character device (cdev) interface introduced in Linux kernel, as well as
> make the API more streamlined and useful. The goal is to simplify device
> management, improve compatibility, and clarify API responsibilities.
>
> The following sections outline the key issues addressed by this patchset and the
> corresponding changes introduced.
>
> 1. Only group mode is supported
> ===============================
>
> Since kernel version 4.14.327 (LTS), VFIO supports the new character device
> (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
> is a device-centric mode and does away with all the complexity regarding groups
> and IOMMU types, delegating it all to the kernel, and exposes a much simpler
> interface to userspace.
>
> The old group interface is still around, and will need to be kept in DPDK both
> for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
> driver, no-IOMMU mode etc.).
>
> To enable this, VFIO is heavily refactored, so that the code can support both
> modes while relying on (mostly) common infrastructure.
>
> Note that the existing `rte_vfio_device_setup/release` model is fundamentally
> incompatible with cdev mode, because for custom container cases, the expected
> flow is that the user binds the IOMMU group (and thus, implicitly, the device
> itself) to a specific container using `rte_vfio_container_group_bind`, whereas
> this step is not needed for cdev as the device fd is assigned to the container
> straight away.
>
> Therefore, what we do instead is introduce a new API for container device
> assignment which, semantically, will assign a device to specified container, so
> that when it is mapped using `rte_pci_map_device`, the appropriate container is
> selected. Under the hood though, we essentially transition to getting device fd
> straight away at assign stage, so that by the time the PCI bus attempts to map
> the device, it is already mapped and we just return an fd. There is no
> "unassign" API because `release_device` already performs that function.
>
> Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
> some introspection into VFIO's internals, with three new modes: group
> (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
> Although no-IOMMU is technically a variant of group mode, the distinction is
> largely irrelevant to the user, as all usages of noiommu checks in our codebase
> are for deciding whether to use IOVA or PA, not anything to do with managing
> groups. The current plan for kernel community is to *not* introduce no-IOMMU
> cdev implementation, and IOMMUFD's own group API compatibility layer also does
> not implement no-IOMMU mode, which is why this will be kept for compatibility
> for these use cases.
>
> There were other users of VFIO which relied on group API but only for convenience
> purposes; no actual VFIO functionality depended on those API's. Therefore, group
> API's are removed and, where appropriate, replaced with the new API's.
>
> List of removed API's:
>
> * `rte_vfio_get_group_fd`
> * `rte_vfio_clear_group`
> * `rte_vfio_container_group_bind` (replaced by container assign API)
> * `rte_vfio_container_group_unbind`
> * `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
>
> 2. The API responsibilities aren't clear and bleed into each other
> ==================================================================
>
> Some API's do multiple things at once. In particular:
>
> * `rte_vfio_get_device_info` will setup the device
> * `rte_vfio_setup_device` will get device info
>
> These API's have been adjusted to do one thing only.
>
> v8:
> - Rebase
> - Fixed build errors due to variable shadowing
> - Removed duplicate fd check as kernel does not provide a way to distinguish
> between device fd's
>
> v7:
> - Rebase
> - Added removal of deprecation notices
> - Fixed implicit numeric comparison in patch 12
>
> v6:
> - Fixed missing header include in vfio cdev file
>
> v5:
> - Added back missing uapi patch
>
> v4:
> - Fixed issues with documenting rte_vfio_mode enum
> - Separated deprecation notices into a separate patchset
>
> v3:
> - Make API removal cleaner
> - Fix `get_group_num` usages to align with new API
> - Fix issues with function exports
> - Fix issues with `setup_device` returning old-style values in some cases
>
> v2:
> - Make the entire API internal
> - More aggressive API pruning, complete removal of group API
> - Fixed a bug in group mode where device could not be used
> - Better documentation and deprecation notice patches
> - Moved doc patches to beginning of patchset
>
> Anatoly Burakov (18):
> uapi: update to v6.17 and add iommufd.h
> vfio: make all functions internal
> vfio: split get device info from setup
> vfio: add container device assignment API
> net/nbl: do not use VFIO group bind API
> net/ntnic: use container device assignment API
> vdpa/ifc: use container device assignment API
> vdpa/nfp: use container device assignment API
> vdpa/sfc: use container device assignment API
> vhost: remove group-related API from drivers
> vfio: remove group-based API
> vfio: cleanup and refactor
> bus/pci: use the new VFIO mode API
> bus/fslmc: use the new VFIO mode API
> net/hinic3: use the new VFIO mode API
> net/ntnic: use the new VFIO mode API
> vfio: remove no-IOMMU check API
> vfio: introduce cdev mode
>
> config/arm/meson.build | 1 +
> config/meson.build | 1 +
> doc/guides/prog_guide/vhost_lib.rst | 4 -
> doc/guides/rel_notes/deprecation.rst | 10 -
> drivers/bus/cdx/cdx_vfio.c | 25 +-
> drivers/bus/fslmc/fslmc_bus.c | 10 +-
> drivers/bus/fslmc/fslmc_vfio.c | 6 +-
> drivers/bus/pci/linux/pci.c | 2 +-
> drivers/bus/pci/linux/pci_vfio.c | 33 +-
> drivers/bus/platform/platform.c | 9 +-
> drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
> drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
> drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +-
> drivers/net/nbl/nbl_include/nbl_include.h | 1 +
> drivers/net/ntnic/ntnic_ethdev.c | 2 +-
> drivers/net/ntnic/ntnic_vfio.c | 30 +-
> drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
> drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
> drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
> drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
> drivers/vdpa/sfc/sfc_vdpa.h | 2 -
> kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
> kernel/linux/uapi/linux/vduse.h | 2 +-
> kernel/linux/uapi/linux/vfio.h | 12 +-
> kernel/linux/uapi/version | 2 +-
> lib/eal/freebsd/eal.c | 98 +-
> lib/eal/include/rte_vfio.h | 387 ++--
> lib/eal/linux/eal_vfio.c | 2437 ++++++++-------------
> lib/eal/linux/eal_vfio.h | 167 +-
> lib/eal/linux/eal_vfio_cdev.c | 390 ++++
> lib/eal/linux/eal_vfio_group.c | 984 +++++++++
> lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
> lib/eal/linux/meson.build | 2 +
> lib/eal/windows/eal.c | 4 +-
> lib/vhost/vdpa_driver.h | 3 -
> 35 files changed, 4248 insertions(+), 1896 deletions(-)
> create mode 100644 kernel/linux/uapi/linux/iommufd.h
> create mode 100644 lib/eal/linux/eal_vfio_cdev.c
> create mode 100644 lib/eal/linux/eal_vfio_group.c
>
Big patchset so sent the big AI model at it...
Patch 4 (vfio: add container device assignment API)
Warning: header doc for rte_vfio_container_assign_device() says "<0 on
failure, rte_errno is set", but neither rte_vfio_get_group_num() nor
rte_vfio_container_group_bind() sets rte_errno on the Linux failure
paths at this point in the series. The rte_errno contract only becomes
true after the patch 12 rewrite. Either set rte_errno here or defer the
doc claim to patch 12.
Patch 5 (net/nbl: do not use VFIO group bind API)
Info: function definition does not follow DPDK style (return type on
its own line, blank line between declarations and statements):
static int
nbl_open_group_fd(int iommu_group_num)
{
char path[PATH_MAX];
snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
return open(path, O_RDWR);
}
Patch 7 (vdpa/ifc: use container device assignment API)
Warning: this patch removes both the "internal->vfio_group_fd = -1"
initialization and the only assignment, but ifcvf_get_vfio_group_fd()
still returns the field until patch 10. Between patches 7 and 10 the
vdpa op returns 0 (zeroed allocation), i.e. a "valid" fd value. Nothing
in lib/vhost calls the op anymore so it is not reachable in practice,
but for bisectability either keep the -1 initialization here or move
patch 10 ahead of patches 7-9.
Patch 8 (vdpa/nfp: use container device assignment API)
Warning: same staging issue as patch 7, plus nfp_vdpa_vfio_teardown()
still calls rte_vfio_container_group_unbind(fd, device->iommu_group)
with device->iommu_group now never assigned (always 0 from calloc), so
every teardown between patches 8 and 10 issues an unbind for group 0
that fails silently. The teardown unbind removal currently in patch 10
belongs in this patch (patch 9 does this correctly for sfc, removing
the fields and all uses in one patch).
Patch 12 (vfio: cleanup and refactor) -- partial review
Warning: missing release notes. This patch (together with patches 2, 11,
17, 18) removes the public rte_vfio API, removes the group-bind API, and
changes rte_vfio_setup_device()/rte_vfio_get_group_num() return
semantics. None of the series touches the current release notes file;
the entire VFIO API removal and the new cdev mode need entries in
"Removed Items" / "New Features".
Info: rte_errno convention comment at top of eal_vfio.c says "ENOXIO";
the errno is ENXIO (code uses the correct one).
Patch 18 (vfio: introduce cdev mode)
Error: ioas_id is corrupted in secondary processes. struct container
puts vfio_group_config and vfio_cdev_config in a union, and both place
their first member at offset 0 (bool dma_setup_done / uint32_t ioas_id).
In vfio_select_mode(), the secondary path does:
if (mode == RTE_VFIO_MODE_CDEV && vfio_cdev_sync_ioas(cfg) < 0)
goto err;
/* primary handles DMA setup for default containers */
group_cfg->dma_setup_done = true;
In cdev mode the unconditional dma_setup_done store overwrites the low
byte of the ioas_id just received from the primary. The corrupted id is
then used by VFIO_DEVICE_ATTACH_IOMMUFD_PT and IOMMU_IOAS_MAP/UNMAP in
the secondary. It happens to work only when the primary's IOAS id has
low byte 1. Fix is to make the store mode-conditional:
if (mode == RTE_VFIO_MODE_GROUP || mode == RTE_VFIO_MODE_NOIOMMU)
group_cfg->dma_setup_done = true;
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v9 00/19] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (2 preceding siblings ...)
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 01/19] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (19 more replies)
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (4 subsequent siblings)
8 siblings, 20 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
the API the way it was before.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (19):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 10 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 33 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 21 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal_vfio.c | 2444 ++++++++-------------
lib/eal/linux/eal_vfio.h | 167 +-
lib/eal/linux/eal_vfio_cdev.c | 390 ++++
lib/eal/linux/eal_vfio_group.c | 984 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
35 files changed, 4256 insertions(+), 1896 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v9 01/19] uapi: update to v6.17 and add iommufd.h
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 02/19] vfio: make all functions internal Anatoly Burakov
` (18 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 02/19] vfio: make all functions internal
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 01/19] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 03/19] vfio: split get device info from setup Anatoly Burakov
` (17 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ----
lib/eal/freebsd/eal.c | 30 +++++++++---------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++----------
lib/eal/windows/eal.c | 4 +--
5 files changed, 74 insertions(+), 45 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index c7f8230278..f2901064f5 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -30,12 +30,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 03/19] vfio: split get device info from setup
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 01/19] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 02/19] vfio: make all functions internal Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 04/19] vfio: add container device assignment API Anatoly Burakov
` (16 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..54e9506058 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 170a2e03d0..3ee4b76781 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..f8ab932962 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 04/19] vfio: add container device assignment API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (2 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 03/19] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 05/19] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (15 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f8ab932962..0c64a62c5a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -944,3 +944,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..4890ecf06d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 05/19] net/nbl: do not use VFIO group bind API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (3 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 04/19] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 06/19] net/ntnic: use container device assignment API Anatoly Burakov
` (14 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 19 ++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..996573638d 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,14 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +432,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +544,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +557,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 06/19] net/ntnic: use container device assignment API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (4 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 05/19] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 07/19] vdpa/ifc: " Anatoly Burakov
` (13 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 07/19] vdpa/ifc: use container device assignment API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (5 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 06/19] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 08/19] vdpa/nfp: " Anatoly Burakov
` (12 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 08/19] vdpa/nfp: use container device assignment API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (6 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 07/19] vdpa/ifc: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 09/19] vdpa/sfc: " Anatoly Burakov
` (11 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index f4fd5c92ec..efd7ee7d95 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -122,33 +120,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
rte_pci_unmap_device(pci_dev);
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
@@ -159,7 +150,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1025,22 +1015,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1192,7 +1166,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 09/19] vdpa/sfc: use container device assignment API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (7 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 08/19] vdpa/nfp: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 10/19] vdpa/mlx5: remove group-related API Anatoly Burakov
` (10 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 10/19] vdpa/mlx5: remove group-related API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (8 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 09/19] vdpa/sfc: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 11/19] vhost: remove group-related API from driver Anatoly Burakov
` (9 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 11/19] vhost: remove group-related API from driver
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (9 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 10/19] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 12/19] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
lib/vhost/vdpa_driver.h | 3 ---
3 files changed, 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 0c2b4d020a..2f80cf4072 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to response to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index f2901064f5..c520129ac3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -30,10 +30,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 12/19] vfio: remove group-based API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (10 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 11/19] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 13/19] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0c64a62c5a..cda72dfd1d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,13 +856,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -881,14 +874,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -905,24 +890,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 4890ecf06d..51ca1c162f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 13/19] vfio: cleanup and refactor
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (11 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 12/19] vfio: remove group-based API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 14/19] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 252 ++-
lib/eal/linux/eal_vfio.c | 2264 +++++++---------------
lib/eal/linux/eal_vfio.h | 142 +-
lib/eal/linux/eal_vfio_group.c | 984 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2104 insertions(+), 1643 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..3ca68ccf24 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 54e9506058..0d30b1cdf1 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 996573638d..27e76bbe0f 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -414,7 +414,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index cda72dfd1d..6c1d1e3751 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -921,3 +921,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 51ca1c162f..6f21fa85ce 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -254,33 +327,46 @@ rte_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..715e329974 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,80 +24,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +312,106 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = (struct container){0};
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +426,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +446,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +456,535 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we would've liked to prevent user from assigning devices twice to
+ * prevent resource leaks, but for group mode this is not possible, as
+ * there is no way to know which fd belongs to which group/device.
+ *
+ * we also do not need to look in other configs as if we were to attempt
+ * to use a different container, the kernel wouldn't have allowed us to
+ * bind the group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1004,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1022,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1041,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1070,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1125,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1140,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1161,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1178,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1214,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1235,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1252,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1281,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..68d3a3ec6e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,161 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we only need to initialize the lock in mem maps */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +176,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..520e61610c
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,984 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ close(vfio_container_fd);
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 14/19] bus/pci: use the new VFIO mode API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (12 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 13/19] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 15/19] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..a1575b84e2 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 15/19] bus/fslmc: use the new VFIO mode API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (13 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 14/19] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 16/19] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 10 +++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index c7549a361a..a225c88b86 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,13 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +507,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 3ca68ccf24..15273fcd57 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 16/19] net/hinic3: use the new VFIO mode API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (14 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 15/19] bus/fslmc: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 17/19] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..224be6f81c 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 17/19] net/ntnic: use the new VFIO mode API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (15 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 16/19] net/hinic3: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 18/19] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..8b6bca974c 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 18/19] vfio: remove no-IOMMU check API
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (16 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 17/19] net/ntnic: " Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 19/19] vfio: introduce cdev mode Anatoly Burakov
2026-08-06 3:52 ` [PATCH v9 00/19] Support VFIO cdev API in DPDK Stephen Hemminger
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6c1d1e3751..bb05a969a9 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,12 +850,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6f21fa85ce..6a4d101e33 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 715e329974..5263bea240 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1277,13 +1277,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v9 19/19] vfio: introduce cdev mode
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (17 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 18/19] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-05 13:45 ` Anatoly Burakov
2026-08-06 3:52 ` [PATCH v9 00/19] Support VFIO cdev API in DPDK Stephen Hemminger
19 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-05 13:45 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 222 +++++++++++++++++-
lib/eal/linux/eal_vfio.h | 29 ++-
lib/eal/linux/eal_vfio_cdev.c | 390 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 721 insertions(+), 4 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index bb05a969a9..a84280a66c 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -931,3 +931,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6a4d101e33..b9c7184616 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 5263bea240..00dc9af78a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -347,6 +347,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -516,6 +530,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -662,6 +725,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -682,6 +788,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -754,6 +863,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -873,6 +1000,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -939,8 +1072,19 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- group_cfg->dma_setup_done = true;
+ if (mode == RTE_VFIO_MODE_CDEV) {
+ /* in cdev mode, sync ioas */
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+ } else if (mode == RTE_VFIO_MODE_GROUP ||
+ mode == RTE_VFIO_MODE_NOIOMMU) {
+ /* in group-like modes, primary handles DMA */
+ group_cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
@@ -957,6 +1101,19 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
@@ -971,6 +1128,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1110,6 +1268,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1309,6 +1501,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1367,6 +1578,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 68d3a3ec6e..52cb3a0e08 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -160,12 +171,24 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -176,6 +199,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..ce61a97853
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,390 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ /* cdev mode does not have different IOMMU ops */
+ vfio_cfg.ops = &iommufd_ops;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0)
+ return -1;
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v9 00/19] Support VFIO cdev API in DPDK
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
` (18 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 19/19] vfio: introduce cdev mode Anatoly Burakov
@ 2026-08-06 3:52 ` Stephen Hemminger
19 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-08-06 3:52 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Wed, 5 Aug 2026 14:45:29 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> Therefore, what we do instead is introduce a new API for container device
> assignment which, semantically, will assign a device to specified container, so
> that when it is mapped using `rte_pci_map_device`, the appropriate container is
> selected. Under the hood though, we essentially transition to getting device fd
> straight away at assign stage, so that by the time the PCI bus attempts to map
> the device, it is already mapped and we just return an fd. There is no
> "unassign" API because `release_device` already performs that function.
>
> Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
> some introspection into VFIO's internals, with three new modes: group
> (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
> Although no-IOMMU is technically a variant of group mode, the distinction is
> largely irrelevant to the user, as all usages of noiommu checks in our codebase
> are for deciding whether to use IOVA or PA, not anything to do with managing
> groups. The current plan for kernel community is to *not* introduce no-IOMMU
> cdev implementation, and IOMMUFD's own group API compatibility layer also does
> not implement no-IOMMU mode, which is why this will be kept for compatibility
> for these use cases.
>
> There were other users of VFIO which relied on group API but only for convenience
> purposes; no actual VFIO functionality depended on those API's. Therefore, group
> API's are removed and, where appropriate, replaced with the new API's.
>
> List of removed API's:
>
> * `rte_vfio_get_group_fd`
> * `rte_vfio_clear_group`
> * `rte_vfio_container_group_bind` (replaced by container assign API)
> * `rte_vfio_container_group_unbind`
> * `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
>
> 2. The API responsibilities aren't clear and bleed into each other
> ==================================================================
>
> Some API's do multiple things at once. In particular:
>
> * `rte_vfio_get_device_info` will setup the device
> * `rte_vfio_setup_device` will get device info
>
> These API's have been adjusted to do one thing only.
AI diagnosed apply failure.
This does not apply to main. Patches 8 and 11 have stale context:
patch 8 still has the rte_pci_unmap_device() call in
nfp_vdpa_vfio_setup() that 7185c65c95c6 ("vdpa/nfp: fix double PCI
unmap on unplug") removed in June, and patch 11 has the pre-1c726e28fd
wording of the RARP line in vhost_lib.rst. The series applies cleanly
at 481aeec3d7, so v9 looks to be based on a tree from before 2026-06-24.
Both are context-only; please rebase for v10.
Much of the stuff in cover letter needs to be release notes.
More AI (Claude Opus) comments:
Patch 13 (vfio: cleanup and refactor)
Error: container fd 0 is closed on the error path. CONTAINER_INITIALIZER
only initializes the mem_maps lock, so container_fd is 0, not -1. The
pre-refactor code initialized every vfio_cfgs[i].vfio_container_fd to -1.
vfio_container_erase() tests
if (cfg->container_fd >= 0 && close(cfg->container_fd))
so any path that allocates a container and fails before assigning the fd
closes stdin. rte_vfio_container_create() has three such paths (group
open failure, iommufd open failure, and the default: arm), as does
vfio_select_mode()'s err label. Add .container_fd = -1 to
CONTAINER_INITIALIZER, and use the same initializer in
vfio_container_erase() instead of (struct container){0}.
Error: double close in vfio_has_supported_extensions(). The VFIO_CHECK_
EXTENSION error path closes vfio_container_fd and returns -1, and the
caller vfio_group_open_container_fd() then closes it again:
ret = vfio_has_supported_extensions(vfio_container_fd);
if (ret) {
EAL_LOG(DEBUG, "No supported IOMMU extensions found!");
close(vfio_container_fd);
return -1;
}
Upstream had the close in the callee on both error paths and none in the
caller; the refactor added the caller close and removed only one of the
two callee closes. The fd may already have been handed out to another
thread by then. Drop the close() inside vfio_has_supported_extensions().
Warning: no release notes. The series makes the entire rte_vfio API
internal, removes the group-based API and rte_vfio_noiommu_is_enabled(),
and adds cdev mode. Only deprecation.rst is touched;
doc/guides/rel_notes/release_26_11.rst needs "Removed Items" and
"New Features" entries.
Patch 14 (bus/pci: use the new VFIO mode API)
Error: noiommu is no longer detected. pci_device_iova_mode() runs from
rte_pci_get_iommu_class() during rte_bus_get_iommu_class(), which
rte_eal_init() calls at line 689, well before rte_vfio_enable() at line
775. rte_vfio_get_mode() returns RTE_VFIO_MODE_NONE at that point, so
is_vfio_noiommu_enabled latches to 0 and IOVA VA is selected on a
noiommu system. The old rte_vfio_noiommu_is_enabled() read
/sys/module/vfio/parameters/enable_unsafe_noiommu_mode directly and was
therefore valid before VFIO init. The noiommu probe needs to stay
sysfs-based, or IOVA mode selection has to move after rte_vfio_enable().
Patch 15 (bus/fslmc: use the new VFIO mode API)
Error: fslmc bus scan now fails on every system. rte_fslmc_scan() is
called from rte_bus_scan() at eal.c:680, again before rte_vfio_enable().
The new check
if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
ret = -EINVAL;
goto scan_fail;
}
always sees RTE_VFIO_MODE_NONE, so no DPAA2 device is ever scanned. The
same ordering problem applies to rte_dpaa2_get_iommu_class() and to
fslmc_vfio_add_group(), which will now always pick VFIO_TYPE1_IOMMU.
Warning: the new scan_fail branch has no log message, unlike every other
failure in that function, so the failure is silent.
Patch 19 (vfio: introduce cdev mode)
Error: vfio_cfg.ops is never set in a cdev-mode secondary. The only
assignment is in vfio_cdev_enable(), reached solely from the primary arm
of vfio_select_mode(). The secondary path calls vfio_sync_mode() and
vfio_cdev_sync_ioas(), neither of which touches ops. A cdev secondary
therefore has ops == NULL and every rte_vfio_container_dma_map()/
dma_unmap() fails in vfio_dma_mem_map() with "VFIO support not
initialized". The same applies to a container made with
rte_vfio_container_create() in a secondary: it opens an iommufd and
allocates an IOAS but leaves ops NULL. Set vfio_cfg.ops to the iommufd
ops in the secondary cdev path as well.
Note also that container_dma_map()/container_dma_unmap() dereference
vfio_cfg.ops->partial_unmap without a NULL check; only vfio_dma_mem_map()
guards it.
Patch 5 (net/nbl: do not use VFIO group bind API)
Info: nbl_open_group_fd() needs a blank line between the declaration and
the statements:
static int
nbl_open_group_fd(int iommu_group_num)
{
char path[PATH_MAX];
snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
return open(path, O_RDWR);
}
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v10 00/20] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (3 preceding siblings ...)
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (20 more replies)
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (3 subsequent siblings)
8 siblings, 21 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
the API the way it was before.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v10:
- Added a patch that renames confusing error labels
- Fixed compiler warning about unused variable
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (20):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
bus/pci: rename mismatching error labels
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 10 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 47 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 21 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal_vfio.c | 2444 ++++++++-------------
lib/eal/linux/eal_vfio.h | 167 +-
lib/eal/linux/eal_vfio_cdev.c | 390 ++++
lib/eal/linux/eal_vfio_group.c | 984 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
35 files changed, 4263 insertions(+), 1903 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v10 01/20] uapi: update to v6.17 and add iommufd.h
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 02/20] vfio: make all functions internal Anatoly Burakov
` (19 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 02/20] vfio: make all functions internal
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
` (18 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ----
lib/eal/freebsd/eal.c | 30 +++++++++---------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++----------
lib/eal/windows/eal.c | 4 +--
5 files changed, 74 insertions(+), 45 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6ad7698c6b..8a82c815e3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,12 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 03/20] bus/pci: rename mismatching error labels
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 02/20] vfio: make all functions internal Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 04/20] vfio: split get device info from setup Anatoly Burakov
` (17 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Currently, there are two VFIO map functions - primary and secondary -
that use a very similar goto error pattern. However, the label names are
confusing.
Specifcally, for primary, we have three error labels: vfio_res (release
VFIO resource), vfio_dev_fd (close the fd), and map (unmap all the mapped
regions). For secondary, there is vfio_dev (close the fd), and
vfio_dev_fd (unmap all mapped regions).
For consistency, align secondary process function's error labels to match
semantics of how they work in primary process function.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..a67de6ea02 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -968,7 +968,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
- goto err_vfio_dev;
+ goto err_vfio_dev_fd;
/* map BARs */
maps = vfio_res->maps;
@@ -979,14 +979,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
if (ret < 0) {
PCI_LOG(ERR, "%s sparse mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
} else {
ret = pci_vfio_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
if (ret < 0) {
PCI_LOG(ERR, "%s mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
}
@@ -995,17 +995,17 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
/* we need save vfio_dev_fd, so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
if (rte_intr_dev_fd_set(dev->vfio_req_intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
return 0;
-err_vfio_dev_fd:
+err_map:
for (j = 0; j < i; j++) {
if (maps[j].addr)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
-err_vfio_dev:
+err_vfio_dev_fd:
rte_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 04/20] vfio: split get device info from setup
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (2 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 05/20] vfio: add container device assignment API Anatoly Burakov
` (16 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index a67de6ea02..bbe3490d0b 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 90d865a8df..6fde196d5e 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..f8ab932962 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 05/20] vfio: add container device assignment API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (3 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 04/20] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (15 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f8ab932962..0c64a62c5a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -944,3 +944,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..4890ecf06d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 06/20] net/nbl: do not use VFIO group bind API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (4 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 05/20] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 07/20] net/ntnic: use container device assignment API Anatoly Burakov
` (14 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 19 ++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..996573638d 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,14 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +432,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +544,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +557,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 07/20] net/ntnic: use container device assignment API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (5 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 08/20] vdpa/ifc: " Anatoly Burakov
` (13 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 08/20] vdpa/ifc: use container device assignment API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (6 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 07/20] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 09/20] vdpa/nfp: " Anatoly Burakov
` (12 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 09/20] vdpa/nfp: use container device assignment API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (7 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 08/20] vdpa/ifc: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 10/20] vdpa/sfc: " Anatoly Burakov
` (11 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4c0ab3ff30..778ab2b8e7 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -120,33 +118,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
@@ -157,7 +148,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1023,22 +1013,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1190,7 +1164,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 10/20] vdpa/sfc: use container device assignment API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (8 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 09/20] vdpa/nfp: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
` (10 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 11/20] vdpa/mlx5: remove group-related API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (9 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 10/20] vdpa/sfc: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 12/20] vhost: remove group-related API from driver Anatoly Burakov
` (9 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 12/20] vhost: remove group-related API from driver
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (10 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 13/20] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
lib/vhost/vdpa_driver.h | 3 ---
3 files changed, 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 345a621716..99355b8afd 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to respond to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 8a82c815e3..e4a754d037 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,10 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 13/20] vfio: remove group-based API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (11 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 12/20] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 14/20] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0c64a62c5a..cda72dfd1d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,13 +856,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -881,14 +874,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -905,24 +890,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 4890ecf06d..51ca1c162f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 14/20] vfio: cleanup and refactor
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (12 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 13/20] vfio: remove group-based API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 252 ++-
lib/eal/linux/eal_vfio.c | 2264 +++++++---------------
lib/eal/linux/eal_vfio.h | 142 +-
lib/eal/linux/eal_vfio_group.c | 984 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2104 insertions(+), 1643 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..3ca68ccf24 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bbe3490d0b..98f941dae0 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 996573638d..27e76bbe0f 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -414,7 +414,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index cda72dfd1d..6c1d1e3751 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -921,3 +921,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 51ca1c162f..6f21fa85ce 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -254,33 +327,46 @@ rte_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..715e329974 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,80 +24,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +312,106 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = (struct container){0};
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +426,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +446,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +456,535 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we would've liked to prevent user from assigning devices twice to
+ * prevent resource leaks, but for group mode this is not possible, as
+ * there is no way to know which fd belongs to which group/device.
+ *
+ * we also do not need to look in other configs as if we were to attempt
+ * to use a different container, the kernel wouldn't have allowed us to
+ * bind the group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1004,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1022,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1041,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1070,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1125,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1140,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1161,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1178,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1214,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1235,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1252,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1281,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..68d3a3ec6e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,161 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we only need to initialize the lock in mem maps */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +176,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..520e61610c
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,984 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ close(vfio_container_fd);
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 15/20] bus/pci: use the new VFIO mode API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (13 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 14/20] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 16/20] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..a1575b84e2 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 16/20] bus/fslmc: use the new VFIO mode API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (14 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 17/20] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 10 +++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 3626b12316..3d563257ca 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,13 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +507,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 3ca68ccf24..15273fcd57 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 17/20] net/hinic3: use the new VFIO mode API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (15 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 16/20] bus/fslmc: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 18/20] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..224be6f81c 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 18/20] net/ntnic: use the new VFIO mode API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (16 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 17/20] net/hinic3: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..8b6bca974c 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 19/20] vfio: remove no-IOMMU check API
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (17 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 18/20] net/ntnic: " Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-06 17:13 ` [PATCH v10 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6c1d1e3751..bb05a969a9 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,12 +850,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6f21fa85ce..6a4d101e33 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 715e329974..5263bea240 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1277,13 +1277,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v10 20/20] vfio: introduce cdev mode
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (18 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-06 14:11 ` Anatoly Burakov
2026-08-06 17:13 ` [PATCH v10 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-06 14:11 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 222 +++++++++++++++++-
lib/eal/linux/eal_vfio.h | 29 ++-
lib/eal/linux/eal_vfio_cdev.c | 390 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 721 insertions(+), 4 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index bb05a969a9..a84280a66c 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -931,3 +931,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6a4d101e33..b9c7184616 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 5263bea240..00dc9af78a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -347,6 +347,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -516,6 +530,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -662,6 +725,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -682,6 +788,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -754,6 +863,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -873,6 +1000,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -939,8 +1072,19 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- group_cfg->dma_setup_done = true;
+ if (mode == RTE_VFIO_MODE_CDEV) {
+ /* in cdev mode, sync ioas */
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+ } else if (mode == RTE_VFIO_MODE_GROUP ||
+ mode == RTE_VFIO_MODE_NOIOMMU) {
+ /* in group-like modes, primary handles DMA */
+ group_cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
@@ -957,6 +1101,19 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
@@ -971,6 +1128,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1110,6 +1268,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1309,6 +1501,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1367,6 +1578,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 68d3a3ec6e..52cb3a0e08 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -160,12 +171,24 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -176,6 +199,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..ce61a97853
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,390 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ /* cdev mode does not have different IOMMU ops */
+ vfio_cfg.ops = &iommufd_ops;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0)
+ return -1;
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v10 00/20] Support VFIO cdev API in DPDK
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
` (19 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 20/20] vfio: introduce cdev mode Anatoly Burakov
@ 2026-08-06 17:13 ` Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-08-06 17:13 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Thu, 6 Aug 2026 15:11:00 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> This patchset introduces a major refactor of the VFIO subsystem in DPDK to
> support character device (cdev) interface introduced in Linux kernel, as well as
> make the API more streamlined and useful. The goal is to simplify device
> management, improve compatibility, and clarify API responsibilities.
>
> The following sections outline the key issues addressed by this patchset and the
> corresponding changes introduced.
>
> 1. Only group mode is supported
> ===============================
>
> Since kernel version 4.14.327 (LTS), VFIO supports the new character device
> (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
> is a device-centric mode and does away with all the complexity regarding groups
> and IOMMU types, delegating it all to the kernel, and exposes a much simpler
> interface to userspace.
>
> The old group interface is still around, and will need to be kept in DPDK both
> for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
> driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
> the API the way it was before.
>
> To enable this, VFIO is heavily refactored, so that the code can support both
> modes while relying on (mostly) common infrastructure.
>
> Note that the existing `rte_vfio_device_setup/release` model is fundamentally
> incompatible with cdev mode, because for custom container cases, the expected
> flow is that the user binds the IOMMU group (and thus, implicitly, the device
> itself) to a specific container using `rte_vfio_container_group_bind`, whereas
> this step is not needed for cdev as the device fd is assigned to the container
> straight away.
>
> Therefore, what we do instead is introduce a new API for container device
> assignment which, semantically, will assign a device to specified container, so
> that when it is mapped using `rte_pci_map_device`, the appropriate container is
> selected. Under the hood though, we essentially transition to getting device fd
> straight away at assign stage, so that by the time the PCI bus attempts to map
> the device, it is already mapped and we just return an fd. There is no
> "unassign" API because `release_device` already performs that function.
>
> Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
> some introspection into VFIO's internals, with three new modes: group
> (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
> Although no-IOMMU is technically a variant of group mode, the distinction is
> largely irrelevant to the user, as all usages of noiommu checks in our codebase
> are for deciding whether to use IOVA or PA, not anything to do with managing
> groups. The current plan for kernel community is to *not* introduce no-IOMMU
> cdev implementation, and IOMMUFD's own group API compatibility layer also does
> not implement no-IOMMU mode, which is why this will be kept for compatibility
> for these use cases.
>
> There were other users of VFIO which relied on group API but only for convenience
> purposes; no actual VFIO functionality depended on those API's. Therefore, group
> API's are removed and, where appropriate, replaced with the new API's.
>
> List of removed API's:
>
> * `rte_vfio_get_group_fd`
> * `rte_vfio_clear_group`
> * `rte_vfio_container_group_bind` (replaced by container assign API)
> * `rte_vfio_container_group_unbind`
> * `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
>
> 2. The API responsibilities aren't clear and bleed into each other
> ==================================================================
>
> Some API's do multiple things at once. In particular:
>
> * `rte_vfio_get_device_info` will setup the device
> * `rte_vfio_setup_device` will get device info
>
> These API's have been adjusted to do one thing only.
>
> v10:
> - Added a patch that renames confusing error labels
> - Fixed compiler warning about unused variable
>
> v9:
> - Moved erroneous rte_errno-related comments to later in the patchset
> - Moved removal of vDPA group fd API's to their respective patches
> - Fixed typo in errno comments (ENXIO vs ENOXIO)
> - Fixed corruption of group config in secondary process (v8 AI review)
>
> v8:
> - Rebase
> - Fixed build errors due to variable shadowing
> - Removed duplicate fd check as kernel does not provide a way to distinguish
> between device fd's
>
> v7:
> - Rebase
> - Added removal of deprecation notices
> - Fixed implicit numeric comparison in patch 12
>
> v6:
> - Fixed missing header include in vfio cdev file
>
> v5:
> - Added back missing uapi patch
>
> v4:
> - Fixed issues with documenting rte_vfio_mode enum
> - Separated deprecation notices into a separate patchset
>
> v3:
> - Make API removal cleaner
> - Fix `get_group_num` usages to align with new API
> - Fix issues with function exports
> - Fix issues with `setup_device` returning old-style values in some cases
>
> v2:
> - Make the entire API internal
> - More aggressive API pruning, complete removal of group API
> - Fixed a bug in group mode where device could not be used
> - Better documentation and deprecation notice patches
> - Moved doc patches to beginning of patchset
>
> Anatoly Burakov (20):
> uapi: update to v6.17 and add iommufd.h
> vfio: make all functions internal
> bus/pci: rename mismatching error labels
> vfio: split get device info from setup
> vfio: add container device assignment API
> net/nbl: do not use VFIO group bind API
> net/ntnic: use container device assignment API
> vdpa/ifc: use container device assignment API
> vdpa/nfp: use container device assignment API
> vdpa/sfc: use container device assignment API
> vdpa/mlx5: remove group-related API
> vhost: remove group-related API from driver
> vfio: remove group-based API
> vfio: cleanup and refactor
> bus/pci: use the new VFIO mode API
> bus/fslmc: use the new VFIO mode API
> net/hinic3: use the new VFIO mode API
> net/ntnic: use the new VFIO mode API
> vfio: remove no-IOMMU check API
> vfio: introduce cdev mode
>
> config/arm/meson.build | 1 +
> config/meson.build | 1 +
> doc/guides/prog_guide/vhost_lib.rst | 4 -
> doc/guides/rel_notes/deprecation.rst | 10 -
> drivers/bus/cdx/cdx_vfio.c | 25 +-
> drivers/bus/fslmc/fslmc_bus.c | 10 +-
> drivers/bus/fslmc/fslmc_vfio.c | 6 +-
> drivers/bus/pci/linux/pci.c | 2 +-
> drivers/bus/pci/linux/pci_vfio.c | 47 +-
> drivers/bus/platform/platform.c | 9 +-
> drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
> drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
> drivers/net/nbl/nbl_common/nbl_userdev.c | 21 +-
> drivers/net/nbl/nbl_include/nbl_include.h | 1 +
> drivers/net/ntnic/ntnic_ethdev.c | 2 +-
> drivers/net/ntnic/ntnic_vfio.c | 30 +-
> drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
> drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
> drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
> drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
> drivers/vdpa/sfc/sfc_vdpa.h | 2 -
> kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
> kernel/linux/uapi/linux/vduse.h | 2 +-
> kernel/linux/uapi/linux/vfio.h | 12 +-
> kernel/linux/uapi/version | 2 +-
> lib/eal/freebsd/eal.c | 98 +-
> lib/eal/include/rte_vfio.h | 387 ++--
> lib/eal/linux/eal_vfio.c | 2444 ++++++++-------------
> lib/eal/linux/eal_vfio.h | 167 +-
> lib/eal/linux/eal_vfio_cdev.c | 390 ++++
> lib/eal/linux/eal_vfio_group.c | 984 +++++++++
> lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
> lib/eal/linux/meson.build | 2 +
> lib/eal/windows/eal.c | 4 +-
> lib/vhost/vdpa_driver.h | 3 -
> 35 files changed, 4263 insertions(+), 1903 deletions(-)
> create mode 100644 kernel/linux/uapi/linux/iommufd.h
> create mode 100644 lib/eal/linux/eal_vfio_cdev.c
> create mode 100644 lib/eal/linux/eal_vfio_group.c
>
Still lots of open AI review feedback:
Patch 14/20 (vfio: cleanup and refactor)
Error: close(fd 0) on the container error path. CONTAINER_INITIALIZER in
eal_vfio.h sets only the mem_maps lock, so container_fd is 0:
#define CONTAINER_INITIALIZER \
((struct container){ \
.mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
})
vfio_container_erase() then does
if (cfg->container_fd >= 0 && close(cfg->container_fd))
so any container allocated but not yet given an fd closes stdin on
teardown. rte_vfio_container_create() reaches this from three arms
(group open failure, iommufd open failure, default:), and
vfio_select_mode()'s err label reaches it too. The pre-refactor code
initialized every vfio_cfgs[i].vfio_container_fd to -1. Add
.container_fd = -1 to CONTAINER_INITIALIZER, and use that initializer in
vfio_container_erase() rather than (struct container){0}.
Error: double close in vfio_has_supported_extensions(). The
VFIO_CHECK_EXTENSION failure path closes vfio_container_fd and returns
-1, and vfio_group_open_container_fd() closes it again:
ret = vfio_has_supported_extensions(vfio_container_fd);
if (ret) {
EAL_LOG(DEBUG, "No supported IOMMU extensions found!");
close(vfio_container_fd);
return -1;
}
Upstream had the close in the callee on both of its error paths and none
in the caller. The refactor added the caller close but removed only the
n_extensions == 0 close, leaving the ioctl path closing twice. Drop the
close() inside vfio_has_supported_extensions().
Warning: no release notes. The series makes the whole rte_vfio API
internal, removes five public functions, and adds cdev mode. Only
deprecation.rst is touched; release_26_11.rst needs "Removed Items" and
"New Features" entries.
Patch 15/20 (bus/pci: use the new VFIO mode API)
Error: noiommu is never detected. pci_device_iova_mode() runs under
rte_bus_get_iommu_class() at eal.c:689; rte_vfio_enable() does not run
until :775. rte_vfio_get_mode() returns RTE_VFIO_MODE_NONE at that
point, so is_vfio_noiommu_enabled latches to 0 and IOVA VA is chosen on
a noiommu system. rte_vfio_noiommu_is_enabled() read the sysfs knob and
did not depend on VFIO init.
Patch 16/20 (bus/fslmc: use the new VFIO mode API)
Error: fslmc bus scan fails on every system. rte_fslmc_scan() runs from
rte_bus_scan() at eal.c:680, also before rte_vfio_enable(), so
if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
ret = -EINVAL;
goto scan_fail;
}
always takes the failure branch and no DPAA2 device is ever scanned. The
same ordering issue applies to rte_dpaa2_get_iommu_class() and to
fslmc_vfio_add_group(), which will now always select VFIO_TYPE1_IOMMU.
Warning: the new scan_fail branch logs nothing, unlike every other
failure in that function, so the failure is silent.
Patch 20/20 (vfio: introduce cdev mode)
Error: vfio_cfg.ops is never set in a cdev-mode secondary. The only
assignment is in vfio_cdev_enable(), reached solely from the primary arm
of vfio_select_mode(); the secondary arm calls vfio_sync_mode() and
vfio_cdev_sync_ioas(), neither of which touches ops. Every
rte_vfio_container_dma_map()/dma_unmap() in a cdev secondary therefore
fails in vfio_dma_mem_map() with "VFIO support not initialized". A
container made by rte_vfio_container_create() in a secondary has the
same problem: iommufd opened, IOAS allocated, ops NULL. Set ops in the
secondary cdev path.
Note also that container_dma_map()/container_dma_unmap() dereference
vfio_cfg.ops->partial_unmap with no NULL check; only vfio_dma_mem_map()
guards it.
Patch 3/20 (bus/pci: rename mismatching error labels)
Info: "Specifcally" in the commit body.
Patch 6/20 (net/nbl: do not use VFIO group bind API)
Info: blank line needed between the declaration and the statements in
nbl_open_group_fd().
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v11 00/20] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (4 preceding siblings ...)
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (20 more replies)
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (2 subsequent siblings)
8 siblings, 21 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
the API the way it was before.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v11:
- Addressed feedback from Stephen's AI review:
- Use CONTAINER_INITIALIZER for reset
- Set container fd to -1 in CONTAINER_INITIALIZER
- Fixed double close()
- Moved VFIO init to earlier in init sequence to account for
bus drivers needing no-IOMMU mode status
- Fixed missing ops set for cdev mode, and missing ops reset
- Fixed fd leak on failed attach in cdev
v10:
- Added a patch that renames confusing error labels
- Fixed compiler warning about unused variable
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (20):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
bus/pci: rename mismatching error labels
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 11 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 47 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 22 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 98 +-
lib/eal/include/rte_vfio.h | 387 ++--
lib/eal/linux/eal.c | 15 +-
lib/eal/linux/eal_vfio.c | 2448 ++++++++-------------
lib/eal/linux/eal_vfio.h | 169 +-
lib/eal/linux/eal_vfio_cdev.c | 396 ++++
lib/eal/linux/eal_vfio_group.c | 983 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
36 files changed, 4286 insertions(+), 1908 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v11 01/20] uapi: update to v6.17 and add iommufd.h
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 02/20] vfio: make all functions internal Anatoly Burakov
` (19 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 02/20] vfio: make all functions internal
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
` (18 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ----
lib/eal/freebsd/eal.c | 30 +++++++++---------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++----------
lib/eal/windows/eal.c | 4 +--
5 files changed, 74 insertions(+), 45 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6ad7698c6b..8a82c815e3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,12 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 03/20] bus/pci: rename mismatching error labels
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 02/20] vfio: make all functions internal Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 04/20] vfio: split get device info from setup Anatoly Burakov
` (17 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Currently, there are two VFIO map functions - primary and secondary -
that use a very similar goto error pattern. However, the label names are
confusing.
Specifically, for primary, we have three error labels: vfio_res (release
VFIO resource), vfio_dev_fd (close the fd), and map (unmap all the mapped
regions). For secondary, there is vfio_dev (close the fd), and
vfio_dev_fd (unmap all mapped regions).
For consistency, align secondary process function's error labels to match
semantics of how they work in primary process function.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..a67de6ea02 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -968,7 +968,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
- goto err_vfio_dev;
+ goto err_vfio_dev_fd;
/* map BARs */
maps = vfio_res->maps;
@@ -979,14 +979,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
if (ret < 0) {
PCI_LOG(ERR, "%s sparse mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
} else {
ret = pci_vfio_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
if (ret < 0) {
PCI_LOG(ERR, "%s mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
}
@@ -995,17 +995,17 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
/* we need save vfio_dev_fd, so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
if (rte_intr_dev_fd_set(dev->vfio_req_intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
return 0;
-err_vfio_dev_fd:
+err_map:
for (j = 0; j < i; j++) {
if (maps[j].addr)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
-err_vfio_dev:
+err_vfio_dev_fd:
rte_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 04/20] vfio: split get device info from setup
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (2 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 05/20] vfio: add container device assignment API Anatoly Burakov
` (16 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index a67de6ea02..bbe3490d0b 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 90d865a8df..6fde196d5e 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..f8ab932962 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 05/20] vfio: add container device assignment API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (3 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 04/20] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (15 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f8ab932962..0c64a62c5a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -944,3 +944,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..4890ecf06d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 06/20] net/nbl: do not use VFIO group bind API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (4 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 05/20] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 07/20] net/ntnic: use container device assignment API Anatoly Burakov
` (14 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..54326e44e1 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,15 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +433,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +545,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +558,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 07/20] net/ntnic: use container device assignment API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (5 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 08/20] vdpa/ifc: " Anatoly Burakov
` (13 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 08/20] vdpa/ifc: use container device assignment API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (6 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 07/20] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 09/20] vdpa/nfp: " Anatoly Burakov
` (12 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 09/20] vdpa/nfp: use container device assignment API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (7 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 08/20] vdpa/ifc: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 10/20] vdpa/sfc: " Anatoly Burakov
` (11 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4c0ab3ff30..778ab2b8e7 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -120,33 +118,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
@@ -157,7 +148,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1023,22 +1013,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1190,7 +1164,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 10/20] vdpa/sfc: use container device assignment API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (8 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 09/20] vdpa/nfp: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
` (10 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 11/20] vdpa/mlx5: remove group-related API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (9 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 10/20] vdpa/sfc: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 12/20] vhost: remove group-related API from driver Anatoly Burakov
` (9 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 12/20] vhost: remove group-related API from driver
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (10 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 13/20] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
lib/vhost/vdpa_driver.h | 3 ---
3 files changed, 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 345a621716..99355b8afd 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to respond to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 8a82c815e3..e4a754d037 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,10 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 13/20] vfio: remove group-based API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (11 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 12/20] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 14/20] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0c64a62c5a..cda72dfd1d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,13 +856,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -881,14 +874,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -905,24 +890,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 4890ecf06d..51ca1c162f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 14/20] vfio: cleanup and refactor
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (12 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 13/20] vfio: remove group-based API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 252 ++-
lib/eal/linux/eal_vfio.c | 2264 +++++++---------------
lib/eal/linux/eal_vfio.h | 143 +-
lib/eal/linux/eal_vfio_group.c | 983 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2104 insertions(+), 1643 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..3ca68ccf24 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bbe3490d0b..98f941dae0 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 54326e44e1..3e0232ecbb 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -415,7 +415,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index cda72dfd1d..6c1d1e3751 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -921,3 +921,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 51ca1c162f..6f21fa85ce 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -254,33 +327,46 @@ rte_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..6bac817111 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,80 +24,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENOXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +312,106 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = CONTAINER_INITIALIZER;
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +426,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +446,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +456,535 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!group_cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we would've liked to prevent user from assigning devices twice to
+ * prevent resource leaks, but for group mode this is not possible, as
+ * there is no way to know which fd belongs to which group/device.
+ *
+ * we also do not need to look in other configs as if we were to attempt
+ * to use a different container, the kernel wouldn't have allowed us to
+ * bind the group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request iommufd from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ group_cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1004,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1022,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1041,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1070,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1125,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1140,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1161,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1178,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1214,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1235,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1252,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1281,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..93dd2d266e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,162 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool dma_setup_done;
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we need to initialize the fd, and mem maps lock */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ .container_fd = -1, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +177,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..9ba3525278
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,983 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ group_cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 15/20] bus/pci: use the new VFIO mode API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (13 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 14/20] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 16/20] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status. This requires moving
the VFIO init earlier in the init sequence, as now checking no-IOMMU mode
is not a sysfs check but instead relies on full VFIO subsystem to be
initialized and available.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
lib/eal/linux/eal.c | 15 ++++++++++-----
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..a1575b84e2 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index fc2e9b8c0e..5bc095bd61 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -677,6 +677,16 @@ rte_eal_init(int argc, char **argv)
}
}
+ /*
+ * VFIO must be initialized before bus scan because buses need to know
+ * about no-IOMMU mode status.
+ */
+ if (rte_vfio_enable("vfio")) {
+ rte_eal_init_alert("Cannot init VFIO");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
if (rte_bus_scan()) {
rte_eal_init_alert("Cannot scan the buses for devices");
rte_errno = ENODEV;
@@ -772,11 +782,6 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (rte_vfio_enable("vfio")) {
- rte_eal_init_alert("Cannot init VFIO");
- rte_errno = EAGAIN;
- goto err_out;
- }
/* in secondary processes, memory init may allocate additional fbarrays
* not present in primary processes, so to avoid any potential issues,
* initialize memzones first.
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 16/20] bus/fslmc: use the new VFIO mode API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (14 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 17/20] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 11 ++++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 3626b12316..7cc2375fca 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,14 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ DPAA2_BUS_DEBUG("DPAA2: VFIO not in group mode");
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +508,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 3ca68ccf24..15273fcd57 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 17/20] net/hinic3: use the new VFIO mode API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (15 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 16/20] bus/fslmc: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 18/20] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..224be6f81c 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 18/20] net/ntnic: use the new VFIO mode API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (16 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 17/20] net/hinic3: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..8b6bca974c 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 19/20] vfio: remove no-IOMMU check API
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (17 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 18/20] net/ntnic: " Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-11 0:30 ` [PATCH v11 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6c1d1e3751..bb05a969a9 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,12 +850,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6f21fa85ce..6a4d101e33 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 6bac817111..7eeaba012e 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1277,13 +1277,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v11 20/20] vfio: introduce cdev mode
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (18 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-07 14:46 ` Anatoly Burakov
2026-08-11 0:30 ` [PATCH v11 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-07 14:46 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
The implementation largely follows the kernel reference, with most
differences concentrated in how the container and the devices are set up.
The VFIO container allocates an "IOAS" (IO address space), maps the memory
into that space, and attaches devices to IOAS - there is no IOMMU type
discovery at device attach time.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 +
lib/eal/include/rte_vfio.h | 31 +++
lib/eal/linux/eal_vfio.c | 226 +++++++++++++++++-
lib/eal/linux/eal_vfio.h | 30 ++-
lib/eal/linux/eal_vfio_cdev.c | 396 +++++++++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 ++++
lib/eal/linux/meson.build | 1 +
7 files changed, 732 insertions(+), 4 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index bb05a969a9..a84280a66c 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -931,3 +931,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 6a4d101e33..b9c7184616 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -197,6 +201,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENOXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7eeaba012e..21ef27d92d 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -347,6 +347,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -516,6 +530,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -662,6 +725,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -682,6 +788,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -754,6 +863,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -873,6 +1000,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -939,8 +1072,20 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- group_cfg->dma_setup_done = true;
+ if (mode == RTE_VFIO_MODE_CDEV) {
+ vfio_cdev_setup_ops();
+ /* in cdev mode, sync ioas */
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+ } else if (mode == RTE_VFIO_MODE_GROUP ||
+ mode == RTE_VFIO_MODE_NOIOMMU) {
+ /* in group-like modes, primary handles DMA */
+ group_cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
@@ -957,10 +1102,26 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* try cdev mode */
+ if (vfio_cdev_enable(cfg) == 0) {
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ goto err_mpsync;
+ if (vfio_register_mem_event_callback() < 0)
+ goto err_mpsync;
+
+ return RTE_VFIO_MODE_CDEV;
+ }
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
vfio_container_erase(cfg);
+ /* reset ops as well */
+ vfio_cfg.ops = NULL;
return RTE_VFIO_MODE_NONE;
}
@@ -971,6 +1132,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1110,6 +1272,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1309,6 +1505,25 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1367,6 +1582,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 93dd2d266e..d2dc06307e 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -61,12 +64,20 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -161,12 +172,25 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+void vfio_cdev_setup_ops(void);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -177,6 +201,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..a659891b60
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,396 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+void
+vfio_cdev_setup_ops(void)
+{
+ vfio_cfg.ops = &iommufd_ops;
+}
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0) {
+ close(device_fd);
+ dev->fd = -1;
+ return -1;
+ }
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v11 00/20] Support VFIO cdev API in DPDK
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
` (19 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 20/20] vfio: introduce cdev mode Anatoly Burakov
@ 2026-08-11 0:30 ` Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-08-11 0:30 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Fri, 7 Aug 2026 15:46:09 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> This patchset introduces a major refactor of the VFIO subsystem in DPDK to
> support character device (cdev) interface introduced in Linux kernel, as well as
> make the API more streamlined and useful. The goal is to simplify device
> management, improve compatibility, and clarify API responsibilities.
>
> The following sections outline the key issues addressed by this patchset and the
> corresponding changes introduced.
>
> 1. Only group mode is supported
> ===============================
>
> Since kernel version 4.14.327 (LTS), VFIO supports the new character device
> (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
> is a device-centric mode and does away with all the complexity regarding groups
> and IOMMU types, delegating it all to the kernel, and exposes a much simpler
> interface to userspace.
>
> The old group interface is still around, and will need to be kept in DPDK both
> for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
> driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
> the API the way it was before.
>
> To enable this, VFIO is heavily refactored, so that the code can support both
> modes while relying on (mostly) common infrastructure.
>
> Note that the existing `rte_vfio_device_setup/release` model is fundamentally
> incompatible with cdev mode, because for custom container cases, the expected
> flow is that the user binds the IOMMU group (and thus, implicitly, the device
> itself) to a specific container using `rte_vfio_container_group_bind`, whereas
> this step is not needed for cdev as the device fd is assigned to the container
> straight away.
>
> Therefore, what we do instead is introduce a new API for container device
> assignment which, semantically, will assign a device to specified container, so
> that when it is mapped using `rte_pci_map_device`, the appropriate container is
> selected. Under the hood though, we essentially transition to getting device fd
> straight away at assign stage, so that by the time the PCI bus attempts to map
> the device, it is already mapped and we just return an fd. There is no
> "unassign" API because `release_device` already performs that function.
>
> Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
> some introspection into VFIO's internals, with three new modes: group
> (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
> Although no-IOMMU is technically a variant of group mode, the distinction is
> largely irrelevant to the user, as all usages of noiommu checks in our codebase
> are for deciding whether to use IOVA or PA, not anything to do with managing
> groups. The current plan for kernel community is to *not* introduce no-IOMMU
> cdev implementation, and IOMMUFD's own group API compatibility layer also does
> not implement no-IOMMU mode, which is why this will be kept for compatibility
> for these use cases.
>
> There were other users of VFIO which relied on group API but only for convenience
> purposes; no actual VFIO functionality depended on those API's. Therefore, group
> API's are removed and, where appropriate, replaced with the new API's.
>
> List of removed API's:
>
> * `rte_vfio_get_group_fd`
> * `rte_vfio_clear_group`
> * `rte_vfio_container_group_bind` (replaced by container assign API)
> * `rte_vfio_container_group_unbind`
> * `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
>
> 2. The API responsibilities aren't clear and bleed into each other
> ==================================================================
>
> Some API's do multiple things at once. In particular:
>
> * `rte_vfio_get_device_info` will setup the device
> * `rte_vfio_setup_device` will get device info
>
> These API's have been adjusted to do one thing only.
>
> v11:
> - Addressed feedback from Stephen's AI review:
> - Use CONTAINER_INITIALIZER for reset
> - Set container fd to -1 in CONTAINER_INITIALIZER
> - Fixed double close()
> - Moved VFIO init to earlier in init sequence to account for
> bus drivers needing no-IOMMU mode status
> - Fixed missing ops set for cdev mode, and missing ops reset
> - Fixed fd leak on failed attach in cdev
>
> v10:
> - Added a patch that renames confusing error labels
> - Fixed compiler warning about unused variable
>
> v9:
> - Moved erroneous rte_errno-related comments to later in the patchset
> - Moved removal of vDPA group fd API's to their respective patches
> - Fixed typo in errno comments (ENXIO vs ENOXIO)
> - Fixed corruption of group config in secondary process (v8 AI review)
>
> v8:
> - Rebase
> - Fixed build errors due to variable shadowing
> - Removed duplicate fd check as kernel does not provide a way to distinguish
> between device fd's
>
> v7:
> - Rebase
> - Added removal of deprecation notices
> - Fixed implicit numeric comparison in patch 12
>
> v6:
> - Fixed missing header include in vfio cdev file
>
> v5:
> - Added back missing uapi patch
>
> v4:
> - Fixed issues with documenting rte_vfio_mode enum
> - Separated deprecation notices into a separate patchset
>
> v3:
> - Make API removal cleaner
> - Fix `get_group_num` usages to align with new API
> - Fix issues with function exports
> - Fix issues with `setup_device` returning old-style values in some cases
>
> v2:
> - Make the entire API internal
> - More aggressive API pruning, complete removal of group API
> - Fixed a bug in group mode where device could not be used
> - Better documentation and deprecation notice patches
> - Moved doc patches to beginning of patchset
>
> Anatoly Burakov (20):
> uapi: update to v6.17 and add iommufd.h
> vfio: make all functions internal
> bus/pci: rename mismatching error labels
> vfio: split get device info from setup
> vfio: add container device assignment API
> net/nbl: do not use VFIO group bind API
> net/ntnic: use container device assignment API
> vdpa/ifc: use container device assignment API
> vdpa/nfp: use container device assignment API
> vdpa/sfc: use container device assignment API
> vdpa/mlx5: remove group-related API
> vhost: remove group-related API from driver
> vfio: remove group-based API
> vfio: cleanup and refactor
> bus/pci: use the new VFIO mode API
> bus/fslmc: use the new VFIO mode API
> net/hinic3: use the new VFIO mode API
> net/ntnic: use the new VFIO mode API
> vfio: remove no-IOMMU check API
> vfio: introduce cdev mode
>
> config/arm/meson.build | 1 +
> config/meson.build | 1 +
> doc/guides/prog_guide/vhost_lib.rst | 4 -
> doc/guides/rel_notes/deprecation.rst | 10 -
> drivers/bus/cdx/cdx_vfio.c | 25 +-
> drivers/bus/fslmc/fslmc_bus.c | 11 +-
> drivers/bus/fslmc/fslmc_vfio.c | 6 +-
> drivers/bus/pci/linux/pci.c | 2 +-
> drivers/bus/pci/linux/pci_vfio.c | 47 +-
> drivers/bus/platform/platform.c | 9 +-
> drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
> drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
> drivers/net/nbl/nbl_common/nbl_userdev.c | 22 +-
> drivers/net/nbl/nbl_include/nbl_include.h | 1 +
> drivers/net/ntnic/ntnic_ethdev.c | 2 +-
> drivers/net/ntnic/ntnic_vfio.c | 30 +-
> drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
> drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
> drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
> drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
> drivers/vdpa/sfc/sfc_vdpa.h | 2 -
> kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
> kernel/linux/uapi/linux/vduse.h | 2 +-
> kernel/linux/uapi/linux/vfio.h | 12 +-
> kernel/linux/uapi/version | 2 +-
> lib/eal/freebsd/eal.c | 98 +-
> lib/eal/include/rte_vfio.h | 387 ++--
> lib/eal/linux/eal.c | 15 +-
> lib/eal/linux/eal_vfio.c | 2448 ++++++++-------------
> lib/eal/linux/eal_vfio.h | 169 +-
> lib/eal/linux/eal_vfio_cdev.c | 396 ++++
> lib/eal/linux/eal_vfio_group.c | 983 +++++++++
> lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
> lib/eal/linux/meson.build | 2 +
> lib/eal/windows/eal.c | 4 +-
> lib/vhost/vdpa_driver.h | 3 -
> 36 files changed, 4286 insertions(+), 1908 deletions(-)
> create mode 100644 kernel/linux/uapi/linux/iommufd.h
> create mode 100644 lib/eal/linux/eal_vfio_cdev.c
> create mode 100644 lib/eal/linux/eal_vfio_group.c
>
Still lots of AI feedback from Claude Opus 5
Patch 20/20 (vfio: introduce cdev mode)
Error: in cdev mode, DPDK memory is never DMA mapped under --legacy-mem.
vfio_select_mode()'s cdev arm calls vfio_setup_dma_mem(cfg), which walks
memsegs via rte_memseg_walk(). rte_vfio_enable() now runs at eal.c:684,
but rte_eal_memory_init() is not until :797, so every memseg list still
has memseg_arr.count == 0 and the walk maps nothing. The mem event
callback registered on the next line covers dynamic mode, but legacy
memory is populated by malloc_add_seg(), which does not call
eal_memalloc_mem_event_notify(). vfio_cdev_setup_device() does no DMA
setup either, so nothing ever maps the initial hugepages.
Group mode is unaffected because vfio_group_assign_device() calls
vfio_setup_dma_mem() lazily at first device assign, which happens after
memory init. The cdev path needs the same deferral, or an explicit
re-walk once memory is up.
Warning: ENOXIO is not an errno. The doc comments introduced in patches
14 and 20 list
* - ENOXIO - VFIO support not initialized.
in twelve places in lib/eal/include/rte_vfio.h. The code sets ENXIO.
Warning: no release notes. The series makes the whole rte_vfio API
internal, removes the group-based API and rte_vfio_noiommu_is_enabled(),
and adds cdev mode. Only deprecation.rst is touched;
doc/guides/rel_notes/release_26_11.rst needs "Removed Items" and
"New Features" entries.
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v12 00/20] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (5 preceding siblings ...)
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
@ 2026-08-25 13:20 ` Anatoly Burakov
2026-08-25 13:20 ` [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (20 more replies)
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
8 siblings, 21 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:20 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, and clarify API responsibilities.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
the API the way it was before.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v12:
- Addressed feedback from Stephen's AI review:
- Add release notes updates to patch 2 and 20
- Fix ENXIO typos
- Added a new init step after memory init to enable DMA mapping for cdev
v11:
- Addressed feedback from Stephen's AI review:
- Use CONTAINER_INITIALIZER for reset
- Set container fd to -1 in CONTAINER_INITIALIZER
- Fixed double close()
- Moved VFIO init to earlier in init sequence to account for
bus drivers needing no-IOMMU mode status
- Fixed missing ops set for cdev mode, and missing ops reset
- Fixed fd leak on failed attach in cdev
v10:
- Added a patch that renames confusing error labels
- Fixed compiler warning about unused variable
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (20):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
bus/pci: rename mismatching error labels
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
doc/guides/rel_notes/release_26_11.rst | 13 +
drivers/bus/cdx/cdx_vfio.c | 25 +-
drivers/bus/fslmc/fslmc_bus.c | 11 +-
drivers/bus/fslmc/fslmc_vfio.c | 6 +-
drivers/bus/pci/linux/pci.c | 2 +-
drivers/bus/pci/linux/pci_vfio.c | 47 +-
drivers/bus/platform/platform.c | 9 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 22 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
drivers/net/ntnic/ntnic_vfio.c | 30 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 104 +-
lib/eal/include/rte_vfio.h | 401 ++--
lib/eal/linux/eal.c | 22 +-
lib/eal/linux/eal_vfio.c | 2471 ++++++++-------------
lib/eal/linux/eal_vfio.h | 169 +-
lib/eal/linux/eal_vfio_cdev.c | 396 ++++
lib/eal/linux/eal_vfio_group.c | 983 ++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 4 +-
lib/vhost/vdpa_driver.h | 3 -
37 files changed, 4350 insertions(+), 1907 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
@ 2026-08-25 13:20 ` Anatoly Burakov
2026-08-25 14:08 ` Stephen Hemminger
2026-08-25 13:20 ` [PATCH v12 02/20] vfio: make all functions internal Anatoly Burakov
` (19 subsequent siblings)
20 siblings, 1 reply; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:20 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 02/20] vfio: make all functions internal
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
2026-08-25 13:20 ` [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-25 13:20 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
` (18 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:20 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/deprecation.rst | 6 ----
doc/guides/rel_notes/release_26_11.rst | 8 +++++
lib/eal/freebsd/eal.c | 30 ++++++++--------
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++---------
lib/eal/windows/eal.c | 4 +--
6 files changed, 82 insertions(+), 45 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6ad7698c6b..8a82c815e3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,12 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index c8cc86295d..b5793b5c94 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -93,6 +93,10 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public API.
+
ABI Changes
-----------
@@ -109,6 +113,10 @@ ABI Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public ABI.
+
Known Issues
------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 03/20] bus/pci: rename mismatching error labels
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
2026-08-25 13:20 ` [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-25 13:20 ` [PATCH v12 02/20] vfio: make all functions internal Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 04/20] vfio: split get device info from setup Anatoly Burakov
` (17 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Currently, there are two VFIO map functions - primary and secondary -
that use a very similar goto error pattern. However, the label names are
confusing.
Specifically, for primary, we have three error labels: vfio_res (release
VFIO resource), vfio_dev_fd (close the fd), and map (unmap all the mapped
regions). For secondary, there is vfio_dev (close the fd), and
vfio_dev_fd (unmap all mapped regions).
For consistency, align secondary process function's error labels to match
semantics of how they work in primary process function.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..a67de6ea02 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -968,7 +968,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
- goto err_vfio_dev;
+ goto err_vfio_dev_fd;
/* map BARs */
maps = vfio_res->maps;
@@ -979,14 +979,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
if (ret < 0) {
PCI_LOG(ERR, "%s sparse mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
} else {
ret = pci_vfio_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
if (ret < 0) {
PCI_LOG(ERR, "%s mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
}
@@ -995,17 +995,17 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
/* we need save vfio_dev_fd, so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
if (rte_intr_dev_fd_set(dev->vfio_req_intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
return 0;
-err_vfio_dev_fd:
+err_map:
for (j = 0; j < i; j++) {
if (maps[j].addr)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
-err_vfio_dev:
+err_vfio_dev_fd:
rte_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 04/20] vfio: split get device info from setup
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (2 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 05/20] vfio: add container device assignment API Anatoly Burakov
` (16 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/rte_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..9bae264409 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index a67de6ea02..bbe3490d0b 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (rte_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 90d865a8df..6fde196d5e 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = rte_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..d00aaf1bb7 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = rte_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..f8ab932962 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..fb666141f6 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -187,19 +184,16 @@ rte_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with rte_vfio_setup_device() first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -207,8 +201,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..47c973e49a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -758,7 +758,7 @@ rte_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -975,7 +975,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -992,18 +992,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1217,26 +1207,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 05/20] vfio: add container device assignment API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (3 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 04/20] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (15 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `rte_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/rte_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index f8ab932962..0c64a62c5a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -944,3 +944,13 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index fb666141f6..4890ecf06d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -271,6 +271,32 @@ __rte_internal
int
rte_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `rte_vfio_setup_device()` to set up the device with the VFIO container
+ * specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 47c973e49a..02fec64658 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2102,6 +2102,38 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = rte_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 06/20] net/nbl: do not use VFIO group bind API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (4 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 05/20] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 07/20] net/ntnic: use container device assignment API Anatoly Burakov
` (14 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 20 +++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 96f0d2e264..54326e44e1 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,15 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+
+ snprintf(path, sizeof(path), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +433,12 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +545,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
rte_vfio_container_destroy(container);
@@ -549,17 +558,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
close(common->devfd);
rte_mcfg_mem_read_lock();
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 07/20] net/ntnic: use container device assignment API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (5 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 08/20] vdpa/ifc: " Anatoly Burakov
` (13 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..c746b300b2 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = rte_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 08/20] vdpa/ifc: use container device assignment API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (6 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 07/20] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 09/20] vdpa/nfp: " Anatoly Burakov
` (12 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..63f4172da5 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = rte_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (rte_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 09/20] vdpa/nfp: use container device assignment API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (7 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 08/20] vdpa/ifc: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 10/20] vdpa/sfc: " Anatoly Burakov
` (11 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4c0ab3ff30..778ab2b8e7 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -120,33 +118,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = rte_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = rte_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
rte_vfio_container_destroy(device->vfio_container_fd);
@@ -157,7 +148,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
rte_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1023,22 +1013,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1190,7 +1164,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 10/20] vdpa/sfc: use container device assignment API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (8 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 09/20] vdpa/nfp: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
` (10 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..99b4ced3f4 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = rte_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 11/20] vdpa/mlx5: remove group-related API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (9 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 10/20] vdpa/sfc: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 12/20] vhost: remove group-related API from driver Anatoly Burakov
` (9 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 12/20] vhost: remove group-related API from driver
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (10 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 13/20] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
lib/vhost/vdpa_driver.h | 3 ---
3 files changed, 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 345a621716..99355b8afd 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to respond to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 8a82c815e3..e4a754d037 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,10 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 13/20] vfio: remove group-based API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (11 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 12/20] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 14/20] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/rte_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0c64a62c5a..cda72dfd1d 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,13 +856,6 @@ int rte_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -881,14 +874,6 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
-int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
@@ -905,24 +890,6 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 4890ecf06d..51ca1c162f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -136,24 +136,6 @@ int rte_vfio_is_enabled(const char *modname);
__rte_internal
int rte_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-rte_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -218,24 +200,6 @@ __rte_internal
int
rte_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-rte_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -297,42 +261,6 @@ int
rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 02fec64658..7893d334eb 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -67,6 +67,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -532,9 +535,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -731,9 +733,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int
-rte_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -787,7 +788,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -813,14 +814,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -841,7 +842,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -865,7 +866,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -882,7 +883,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -951,7 +952,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -988,7 +989,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1026,9 +1027,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1064,7 +1065,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2091,7 +2092,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2122,7 +2123,7 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = rte_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2134,9 +2135,8 @@ rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
-int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2149,9 +2149,8 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
-int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..3eaeef2fc8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 14/20] vfio: cleanup and refactor
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (12 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 13/20] vfio: remove group-based API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple rte_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, rte_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
This will change behavior of the following functions:
- `rte_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `rte_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `rte_vfio_setup_device` and `rte_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 252 ++-
lib/eal/linux/eal_vfio.c | 2262 +++++++---------------
lib/eal/linux/eal_vfio.h | 143 +-
lib/eal/linux/eal_vfio_group.c | 983 ++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 38 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2102 insertions(+), 1643 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 9bae264409..873f0f3460 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <rte_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..3ca68ccf24 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -364,9 +364,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bbe3490d0b..98f941dae0 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <rte_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = rte_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index d00aaf1bb7..92d8de4443 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <rte_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = rte_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index 54326e44e1..3e0232ecbb 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -415,7 +415,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index cda72dfd1d..6c1d1e3751 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -921,3 +921,19 @@ rte_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+int
+rte_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return RTE_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 51ca1c162f..0af41c3610 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,49 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum rte_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - RTE_VFIO_MODE_NONE: VFIO is not enabled.
+ * - RTE_VFIO_MODE_GROUP: Legacy group mode.
+ * - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum rte_vfio_mode {
+ RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ RTE_VFIO_MODE_GROUP, /**< Group mode */
+ RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `rte_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,99 +89,127 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `rte_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int rte_vfio_enable(const char *modname);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+enum rte_vfio_mode
+rte_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+rte_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -179,7 +228,12 @@ rte_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -187,14 +241,17 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -202,7 +259,9 @@ rte_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -213,8 +272,14 @@ rte_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -222,14 +287,22 @@ rte_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -254,33 +327,46 @@ rte_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+rte_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -291,22 +377,26 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use RTE_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7893d334eb..660a28c90e 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,80 +24,39 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
};
-struct vfio_config {
- int vfio_enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct container containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_cfg = {
+ .mode = RTE_VFIO_MODE_NONE,
+ .default_cfg = &containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
is_null_map(const struct user_mem_map *map)
{
@@ -353,279 +312,106 @@ compact_user_maps(struct user_mem_maps *user_mem_maps)
sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct container *cfg;
if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct container *cfg;
+
+ /* find an unused container config */
+ CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = CONTAINER_INITIALIZER;
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
-}
+ DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ /* set to invalid fd */
+ dev->fd = -1;
-static int
-vfio_group_device_count(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ cfg->n_devices++;
+ return dev;
}
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
- }
+static void
+vfio_device_erase(struct container *cfg, struct vfio_device *dev)
+{
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- return vfio_cfg->vfio_groups[i].devices;
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct container *cfg = vfio_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -640,11 +426,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -662,11 +446,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -674,445 +456,533 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct user_mem_maps *user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < user_mem_maps->n_maps; i++) {
+ struct user_mem_map *map;
+ map = &user_mem_maps->maps[i];
+
+ ret = vfio_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we would've liked to prevent user from assigning devices twice to
+ * prevent resource leaks, but for group mode this is not possible, as
+ * there is no way to know which fd belongs to which group/device.
+ *
+ * we also do not need to look in other configs as if we were to attempt
+ * to use a different container, the kernel wouldn't have allowed us to
+ * bind the group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
+int
+rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+int
+rte_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ CONTAINER_FOREACH_ACTIVE(icfg) {
+ DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct container *cfg, enum rte_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request container from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum rte_vfio_mode
+vfio_select_mode(void)
+{
+ struct container *cfg;
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return RTE_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ cfg->dma_setup_done = true;
+ return mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return RTE_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return RTE_VFIO_MODE_NOIOMMU;
+ return RTE_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return RTE_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum rte_vfio_mode mode)
+{
+ switch (mode) {
+ case RTE_VFIO_MODE_GROUP: return "group";
+ case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum rte_vfio_mode mode = RTE_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
EAL_LOG(DEBUG, "Probing VFIO support...");
@@ -1132,36 +1002,16 @@ rte_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
- } else {
+ /* have we initialized anything? */
+ if (mode == RTE_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_cfg.mode = mode;
return 0;
}
@@ -1170,40 +1020,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
@@ -1212,126 +1039,27 @@ rte_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
@@ -1340,511 +1068,54 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
- return -1;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_NONE)
+ return vfio_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP && vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1852,16 +1123,14 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct user_mem_map *new_map;
@@ -1869,16 +1138,15 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1891,7 +1159,7 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/* create new user mem map entry */
new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
@@ -1908,17 +1176,17 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
+ struct user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
struct user_mem_map new_maps[2]; /* can be at most 2 */
struct user_mem_maps *user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
+ user_mem_maps = &cfg->mem_maps;
rte_spinlock_recursive_lock(&user_mem_maps->lock);
/*
@@ -1944,13 +1212,12 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1966,7 +1233,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1984,28 +1250,20 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
newlen = (user_mem_maps->n_maps - n_orig) + n_new;
if (newlen >= RTE_DIM(user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
@@ -2021,212 +1279,178 @@ RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
- unsigned int i;
+ struct container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_assign_device)
-int
-rte_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_GROUP:
+ case RTE_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_mode)
+enum rte_vfio_mode
+rte_vfio_get_mode(void)
+{
+ return vfio_cfg.mode;
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..c49f55bb8f 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,60 +6,162 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <rte_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ int group; /**< back-reference to group list (group mode) */
+ int fd;
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct container {
+ bool active;
+ bool dma_setup_done;
+ int container_fd;
+ struct user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*dma_func_t)(struct container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*dma_user_func_t)(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
- vfio_dma_user_func_t dma_user_map_func;
- vfio_dma_func_t dma_map_func;
+ dma_user_func_t dma_user_map_func;
+ dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct container *default_cfg;
+ enum rte_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
+
+/* per-process, per-container data */
+extern struct container containers[RTE_MAX_VFIO_CONTAINERS];
+
+/* current configuration */
+extern struct vfio_config vfio_cfg;
+
+#define CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &containers[0]; \
+ (cfg) < &containers[RTE_DIM(containers)]; \
+ (cfg)++)
+
+#define CONTAINER_FOREACH_ACTIVE(cfg) \
+ CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+
+#define GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+#define GROUP_FOREACH_ACTIVE(cfg, grp) \
+ GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
-int
-vfio_get_iommu_type(void);
+#define DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+/* for containers, we need to initialize the fd, and mem maps lock */
+#define CONTAINER_INITIALIZER \
+ ((struct container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ .container_fd = -1, \
+ })
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct container *cfg, int iommu_group);
+void vfio_group_erase(struct container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
@@ -75,6 +177,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..8c0beabc94
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,983 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <rte_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct container *);
+static int vfio_type1_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct container *);
+static int vfio_spapr_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct container *);
+static int vfio_noiommu_dma_mem_map(struct container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+uint64_t spapr_dma_win_len;
+uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(RTE_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_cfg.ops == NULL) {
+ vfio_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_cfg.ops->type_id, vfio_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 3eaeef2fc8..9a07d35023 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -32,21 +32,32 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
switch (m->req) {
case SOCKET_REQ_GROUP:
+ {
+ struct container *cfg = vfio_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
r->result = SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
+ }
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
fd = rte_vfio_get_container_fd();
@@ -54,6 +65,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
r->result = SOCKET_ERR;
else {
r->result = SOCKET_OK;
+ r->mode = vfio_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -62,6 +74,13 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
{
int iommu_type_id;
+ if (vfio_cfg.mode != RTE_VFIO_MODE_GROUP &&
+ vfio_cfg.mode != RTE_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
r->req = SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 15/20] bus/pci: use the new VFIO mode API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (13 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 14/20] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 16/20] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status. This requires moving
the VFIO init earlier in the init sequence, as now checking no-IOMMU mode
is not a sysfs check but instead relies on full VFIO subsystem to be
initialized and available.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
lib/eal/linux/eal.c | 15 ++++++++++-----
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..a1575b84e2 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index fc2e9b8c0e..5bc095bd61 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -677,6 +677,16 @@ rte_eal_init(int argc, char **argv)
}
}
+ /*
+ * VFIO must be initialized before bus scan because buses need to know
+ * about no-IOMMU mode status.
+ */
+ if (rte_vfio_enable("vfio")) {
+ rte_eal_init_alert("Cannot init VFIO");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
if (rte_bus_scan()) {
rte_eal_init_alert("Cannot scan the buses for devices");
rte_errno = ENODEV;
@@ -772,11 +782,6 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (rte_vfio_enable("vfio")) {
- rte_eal_init_alert("Cannot init VFIO");
- rte_errno = EAGAIN;
- goto err_out;
- }
/* in secondary processes, memory init may allocate additional fbarrays
* not present in primary processes, so to avoid any potential issues,
* initialize memzones first.
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 16/20] bus/fslmc: use the new VFIO mode API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (14 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 17/20] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 11 ++++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 3626b12316..7cc2375fca 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,14 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (rte_vfio_get_mode() != RTE_VFIO_MODE_GROUP &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU) {
+ DPAA2_BUS_DEBUG("DPAA2: VFIO not in group mode");
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +508,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 3ca68ccf24..15273fcd57 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -192,7 +192,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 17/20] net/hinic3: use the new VFIO mode API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (15 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 16/20] bus/fslmc: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 18/20] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..224be6f81c 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ rte_vfio_get_mode() != RTE_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 18/20] net/ntnic: use the new VFIO mode API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (16 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 17/20] net/hinic3: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..8b6bca974c 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
+ const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_get_mode() == RTE_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
if (n_rte_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 19/20] vfio: remove no-IOMMU check API
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (17 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 18/20] net/ntnic: " Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-25 14:17 ` [PATCH v12 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Bruce Richardson
The `rte_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/rte_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 6c1d1e3751..bb05a969a9 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -850,12 +850,6 @@ int rte_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0af41c3610..c4ba0e5cda 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -170,20 +170,6 @@ __rte_internal
enum rte_vfio_mode
rte_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-rte_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 660a28c90e..2092ccba53 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1275,13 +1275,6 @@ container_dma_unmap(struct container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int
-rte_vfio_noiommu_is_enabled(void)
-{
- return vfio_cfg.mode == RTE_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v12 20/20] vfio: introduce cdev mode
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (18 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-25 13:21 ` Anatoly Burakov
2026-08-25 14:17 ` [PATCH v12 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
20 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-25 13:21 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
The implementation largely follows the kernel reference code, with most
differences from group mode concentrated in how the container and the
devices are set up. The VFIO container allocates an "IOAS" (IO address
space), maps the memory into that space, and attaches devices to IOAS -
there is no IOMMU type discovery at device attach time.
Unlike group mode, the cdev mode is device-centric, and all DMA mappings
are made for an IOAS and not into VFIO container itself. As a result, while
group mode only maps memory during first device attach, cdev mode can map
memory without having any devices attached. However, current VFIO init has
moved to before bus scan, because we need VFIO mode information for buses
to decide on IOVA mode. DMA memory mapping, however, must be performed
after the DPDK memory was already initialized, so we add another internal
API to trigger VFIO DMA maps for cdev mode right after memory init.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/release_26_11.rst | 5 +
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/rte_vfio.h | 45 +++
lib/eal/linux/eal.c | 7 +
lib/eal/linux/eal_vfio.c | 253 +++++++++++++++-
lib/eal/linux/eal_vfio.h | 30 +-
lib/eal/linux/eal_vfio_cdev.c | 396 +++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 +++
lib/eal/linux/meson.build | 1 +
9 files changed, 791 insertions(+), 4 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index b5793b5c94..edee37d304 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added VFIO cdev (IOMMUFD) mode.**
+
+ Added support for the VFIO character device (cdev) API, also known as IOMMUFD.
+ DPDK now supports both the group mode and the new cdev mode.
+
Removed Items
-------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index bb05a969a9..39442d2bad 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -844,6 +844,12 @@ int rte_vfio_enable(__rte_unused const char *modname)
return -1;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_init_mem)
+int rte_vfio_init_mem(void)
+{
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
@@ -931,3 +937,13 @@ rte_vfio_get_mode(void)
{
return RTE_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index c4ba0e5cda..ba557cbfdb 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define RTE_VFIO_DIR "/dev/vfio"
#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define RTE_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define RTE_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define RTE_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - RTE_VFIO_MODE_NONE: VFIO is not enabled.
* - RTE_VFIO_MODE_GROUP: Legacy group mode.
* - RTE_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - RTE_VFIO_MODE_CDEV: Character device mode.
*/
enum rte_vfio_mode {
RTE_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
RTE_VFIO_MODE_GROUP, /**< Group mode */
RTE_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ RTE_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -141,6 +145,20 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
__rte_internal
int rte_vfio_enable(const char *modname);
+/**
+ * @internal
+ * Initialize VFIO memory mapping. Must be called after `rte_vfio_enable()` and
+ * after EAL memory initialization has completed.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int rte_vfio_init_mem(void);
+
/**
* @internal
* Check if VFIO subsystem is initialized and a specified kernel module is loaded.
@@ -197,6 +215,33 @@ __rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 5bc095bd61..97350fbb2a 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -819,6 +819,13 @@ rte_eal_init(int argc, char **argv)
goto err_out;
}
+ /* VFIO memory setup that requires DPDK memory to be available. */
+ if (rte_vfio_init_mem() < 0) {
+ rte_eal_init_alert("Cannot init VFIO memory");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
/* register multi-process action callbacks for hotplug after memory init */
if (eal_mp_dev_hotplug_init() < 0) {
rte_eal_init_alert("failed to register mp callback for hotplug");
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 2092ccba53..8fd32aed6c 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -347,6 +347,20 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ CONTAINER_FOREACH_ACTIVE(cfg) {
+ DEVICE_FOREACH_ACTIVE(cfg, dev)
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ return NULL;
+}
+
static struct container *
vfio_container_create(void)
{
@@ -516,6 +530,55 @@ vfio_setup_dma_mem(struct container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(ERR, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -662,6 +725,49 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when rte_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when rte_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -682,6 +788,9 @@ rte_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case RTE_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case RTE_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -754,6 +863,24 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -873,6 +1000,12 @@ rte_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -937,8 +1070,24 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- cfg->dma_setup_done = true;
+ if (mode == RTE_VFIO_MODE_CDEV) {
+ /* set up ops and sync IOAS */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else if (mode == RTE_VFIO_MODE_GROUP ||
+ mode == RTE_VFIO_MODE_NOIOMMU) {
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
+
return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
@@ -955,10 +1104,17 @@ vfio_select_mode(void)
return RTE_VFIO_MODE_NOIOMMU;
return RTE_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* cdev ops, IOAS and DMA setup are deferred to rte_vfio_init_mem() */
+ if (vfio_cdev_enable(cfg) == 0)
+ return RTE_VFIO_MODE_CDEV;
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
vfio_container_erase(cfg);
+ /* reset ops as well */
+ vfio_cfg.ops = NULL;
return RTE_VFIO_MODE_NONE;
}
@@ -969,6 +1125,7 @@ vfio_mode_to_str(enum rte_vfio_mode mode)
switch (mode) {
case RTE_VFIO_MODE_GROUP: return "group";
case RTE_VFIO_MODE_NOIOMMU: return "noiommu";
+ case RTE_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1016,6 +1173,36 @@ rte_vfio_enable(const char *modname)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_init_mem)
+int
+rte_vfio_init_mem(void)
+{
+ struct container *cfg = vfio_cfg.default_cfg;
+
+ /* secondary already synced everything during rte_vfio_enable() */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ switch (vfio_cfg.mode) {
+ case RTE_VFIO_MODE_CDEV:
+ /* iommufd IOAS is populated once DPDK memory is available */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ return -1;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ return -1;
+ if (vfio_register_mem_event_callback() < 0)
+ return -1;
+ cfg->dma_setup_done = true;
+ break;
+ default:
+ /* group/noiommu map memory at device attach, or no VFIO */
+ break;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
@@ -1108,6 +1295,40 @@ rte_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_num)
+int
+rte_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_cfg.mode == RTE_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1307,6 +1528,27 @@ rte_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case RTE_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ /* we only need to set up IOAS for DMA to work */
+ cfg->dma_setup_done = true;
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1365,6 +1607,13 @@ rte_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case RTE_VFIO_MODE_CDEV:
+ /* erase all devices */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index c49f55bb8f..5fda472e29 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,7 +48,10 @@ struct vfio_group {
/* device tracking (common for group and cdev modes) */
struct vfio_device {
bool active;
- int group; /**< back-reference to group list (group mode) */
+ union {
+ int group; /**< back-reference to group list (group mode) */
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX (cdev mode) */
+ };
int fd;
};
@@ -60,13 +63,21 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct container {
bool active;
bool dma_setup_done;
int container_fd;
struct user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -161,12 +172,25 @@ int vfio_group_setup_iommu(struct container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct container *cfg);
+void vfio_cdev_setup_ops(void);
+int vfio_cdev_setup_ioas(struct container *cfg);
+int vfio_cdev_sync_ioas(struct container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define SOCKET_REQ_CONTAINER 0x100
#define SOCKET_REQ_GROUP 0x200
#define SOCKET_REQ_IOMMU_TYPE 0x400
+#define SOCKET_REQ_CDEV 0x800
+#define SOCKET_REQ_IOAS_ID 0x1000
#define SOCKET_OK 0x0
#define SOCKET_NO_FD 0x1
#define SOCKET_ERR 0xFF
@@ -177,6 +201,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum rte_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..a659891b60
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,396 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct container *cfg);
+static int vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+void
+vfio_cdev_setup_ops(void)
+{
+ vfio_cfg.ops = &iommufd_ops;
+}
+
+static int
+vfio_cdev_dma_mem_map(struct container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(RTE_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ RTE_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(RTE_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ RTE_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd - only in primary */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0) {
+ close(device_fd);
+ dev->fd = -1;
+ return -1;
+ }
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 9a07d35023..6d94f44af8 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case SOCKET_REQ_CDEV:
+ {
+ struct container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = SOCKET_NO_FD;
+ } else {
+ r->result = SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case SOCKET_REQ_IOAS_ID:
+ {
+ struct container *cfg;
+
+ if (vfio_cfg.mode != RTE_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = SOCKET_ERR;
+ break;
+ }
+
+ r->req = SOCKET_REQ_IOAS_ID;
+ cfg = vfio_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h
2026-08-25 13:20 ` [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-25 14:08 ` Stephen Hemminger
0 siblings, 0 replies; 203+ messages in thread
From: Stephen Hemminger @ 2026-08-25 14:08 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev, Maxime Coquelin
On Tue, 25 Aug 2026 14:20:58 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
> required. Update internal headers to version v6.17 and include iommufd.h.
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Acked-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
You should probably go to 7.2 kernel version if patch series gets updated.
Since kernel / userspace is stable this would help.
^ permalink raw reply [flat|nested] 203+ messages in thread
* Re: [PATCH v12 00/20] Support VFIO cdev API in DPDK
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
` (19 preceding siblings ...)
2026-08-25 13:21 ` [PATCH v12 20/20] vfio: introduce cdev mode Anatoly Burakov
@ 2026-08-25 14:17 ` Stephen Hemminger
2026-08-26 10:09 ` Burakov, Anatoly
20 siblings, 1 reply; 203+ messages in thread
From: Stephen Hemminger @ 2026-08-25 14:17 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
On Tue, 25 Aug 2026 14:20:57 +0100
Anatoly Burakov <anatoly.burakov@intel.com> wrote:
> This patchset introduces a major refactor of the VFIO subsystem in DPDK to
> support character device (cdev) interface introduced in Linux kernel, as well as
> make the API more streamlined and useful. The goal is to simplify device
> management, improve compatibility, and clarify API responsibilities.
>
> The following sections outline the key issues addressed by this patchset and the
> corresponding changes introduced.
>
> 1. Only group mode is supported
> ===============================
>
> Since kernel version 4.14.327 (LTS), VFIO supports the new character device
> (cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
> is a device-centric mode and does away with all the complexity regarding groups
> and IOMMU types, delegating it all to the kernel, and exposes a much simpler
> interface to userspace.
>
> The old group interface is still around, and will need to be kept in DPDK both
> for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
> driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
> the API the way it was before.
>
> To enable this, VFIO is heavily refactored, so that the code can support both
> modes while relying on (mostly) common infrastructure.
>
> Note that the existing `rte_vfio_device_setup/release` model is fundamentally
> incompatible with cdev mode, because for custom container cases, the expected
> flow is that the user binds the IOMMU group (and thus, implicitly, the device
> itself) to a specific container using `rte_vfio_container_group_bind`, whereas
> this step is not needed for cdev as the device fd is assigned to the container
> straight away.
>
> Therefore, what we do instead is introduce a new API for container device
> assignment which, semantically, will assign a device to specified container, so
> that when it is mapped using `rte_pci_map_device`, the appropriate container is
> selected. Under the hood though, we essentially transition to getting device fd
> straight away at assign stage, so that by the time the PCI bus attempts to map
> the device, it is already mapped and we just return an fd. There is no
> "unassign" API because `release_device` already performs that function.
>
> Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
> some introspection into VFIO's internals, with three new modes: group
> (old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
> Although no-IOMMU is technically a variant of group mode, the distinction is
> largely irrelevant to the user, as all usages of noiommu checks in our codebase
> are for deciding whether to use IOVA or PA, not anything to do with managing
> groups. The current plan for kernel community is to *not* introduce no-IOMMU
> cdev implementation, and IOMMUFD's own group API compatibility layer also does
> not implement no-IOMMU mode, which is why this will be kept for compatibility
> for these use cases.
>
> There were other users of VFIO which relied on group API but only for convenience
> purposes; no actual VFIO functionality depended on those API's. Therefore, group
> API's are removed and, where appropriate, replaced with the new API's.
>
> List of removed API's:
>
> * `rte_vfio_get_group_fd`
> * `rte_vfio_clear_group`
> * `rte_vfio_container_group_bind` (replaced by container assign API)
> * `rte_vfio_container_group_unbind`
> * `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
>
> 2. The API responsibilities aren't clear and bleed into each other
> ==================================================================
>
> Some API's do multiple things at once. In particular:
>
> * `rte_vfio_get_device_info` will setup the device
> * `rte_vfio_setup_device` will get device info
>
> These API's have been adjusted to do one thing only.
>
> v12:
> - Addressed feedback from Stephen's AI review:
> - Add release notes updates to patch 2 and 20
> - Fix ENXIO typos
> - Added a new init step after memory init to enable DMA mapping for cdev
>
> v11:
> - Addressed feedback from Stephen's AI review:
> - Use CONTAINER_INITIALIZER for reset
> - Set container fd to -1 in CONTAINER_INITIALIZER
> - Fixed double close()
> - Moved VFIO init to earlier in init sequence to account for
> bus drivers needing no-IOMMU mode status
> - Fixed missing ops set for cdev mode, and missing ops reset
> - Fixed fd leak on failed attach in cdev
>
> v10:
> - Added a patch that renames confusing error labels
> - Fixed compiler warning about unused variable
>
> v9:
> - Moved erroneous rte_errno-related comments to later in the patchset
> - Moved removal of vDPA group fd API's to their respective patches
> - Fixed typo in errno comments (ENXIO vs ENOXIO)
> - Fixed corruption of group config in secondary process (v8 AI review)
>
> v8:
> - Rebase
> - Fixed build errors due to variable shadowing
> - Removed duplicate fd check as kernel does not provide a way to distinguish
> between device fd's
>
> v7:
> - Rebase
> - Added removal of deprecation notices
> - Fixed implicit numeric comparison in patch 12
>
> v6:
> - Fixed missing header include in vfio cdev file
>
> v5:
> - Added back missing uapi patch
>
> v4:
> - Fixed issues with documenting rte_vfio_mode enum
> - Separated deprecation notices into a separate patchset
>
> v3:
> - Make API removal cleaner
> - Fix `get_group_num` usages to align with new API
> - Fix issues with function exports
> - Fix issues with `setup_device` returning old-style values in some cases
>
> v2:
> - Make the entire API internal
> - More aggressive API pruning, complete removal of group API
> - Fixed a bug in group mode where device could not be used
> - Better documentation and deprecation notice patches
> - Moved doc patches to beginning of patchset
>
> Anatoly Burakov (20):
> uapi: update to v6.17 and add iommufd.h
> vfio: make all functions internal
> bus/pci: rename mismatching error labels
> vfio: split get device info from setup
> vfio: add container device assignment API
> net/nbl: do not use VFIO group bind API
> net/ntnic: use container device assignment API
> vdpa/ifc: use container device assignment API
> vdpa/nfp: use container device assignment API
> vdpa/sfc: use container device assignment API
> vdpa/mlx5: remove group-related API
> vhost: remove group-related API from driver
> vfio: remove group-based API
> vfio: cleanup and refactor
> bus/pci: use the new VFIO mode API
> bus/fslmc: use the new VFIO mode API
> net/hinic3: use the new VFIO mode API
> net/ntnic: use the new VFIO mode API
> vfio: remove no-IOMMU check API
> vfio: introduce cdev mode
>
> config/arm/meson.build | 1 +
> config/meson.build | 1 +
> doc/guides/prog_guide/vhost_lib.rst | 4 -
> doc/guides/rel_notes/deprecation.rst | 10 -
> doc/guides/rel_notes/release_26_11.rst | 13 +
> drivers/bus/cdx/cdx_vfio.c | 25 +-
> drivers/bus/fslmc/fslmc_bus.c | 11 +-
> drivers/bus/fslmc/fslmc_vfio.c | 6 +-
> drivers/bus/pci/linux/pci.c | 2 +-
> drivers/bus/pci/linux/pci_vfio.c | 47 +-
> drivers/bus/platform/platform.c | 9 +-
> drivers/crypto/bcmfs/bcmfs_vfio.c | 14 +-
> drivers/net/hinic3/base/hinic3_hwdev.c | 3 +-
> drivers/net/nbl/nbl_common/nbl_userdev.c | 22 +-
> drivers/net/nbl/nbl_include/nbl_include.h | 1 +
> drivers/net/ntnic/ntnic_ethdev.c | 2 +-
> drivers/net/ntnic/ntnic_vfio.c | 30 +-
> drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +-
> drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
> drivers/vdpa/nfp/nfp_vdpa.c | 37 +-
> drivers/vdpa/sfc/sfc_vdpa.c | 39 +-
> drivers/vdpa/sfc/sfc_vdpa.h | 2 -
> kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++
> kernel/linux/uapi/linux/vduse.h | 2 +-
> kernel/linux/uapi/linux/vfio.h | 12 +-
> kernel/linux/uapi/version | 2 +-
> lib/eal/freebsd/eal.c | 104 +-
> lib/eal/include/rte_vfio.h | 401 ++--
> lib/eal/linux/eal.c | 22 +-
> lib/eal/linux/eal_vfio.c | 2471 ++++++++-------------
> lib/eal/linux/eal_vfio.h | 169 +-
> lib/eal/linux/eal_vfio_cdev.c | 396 ++++
> lib/eal/linux/eal_vfio_group.c | 983 ++++++++
> lib/eal/linux/eal_vfio_mp_sync.c | 80 +-
> lib/eal/linux/meson.build | 2 +
> lib/eal/windows/eal.c | 4 +-
> lib/vhost/vdpa_driver.h | 3 -
> 37 files changed, 4350 insertions(+), 1907 deletions(-)
> create mode 100644 kernel/linux/uapi/linux/iommufd.h
> create mode 100644 lib/eal/linux/eal_vfio_cdev.c
> create mode 100644 lib/eal/linux/eal_vfio_group.c
>
The patch series in pretty good shape. AI still has feedback but
not sure how important the details are now.
Review: [PATCH v12 00/20] vfio: introduce cdev mode
Applied cleanly on top of 26.11-rc0 (d55ccd4 "pci: remove deprecated
catch-all flag") with plain git am. Patch 01 was diffed against the
v6.17 kernel tree; iommufd.h, vfio.h and vduse.h match exactly apart
from the usual headers_install guard rewrites. Build testing waived
per request.
Patches 03, 05, 07-13, 15-19 have no findings.
Patch 02 vfio: make all functions internal
------------------------------------------
Info: doc/api/doxy-api-index.md still lists rte_vfio.h under the
public API index. With every function now tagged @internal (and
INTERNAL_DOCS off), that page will render almost empty. Drop the
entry or move it alongside the other driver-SDK headers.
Patch 04 vfio: split get device info from setup
-----------------------------------------------
Info: in pci_vfio_ioport_map() secondary path, the old
rte_vfio_get_device_info() with fd == 0 cleaned up internally on a
GET_INFO failure. Now the sequence is setup, intr_dev_fd_set, then
get_device_info, and a get_device_info failure returns -1 without
rte_vfio_release_device(). The fd is at least recorded in the intr
handle so a later unmap can close it; the surrounding fill_regions
path already has the same shape, so this only preserves the existing
pattern rather than adding a new class of leak.
Patch 06 net/nbl: do not use VFIO group bind API
------------------------------------------------
Warning: replacing rte_vfio_container_group_bind() with a direct
open(RTE_VFIO_GROUP_FMT) changes behaviour when two nbl devices share
an IOMMU group. The kernel allows a VFIO group file to be open only
once (vfio_group_fops_open returns -EBUSY when group->opened_file is
already set). Previously the EAL cache returned the same fd for the
second device; now the second open() fails and the driver drops to
the rte_pci_map_device() fallback. If shared groups are not a
supported nbl topology, a comment saying so would avoid the next
person tripping on this; otherwise keep a small per-group fd table in
the driver.
Patch 14 vfio: cleanup and refactor
-----------------------------------
Warning: rte_vfio_enable() is no longer safe to call twice, and
drivers/bus/pci/linux/pci_vfio.c:pci_vfio_is_enabled() still calls
it a second time when rte_vfio_is_enabled("vfio_pci") is false. On a
second call with VFIO already up, vfio_select_mode() does
vfio_container_create(), which returns containers[1] (containers[0]
is active), hits "cfg != vfio_cfg.default_cfg", returns
RTE_VFIO_MODE_NONE, and rte_vfio_enable() then stores that into
vfio_cfg.mode, disabling a working VFIO instance and leaving
containers[1] marked active. Reachability is narrow (vfio loaded,
vfio_pci sysfs entry absent, device reports kdrv VFIO), but the fix
is trivial: return 0 early from rte_vfio_enable() when vfio_cfg.mode
!= RTE_VFIO_MODE_NONE.
Warning: new non-static globals `containers` and `vfio_cfg` in
eal_vfio.c (declared extern in eal_vfio.h) have no subsystem prefix.
These are exactly the kind of names that collide when an application
is statically linked against EAL. spapr_dma_win_len /
spapr_dma_win_page_sz were already unprefixed globals and could be
made static while the file is being reorganised. Suggest
vfio_containers / vfio_global_cfg (or similar) and static for the
spapr pair.
Info: RTE_MAX_VFIO_DEVICES (256) is a new hard per-container limit.
The previous implementation did not track device fds at all, so a
process that maps more than 256 VFIO devices in the default container
now fails with ENOSPC where it used to work. Probably fine for real
deployments, but worth a line in the release notes next to the
config knob.
Info: in group mode, rte_vfio_container_assign_device() now opens a
device fd of its own, and the subsequent rte_vfio_setup_device()
(via rte_pci_map_device) opens a second one for the same device, as
the in-code comment acknowledges. Both are closed by
rte_vfio_container_destroy(), so nothing leaks, but note that
rte_vfio_release_device() on the second fd no longer triggers the
vfio-pci last-close reset because the assign fd is still open. The
vdpa drivers destroy the container immediately afterwards, so no
practical issue today.
Info: the primary-side SOCKET_REQ_GROUP handler now only returns a
group fd that the primary already has open; the old handler opened
the group on demand. Secondary processes cannot map devices the
primary has not set up anyway (they need the primary's resource
tailq entry), so this is fine, just a behaviour change to be aware
of.
Patch 20 vfio: introduce cdev mode
----------------------------------
Warning: vfio_cdev_assign_device() logs at ERR level
("Device %s already assigned to this container") on the expected
path. Every vdpa driver does rte_vfio_container_assign_device()
followed by rte_pci_map_device(), and in cdev mode the second call
lands here and returns VFIO_EXISTS, which rte_vfio_setup_device()
then treats as success. That should be DEBUG, not ERR.
Warning: no user-facing documentation for the new mode.
doc/guides/linux_gsg/linux_drivers.rst still describes VFIO purely in
terms of IOMMU groups. It should say that cdev/iommufd is now
supported, that group mode is preferred whenever /dev/vfio/vfio is
usable and cdev is only selected as a fallback (there is no EAL
option to force it), that /dev/iommu must be accessible, and that VF
tokens in cdev mode need kernel >= 6.17 (the NOTICE in
cdev_attach_device_to_iommufd is the only place this is stated).
Info: cdev mode does not handle no-IOMMU cdev devices. The kernel
exposes those as /dev/vfio/devices/noiommu-vfioX, but
cdev_open_device_fd() only builds "vfioX", so the open fails and
rte_vfio_get_mode() would never report RTE_VFIO_MODE_NOIOMMU for
them. Since group mode wins when available this is mostly
theoretical, but a comment or explicit rejection would be clearer
than an open() failure.
Info: the comment in vfio_cdev_setup_device() says "attach device to
iommufd - only in primary" but the enclosing condition is primary OR
non-default container. Fix the comment.
Info: rte_vfio_get_device_num() is added with no in-tree caller. The
commit message says it is for future use; that is acceptable for an
internal symbol but easy to forget, so consider dropping it until
something needs it.
Summary
-------
The series is in good shape and the refactor is a real improvement in
readability. The remaining items are all Warning or below; the
rte_vfio_enable() re-entry and the unprefixed globals are the two I
would want addressed before merge, plus the ERR-level log on the
normal cdev assign path and a docs paragraph for the new mode.
^ permalink raw reply [flat|nested] 203+ messages in thread
* Re: [PATCH v12 00/20] Support VFIO cdev API in DPDK
2026-08-25 14:17 ` [PATCH v12 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
@ 2026-08-26 10:09 ` Burakov, Anatoly
0 siblings, 0 replies; 203+ messages in thread
From: Burakov, Anatoly @ 2026-08-26 10:09 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On 8/25/2026 4:17 PM, Stephen Hemminger wrote:
> On Tue, 25 Aug 2026 14:20:57 +0100
> Anatoly Burakov <anatoly.burakov@intel.com> wrote:
>
>
> The patch series in pretty good shape. AI still has feedback but
> not sure how important the details are now.
there's still a bunch of reasonable feedback, so I'll address it. Thanks
for the review!
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v13 00/24] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (6 preceding siblings ...)
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 01/24] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (23 more replies)
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
8 siblings, 24 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, make the code readable, and clarify API.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace.
The old group interface is still around, and will need to be kept in DPDK both
for compatibility reasons, as well as supporting special cases (FSLMC bus, NBL
driver, no-IOMMU mode etc.), but it is now internal-only and not exposed through
the API the way it was before.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Note that the existing `rte_vfio_device_setup/release` model is fundamentally
incompatible with cdev mode, because for custom container cases, the expected
flow is that the user binds the IOMMU group (and thus, implicitly, the device
itself) to a specific container using `rte_vfio_container_group_bind`, whereas
this step is not needed for cdev as the device fd is assigned to the container
straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why this will be kept for compatibility
for these use cases.
There were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's. Therefore, group
API's are removed and, where appropriate, replaced with the new API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
2. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
v13:
- Addressed feedback from Stephen's AI review:
- Added strdup-based deduplication for group mode to avoid leaking fd's when
container assignment API is used
- Added more documentation about cdev mode to Linux GSG
- Fixed comments and log levels in a couple of places
- Renamed the API from `rte_vfio_*` to `dev_vfio_*` on account of it no longer
being a publicly visible API
- Moved `dev_vfio.h` into driver SDK headers
- Decoupled VFIO internals from FSLMC and EAL
- Added proper teardown for VFIO subsystem on cleanup
v12:
- Addressed feedback from Stephen's AI review:
- Add release notes updates to patch 2 and 20
- Fix ENXIO typos
- Added a new init step after memory init to enable DMA mapping for cdev
v11:
- Addressed feedback from Stephen's AI review:
- Use CONTAINER_INITIALIZER for reset
- Set container fd to -1 in CONTAINER_INITIALIZER
- Fixed double close()
- Moved VFIO init to earlier in init sequence to account for
bus drivers needing no-IOMMU mode status
- Fixed missing ops set for cdev mode, and missing ops reset
- Fixed fd leak on failed attach in cdev
v10:
- Added a patch that renames confusing error labels
- Fixed compiler warning about unused variable
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
This patchset depends on the following patchsets:
1) net/nbl: fix use-after-free: https://patches.dpdk.org/project/dpdk/list/?series=39072
Anatoly Burakov (24):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
bus/fslmc: decouple from EAL VFIO
vfio: decouple EAL from VFIO internals
vfio: do proper teardown on VFIO cleanup
vfio: rename API to reflect internal status
bus/pci: rename mismatching error labels
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/linux_drivers.rst | 30 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
doc/guides/rel_notes/release_26_11.rst | 16 +
drivers/bus/cdx/cdx.c | 6 +-
drivers/bus/cdx/cdx_vfio.c | 37 +-
drivers/bus/fslmc/fslmc_bus.c | 13 +-
drivers/bus/fslmc/fslmc_vfio.c | 66 +-
drivers/bus/fslmc/meson.build | 1 -
drivers/bus/pci/linux/pci.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 79 +-
drivers/bus/pci/pci_common.c | 10 +-
drivers/bus/platform/platform.c | 17 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 20 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 5 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 34 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 10 +-
drivers/net/ntnic/ntnic_vfio.c | 48 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 54 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 53 +-
drivers/vdpa/sfc/sfc_vdpa.c | 47 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
drivers/vdpa/sfc/sfc_vdpa_hw.c | 12 +-
kernel/linux/uapi/linux/iommufd.h | 1292 ++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 94 +-
lib/eal/include/dev_vfio.h | 452 ++++
lib/eal/include/meson.build | 2 +-
lib/eal/include/rte_vfio.h | 338 ---
lib/eal/linux/eal.c | 27 +-
lib/eal/linux/eal_vfio.c | 2636 +++++++++------------
lib/eal/linux/eal_vfio.h | 190 +-
lib/eal/linux/eal_vfio_cdev.c | 396 ++++
lib/eal/linux/eal_vfio_group.c | 983 ++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 108 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 16 +-
lib/vhost/vdpa_driver.h | 3 -
lib/vhost/vhost_user.c | 16 +-
46 files changed, 4881 insertions(+), 2275 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/include/dev_vfio.h
delete mode 100644 lib/eal/include/rte_vfio.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v13 01/24] uapi: update to v6.17 and add iommufd.h
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 02/24] vfio: make all functions internal Anatoly Burakov
` (22 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
Depends-on: series-39072 ("net/nbl: fix use-after-free")
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 02/24] vfio: make all functions internal
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 01/24] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 03/24] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
` (21 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/api/doxy-api-index.md | 3 +-
doc/guides/rel_notes/deprecation.rst | 6 ----
doc/guides/rel_notes/release_26_11.rst | 8 +++++
lib/eal/freebsd/eal.c | 30 ++++++++--------
lib/eal/include/meson.build | 2 +-
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++---------
lib/eal/windows/eal.c | 4 +--
8 files changed, 84 insertions(+), 48 deletions(-)
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 9296042119..4216b1b9e4 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -37,8 +37,7 @@ The public API headers are grouped by topics:
[latency](@ref rte_latencystats.h),
[devargs](@ref rte_devargs.h),
[PCI](@ref rte_pci.h),
- [vdev](@ref rte_bus_vdev.h),
- [vfio](@ref rte_vfio.h)
+ [vdev](@ref rte_bus_vdev.h)
- **device specific**:
[softnic](@ref rte_eth_softnic.h),
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6ad7698c6b..8a82c815e3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,12 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index c8cc86295d..b5793b5c94 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -93,6 +93,10 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public API.
+
ABI Changes
-----------
@@ -109,6 +113,10 @@ ABI Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public ABI.
+
Known Issues
------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index aef5824e5f..9282143918 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -55,12 +55,12 @@ headers += files(
'rte_trace_point_register.h',
'rte_uuid.h',
'rte_version.h',
- 'rte_vfio.h',
)
driver_sdk_headers = files(
'bus_driver.h',
'dev_driver.h',
+ 'rte_vfio.h',
)
# special case install the generic headers, since they go in a subdir
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 03/24] bus/fslmc: decouple from EAL VFIO
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 01/24] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 02/24] vfio: make all functions internal Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 04/24] vfio: decouple EAL from VFIO internals Anatoly Burakov
` (20 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus has its own VFIO implementation which is completely separate from
one in EAL or rte_vfio. Yet, it references a couple of macros and uses a
struct definition. Fully decoupling them is not a huge lift, so do that.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/fslmc/fslmc_vfio.c | 50 +++++++++++++++++++++-------------
drivers/bus/fslmc/meson.build | 1 -
2 files changed, 31 insertions(+), 20 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..8d39895bac 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -35,7 +35,6 @@
#include <dev_driver.h>
#include <rte_eal_memconfig.h>
#include <rte_vfio.h>
-#include <eal_vfio.h>
#include "private.h"
#include "fslmc_vfio.h"
@@ -73,8 +72,21 @@ struct fslmc_dmaseg_list fslmc_iosegs =
static uint64_t fslmc_mem_va2iova = RTE_BAD_IOVA;
static int fslmc_mem_map_num;
+#define FSLMC_VFIO_SOCKET_OK 0x0
+#define FSLMC_VFIO_SOCKET_NO_FD 0x1
+#define FSLMC_VFIO_SOCKET_ERR 0xFF
+
+struct fslmc_vfio_mp_param {
+ int req;
+ int result;
+ union {
+ int group_num;
+ int iommu_type_id;
+ };
+};
+
struct fslmc_mem_param {
- struct vfio_mp_param mp_param;
+ struct fslmc_vfio_mp_param mp_param;
struct fslmc_dmaseg_list memsegs;
struct fslmc_dmaseg_list iosegs;
uint64_t mem_va2iova;
@@ -386,7 +398,7 @@ fslmc_vfio_open_group_fd(const char *group_name)
struct rte_mp_msg mp_req, *mp_rep;
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ struct fslmc_vfio_mp_param *p = (struct fslmc_vfio_mp_param *)mp_req.param;
int iommu_group_num, ret;
vfio_group_fd = fslmc_vfio_group_fd_by_name(group_name);
@@ -417,10 +429,10 @@ fslmc_vfio_open_group_fd(const char *group_name)
if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
mp_reply.nb_received == 1) {
mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ p = (struct fslmc_vfio_mp_param *)mp_rep->param;
+ if (p->result == FSLMC_VFIO_SOCKET_OK && mp_rep->num_fds == 1)
vfio_group_fd = mp_rep->fds[0];
- else if (p->result == SOCKET_NO_FD)
+ else if (p->result == FSLMC_VFIO_SOCKET_NO_FD)
DPAA2_BUS_ERR("Bad VFIO group fd");
}
@@ -490,7 +502,7 @@ fslmc_vfio_open_container_fd(void)
struct rte_mp_msg mp_req, *mp_rep;
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (void *)mp_req.param;
+ struct fslmc_vfio_mp_param *p = (void *)mp_req.param;
if (fslmc_vfio_container_fd() > 0)
return fslmc_vfio_container_fd();
@@ -551,7 +563,7 @@ fslmc_vfio_open_container_fd(void)
mp_rep = &mp_reply.msgs[0];
p = (void *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ if (p->result == FSLMC_VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
vfio_container_fd = mp_rep->fds[0];
free(mp_reply.msgs);
}
@@ -594,8 +606,8 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
int fd = -1;
int ret;
struct rte_mp_msg reply;
- struct vfio_mp_param *r = (void *)reply.param;
- const struct vfio_mp_param *m = (const void *)msg->param;
+ struct fslmc_vfio_mp_param *r = (void *)reply.param;
+ const struct fslmc_vfio_mp_param *m = (const void *)msg->param;
struct fslmc_mem_param *map;
if (msg->len_param != sizeof(*m)) {
@@ -612,13 +624,13 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
r->group_num = m->group_num;
fd = fslmc_vfio_group_fd_by_id(m->group_num);
if (fd < 0) {
- r->result = SOCKET_ERR;
+ r->result = FSLMC_VFIO_SOCKET_ERR;
} else if (!fd) {
/* if group exists but isn't bound to VFIO driver */
- r->result = SOCKET_NO_FD;
+ r->result = FSLMC_VFIO_SOCKET_NO_FD;
} else {
/* if group exists and is bound to VFIO driver */
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -628,9 +640,9 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
r->req = FSLMC_VFIO_SOCKET_REQ_CONTAINER;
fd = fslmc_vfio_container_fd();
if (fd <= 0) {
- r->result = SOCKET_ERR;
+ r->result = FSLMC_VFIO_SOCKET_ERR;
} else {
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -640,7 +652,7 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
map = (void *)reply.param;
r = &map->mp_param;
r->req = FSLMC_VFIO_SOCKET_REQ_MEM;
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
map->memsegs = fslmc_memsegs;
map->iosegs = fslmc_iosegs;
map->mem_va2iova = fslmc_mem_va2iova;
@@ -666,19 +678,19 @@ fslmc_vfio_mp_sync_mem_req(void)
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
int ret = 0;
- struct vfio_mp_param *mp_param;
+ struct fslmc_vfio_mp_param *mp_param;
struct fslmc_mem_param *mem_rsp;
mp_param = (void *)mp_req.param;
memset(&mp_req, 0, sizeof(struct rte_mp_msg));
mp_param->req = FSLMC_VFIO_SOCKET_REQ_MEM;
rte_strscpy(mp_req.name, FSLMC_VFIO_MP, sizeof(mp_req.name));
- mp_req.len_param = sizeof(struct vfio_mp_param);
+ mp_req.len_param = sizeof(struct fslmc_vfio_mp_param);
if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
mp_reply.nb_received == 1) {
mp_rep = &mp_reply.msgs[0];
mem_rsp = (struct fslmc_mem_param *)mp_rep->param;
- if (mem_rsp->mp_param.result == SOCKET_OK) {
+ if (mem_rsp->mp_param.result == FSLMC_VFIO_SOCKET_OK) {
fslmc_memsegs = mem_rsp->memsegs;
fslmc_mem_va2iova = mem_rsp->mem_va2iova;
fslmc_mem_map_num = mem_rsp->mem_map_num;
diff --git a/drivers/bus/fslmc/meson.build b/drivers/bus/fslmc/meson.build
index 50d9e91a37..55f9d2e808 100644
--- a/drivers/bus/fslmc/meson.build
+++ b/drivers/bus/fslmc/meson.build
@@ -30,4 +30,3 @@ sources = files(
)
includes += include_directories('mc', 'qbman/include', 'portal')
-includes += include_directories('../../../lib/eal/linux')
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 04/24] vfio: decouple EAL from VFIO internals
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (2 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 03/24] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 05/24] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
` (19 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
Currently, the multiprocess cleanup for VFIO is called via an internal-only
API, and this is the only thing `eal_vfio.h` is used for in EAL. In order
to make `eal_vfio.h` truly VFIO-internal, export a cleanup function from
`rte_vfio` API instead, and call that. This allows us to remove all EAL
references to `eal_vfio.h` in non-VFIO related files.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ++++++
lib/eal/include/rte_vfio.h | 9 +++++++++
lib/eal/linux/eal.c | 3 +--
lib/eal/linux/eal_vfio.c | 7 +++++++
lib/eal/windows/eal.c | 6 ++++++
5 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..991ca45064 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -845,6 +845,12 @@ int rte_vfio_enable(__rte_unused const char *modname)
return -1;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..e76181c83c 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -109,6 +109,15 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
__rte_internal
int rte_vfio_enable(const char *modname);
+/**
+ * @internal
+ * Cleanup VFIO resources.
+ *
+ * This function is only relevant to Linux.
+ */
+__rte_internal
+void rte_vfio_cleanup(void);
+
/**
* @internal
* Check whether a VFIO-related kmod is enabled.
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index fc2e9b8c0e..5577bab24a 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -54,7 +54,6 @@
#include "eal_memcfg.h"
#include "eal_trace.h"
#include "eal_options.h"
-#include "eal_vfio.h"
#include "hotplug_mp.h"
#include "log_internal.h"
@@ -986,7 +985,7 @@ rte_eal_cleanup(void)
rte_service_finalize();
eal_bus_cleanup();
- vfio_mp_sync_cleanup();
+ rte_vfio_cleanup();
rte_mp_channel_cleanup();
rte_eal_alarm_cleanup();
rte_trace_save();
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..6c77076bc3 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2217,3 +2217,10 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_unmap(vfio_cfg, vaddr, iova, len);
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+ vfio_mp_sync_cleanup();
+}
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index de7a89a829..b8771d4d85 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -152,6 +152,12 @@ rte_eal_cleanup(void)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+}
+
/* Launch threads, called at application init(). */
RTE_EXPORT_SYMBOL(rte_eal_init)
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 05/24] vfio: do proper teardown on VFIO cleanup
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (3 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 04/24] vfio: decouple EAL from VFIO internals Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 06/24] vfio: rename API to reflect internal status Anatoly Burakov
` (18 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev
Currently, VFIO cleanup only unregisters multiprocess callback, but does
not destroy containers, groups, and user mem maps. Do all of that on VFIO
cleanup. In order to distinguish between config that is not initialized vs.
config that has been initialized but happens to have fd == 0, move the
global VFIO enabled flag out of the config, and add a separate per-config
"enabled" flag that can be checked to avoid attempting to clean up configs
that were never initialized in the first place.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/linux/eal_vfio.c | 77 +++++++++++++++++++++++++++++++++++++---
1 file changed, 72 insertions(+), 5 deletions(-)
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 6c77076bc3..84bb845e96 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -46,7 +46,7 @@ struct user_mem_maps {
};
struct vfio_config {
- int vfio_enabled;
+ bool enabled;
int vfio_container_fd;
int vfio_active_groups;
const struct vfio_iommu_type *vfio_iommu_type;
@@ -58,6 +58,9 @@ struct vfio_config {
static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
+/* whether VFIO is enabled (usable) in this process */
+static bool vfio_enabled;
+
static int vfio_type1_dma_map(int);
static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_spapr_dma_map(int);
@@ -686,7 +689,7 @@ vfio_sync_default_container(void)
return -1;
/* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
+ if (!vfio_enabled ||
default_vfio_cfg->vfio_container_fd < 0) {
EAL_LOG(ERR, "VFIO support is not initialized");
return -1;
@@ -1112,6 +1115,7 @@ rte_vfio_enable(const char *modname)
rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
+ vfio_cfgs[i].enabled = false;
vfio_cfgs[i].vfio_container_fd = -1;
vfio_cfgs[i].vfio_active_groups = 0;
vfio_cfgs[i].vfio_iommu_type = NULL;
@@ -1167,7 +1171,8 @@ rte_vfio_enable(const char *modname)
/* check if we have VFIO driver enabled */
if (default_vfio_cfg->vfio_container_fd != -1) {
EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
+ default_vfio_cfg->enabled = true;
+ vfio_enabled = true;
} else {
EAL_LOG(NOTICE, "VFIO support could not be initialized");
}
@@ -1180,7 +1185,7 @@ int
rte_vfio_is_enabled(const char *modname)
{
const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ return vfio_enabled && mod_available;
}
int
@@ -1357,7 +1362,7 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
+ if (!vfio_enabled)
return -1;
return default_vfio_cfg->vfio_container_fd;
@@ -2090,6 +2095,7 @@ rte_vfio_container_create(void)
EAL_LOG(NOTICE, "Fail to create a new VFIO container");
return -1;
}
+ vfio_cfgs[i].enabled = true;
return vfio_cfgs[i].vfio_container_fd;
}
@@ -2113,6 +2119,7 @@ rte_vfio_container_destroy(int container_fd)
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
+ vfio_cfg->enabled = false;
vfio_cfg->vfio_container_fd = -1;
vfio_cfg->vfio_active_groups = 0;
vfio_cfg->vfio_iommu_type = NULL;
@@ -2218,9 +2225,69 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_unmap(vfio_cfg, vaddr, iova, len);
}
+static void
+vfio_cleanup_config(struct vfio_config *vfio_cfg)
+{
+ unsigned int i;
+
+ for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
+ struct vfio_group *group = &vfio_cfg->vfio_groups[i];
+
+ if (group->group_num == -1)
+ continue;
+ if (group->devices != 0) {
+ EAL_LOG(ERR, "Cannot cleanup VFIO group %d with %d devices",
+ group->group_num, group->devices);
+ return;
+ }
+ if (group->fd >= 0 && close(group->fd) < 0) {
+ EAL_LOG(ERR, "Cannot close VFIO group %d: %s",
+ group->group_num, strerror(errno));
+ return;
+ }
+
+ group->group_num = -1;
+ group->fd = -1;
+ group->devices = 0;
+ vfio_cfg->vfio_active_groups--;
+ }
+
+ /* if there are still active groups, we cannot cleanup the container */
+ if (vfio_cfg->vfio_active_groups != 0) {
+ EAL_LOG(ERR, "Cannot cleanup VFIO container with %d active groups",
+ vfio_cfg->vfio_active_groups);
+ return;
+ }
+
+ if (vfio_cfg->vfio_container_fd >= 0 &&
+ close(vfio_cfg->vfio_container_fd) < 0) {
+ EAL_LOG(ERR, "Cannot close VFIO container: %s", strerror(errno));
+ return;
+ }
+
+ vfio_cfg->vfio_container_fd = -1;
+ vfio_cfg->enabled = false;
+ vfio_cfg->vfio_iommu_type = NULL;
+
+ vfio_cfg->mem_maps.n_maps = 0;
+ memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
void
rte_vfio_cleanup(void)
{
+ unsigned int i;
+
vfio_mp_sync_cleanup();
+
+ /* mem events can only be unregistered from the primary process */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+
+ /* cleanup all enabled configs, ignore failures */
+ for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
+ if (vfio_cfgs[i].enabled)
+ vfio_cleanup_config(&vfio_cfgs[i]);
+ }
}
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 06/24] vfio: rename API to reflect internal status
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (4 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 05/24] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 07/24] bus/pci: rename mismatching error labels Anatoly Burakov
` (17 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Chenbo Xia, Tomasz Duszynski, Ajit Khaparde, Vikas Gupta,
Feifei Wang, Dimon Zhao, Leon Yu, Sam Chen, Christian Koue Muf,
Serhii Iliushyk, Chaoyong He, Vijay Kumar Srivastava,
Bruce Richardson, Dmitry Kozlyuk, Maxime Coquelin
Currently, even though VFIO API is internal-only, it is still named with an
`rte_` prefix, which implies public API. To reflect the fact that this API
is no longer public, rename all symbols and macros from `rte_`/`RTE_` to
`dev_`/`DEV_`, and adjust all callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/release_26_11.rst | 7 +-
drivers/bus/cdx/cdx.c | 6 +-
drivers/bus/cdx/cdx_vfio.c | 14 +--
drivers/bus/fslmc/fslmc_bus.c | 4 +-
drivers/bus/fslmc/fslmc_vfio.c | 12 +--
drivers/bus/pci/linux/pci.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 38 +++----
drivers/bus/pci/pci_common.c | 10 +-
drivers/bus/platform/platform.c | 10 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 4 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 16 +--
drivers/net/ntnic/ntnic_ethdev.c | 10 +-
drivers/net/ntnic/ntnic_vfio.c | 22 ++--
drivers/vdpa/ifc/ifcvf_vdpa.c | 24 ++---
drivers/vdpa/nfp/nfp_vdpa.c | 24 ++---
drivers/vdpa/sfc/sfc_vdpa.c | 16 +--
drivers/vdpa/sfc/sfc_vdpa_hw.c | 12 +--
lib/eal/freebsd/eal.c | 66 ++++++------
lib/eal/include/{rte_vfio.h => dev_vfio.h} | 58 +++++-----
lib/eal/include/meson.build | 2 +-
lib/eal/linux/eal.c | 6 +-
lib/eal/linux/eal_vfio.c | 120 ++++++++++-----------
lib/eal/linux/eal_vfio_mp_sync.c | 6 +-
lib/eal/windows/eal.c | 14 +--
lib/vhost/vhost_user.c | 16 +--
26 files changed, 266 insertions(+), 263 deletions(-)
rename lib/eal/include/{rte_vfio.h => dev_vfio.h} (82%)
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index b5793b5c94..d5899b8d66 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -94,8 +94,8 @@ API Changes
=======================================================
* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
- These functions are now available only to EAL and drivers,
- and are no longer part of the public API.
+ These functions were renamed to ``dev_vfio_*``, are now available
+ only to EAL and drivers, and are no longer part of public API.
ABI Changes
@@ -117,6 +117,9 @@ ABI Changes
These functions are now available only to EAL and drivers,
and are no longer part of the public ABI.
+* vfio: The internal VFIO API was renamed from ``rte_vfio_*``/``RTE_VFIO_*``
+ to ``dev_vfio_*``/``DEV_VFIO_*``.
+
Known Issues
------------
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c0b46a41ad..ee8f24e67e 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -73,7 +73,7 @@
#include <rte_devargs.h>
#include <rte_kvargs.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include <eal_filesystem.h>
@@ -403,7 +403,7 @@ cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
RTE_SET_USED(dev);
- return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ return dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD,
(uintptr_t)addr, iova, len);
}
@@ -412,7 +412,7 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
RTE_SET_USED(dev);
- return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ return dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD,
(uintptr_t)addr, iova, len);
}
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..02a34d6f18 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -23,7 +23,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "bus_cdx_driver.h"
#include "cdx_logs.h"
@@ -121,7 +121,7 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ ret = dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
vfio_dev_fd);
if (ret < 0) {
CDX_BUS_ERR("Cannot release VFIO device");
@@ -155,7 +155,7 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ ret = dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
vfio_dev_fd);
if (ret < 0) {
CDX_BUS_ERR("Cannot release VFIO device");
@@ -400,7 +400,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
if (rte_intr_fd_set(dev->intr_handle, -1))
return -1;
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -477,7 +477,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
rte_free(vfio_res);
err_vfio_dev_fd:
- rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
return -1;
}
@@ -509,7 +509,7 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -535,7 +535,7 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return 0;
err_vfio_dev_fd:
- rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
return -1;
}
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 3626b12316..d55c414ead 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -17,7 +17,7 @@
#include <rte_memcpy.h>
#include <ethdev_driver.h>
#include <rte_mbuf_dyn.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
#include <fslmc_vfio.h>
@@ -500,7 +500,7 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 && dev_vfio_noiommu_is_enabled() == 0)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 8d39895bac..153e6112db 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -34,7 +34,7 @@
#include <rte_kvargs.h>
#include <dev_driver.h>
#include <rte_eal_memconfig.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
#include "fslmc_vfio.h"
@@ -204,7 +204,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (dev_vfio_noiommu_is_enabled() > 0)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
@@ -374,7 +374,7 @@ fslmc_get_group_id(const char *group_name,
int ret;
/* get group number */
- ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
+ ret = dev_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
if (ret <= 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
@@ -411,7 +411,7 @@ fslmc_vfio_open_group_fd(const char *group_name)
/* if primary, try to open the group */
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
/* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
goto add_vfio_group;
@@ -509,10 +509,10 @@ fslmc_vfio_open_container_fd(void)
/* if we're in a primary process, try to open the container */
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
if (vfio_container_fd < 0) {
DPAA2_BUS_ERR("Open VFIO container(%s), err(%d)",
- RTE_VFIO_CONTAINER_PATH, vfio_container_fd);
+ DEV_VFIO_CONTAINER_PATH, vfio_container_fd);
ret = vfio_container_fd;
goto err_exit;
}
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..af37a65c45 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -11,7 +11,7 @@
#include <rte_malloc.h>
#include <rte_devargs.h>
#include <rte_memcpy.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include "eal_filesystem.h"
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (dev_vfio_noiommu_is_enabled() == 1)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..86334d34e6 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -18,7 +18,7 @@
#include <rte_bus_pci.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
@@ -29,10 +29,10 @@
#include "pci_init.h"
#include "private.h"
-static struct rte_tailq_elem rte_vfio_tailq = {
+static struct rte_tailq_elem dev_vfio_tailq = {
.name = "VFIO_RESOURCE_LIST",
};
-EAL_REGISTER_TAILQ(rte_vfio_tailq)
+EAL_REGISTER_TAILQ(dev_vfio_tailq)
static int
pci_vfio_get_region(const struct rte_pci_device *dev, int index,
@@ -418,7 +418,7 @@ pci_vfio_is_ioport_bar(const struct rte_pci_device *dev, int vfio_dev_fd,
}
static int
-pci_rte_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
+pci_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
{
if (pci_vfio_setup_interrupts(dev, vfio_dev_fd) != 0) {
PCI_LOG(ERR, "Error setting up interrupts!");
@@ -738,7 +738,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
int i, j, ret;
struct mapped_pci_resource *vfio_res = NULL;
struct mapped_pci_res_list *vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
struct pci_map *maps;
@@ -752,7 +752,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -897,7 +897,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
free(reg);
}
- if (pci_rte_vfio_setup_device(dev, vfio_dev_fd) < 0) {
+ if (pci_vfio_setup_device(dev, vfio_dev_fd) < 0) {
PCI_LOG(ERR, "%s setup device failed", pci_addr);
goto err_map;
}
@@ -920,7 +920,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
err_vfio_res:
rte_free(vfio_res);
err_vfio_dev_fd:
- rte_vfio_release_device(rte_pci_get_sysfs_path(),
+ dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
}
@@ -935,7 +935,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
int j, ret, i = 0;
struct mapped_pci_resource *vfio_res = NULL;
struct mapped_pci_res_list *vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
struct pci_map *maps;
@@ -961,7 +961,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -1006,7 +1006,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
err_vfio_dev:
- rte_vfio_release_device(rte_pci_get_sysfs_path(),
+ dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
}
@@ -1101,7 +1101,7 @@ pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
vfio_dev_fd);
if (ret < 0) {
PCI_LOG(ERR, "Cannot release VFIO device");
@@ -1109,7 +1109,7 @@ pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
}
vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
/* if we haven't found our tailq entry, something's wrong */
@@ -1140,7 +1140,7 @@ pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
vfio_dev_fd);
if (ret < 0) {
PCI_LOG(ERR, "Cannot release VFIO device");
@@ -1148,7 +1148,7 @@ pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
}
vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
/* if we haven't found our tailq entry, something's wrong */
@@ -1195,7 +1195,7 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
+ if (dev_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
@@ -1300,11 +1300,11 @@ pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar,
int
pci_vfio_is_enabled(void)
{
- int status = rte_vfio_is_enabled("vfio_pci");
+ int status = dev_vfio_is_enabled("vfio_pci");
if (!status) {
- rte_vfio_enable("vfio");
- status = rte_vfio_is_enabled("vfio_pci");
+ dev_vfio_enable("vfio");
+ status = dev_vfio_is_enabled("vfio_pci");
}
return status;
}
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index dc8db80d3b..fcc4495b92 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -25,7 +25,7 @@
#include <rte_string_fns.h>
#include <rte_common.h>
#include <rte_devargs.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_tailq.h>
#include "private.h"
@@ -487,8 +487,8 @@ pci_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
* try fallback to VFIO.
*/
if (pdev->kdrv == RTE_PCI_KDRV_VFIO)
- return rte_vfio_container_dma_map
- (RTE_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
+ return dev_vfio_container_dma_map
+ (DEV_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
iova, len);
rte_errno = ENOTSUP;
return -1;
@@ -507,8 +507,8 @@ pci_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
* try fallback to VFIO.
*/
if (pdev->kdrv == RTE_PCI_KDRV_VFIO)
- return rte_vfio_container_dma_unmap
- (RTE_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
+ return dev_vfio_container_dma_unmap
+ (DEV_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
iova, len);
rte_errno = ENOTSUP;
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 90d865a8df..9d7df9be80 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -23,7 +23,7 @@
#include <rte_log.h>
#include <rte_memory.h>
#include <rte_string_fns.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
@@ -282,7 +282,7 @@ static void
device_cleanup(struct rte_platform_device *pdev)
{
device_unmap_resources(pdev);
- rte_vfio_release_device(PLATFORM_BUS_DEVICES_PATH, pdev->name, pdev->dev_fd);
+ dev_vfio_release_device(PLATFORM_BUS_DEVICES_PATH, pdev->name, pdev->dev_fd);
}
static int
@@ -292,7 +292,7 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
@@ -458,7 +458,7 @@ platform_bus_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t l
if (pdrv->dma_map != NULL)
return pdrv->dma_map(pdev, addr, iova, len);
- return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova, len);
+ return dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova, len);
}
static int
@@ -470,7 +470,7 @@ platform_bus_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t
if (pdrv->dma_unmap != NULL)
return pdrv->dma_unmap(pdev, addr, iova, len);
- return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova,
+ return dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova,
len);
}
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..053af0d227 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,7 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "bcmfs_device.h"
#include "bcmfs_logs.h"
@@ -25,7 +25,7 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = dev_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
@@ -51,7 +51,7 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
return 0;
map_failed:
- rte_vfio_release_device(path, dev_obj, *dev_fd);
+ dev_vfio_release_device(path, dev_obj, *dev_fd);
return ret;
}
@@ -87,7 +87,7 @@ bcmfs_release_vfio(struct bcmfs_device *dev)
/* unmap the addr */
munmap(dev->mmap_addr, dev->mmap_size);
/* release the device */
- ret = rte_vfio_release_device(dev->dirname, dev->name,
+ ret = dev_vfio_release_device(dev->dirname, dev->name,
dev->vfio_dev_fd);
if (ret < 0) {
BCMFS_LOG(ERR, "cannot release device");
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..6a6cf44752 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -2,7 +2,7 @@
* Copyright(c) 2025 Huawei Technologies Co., Ltd
*/
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "hinic3_compat.h"
#include "hinic3_cmd.h"
@@ -78,7 +78,7 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && dev_vfio_noiommu_is_enabled() != 1;
}
int
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index ec6840d60e..f1522772f5 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -4,7 +4,7 @@
#include "nbl_userdev.h"
#include <uapi/linux/vfio.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#define NBL_USERDEV_EVENT_CLB_NAME "nbl_userspace_mem_event_clb"
#define NBL_USERDEV_BAR0_SIZE 65536
@@ -405,7 +405,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
snprintf(pathname, sizeof(pathname),
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
- ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
+ ret = dev_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
if (ret <= 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
@@ -414,7 +414,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
NBL_LOG(DEBUG, "nbl vfio group number %d", common->iommu_group_num);
/* vfio_container */
if (nbl_default_container < 0) {
- container = rte_vfio_container_create();
+ container = dev_vfio_container_create();
container_create = 1;
if (container < 0) {
@@ -424,7 +424,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = dev_vfio_container_group_bind(container, common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
@@ -535,10 +535,10 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
- rte_vfio_container_destroy(container);
+ dev_vfio_container_destroy(container);
return -1;
}
@@ -550,7 +550,7 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
rte_mcfg_mem_read_lock();
TAILQ_REMOVE(&nbl_adapter_list, adapter, next);
close(common->devfd);
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
+ vfio_group_fd = dev_vfio_container_group_bind(nbl_default_container,
common->iommu_group_num);
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
@@ -558,7 +558,7 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
+ ret = dev_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
if (ret)
NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
if (!nbl_group_count) {
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..d9744a7d05 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -7,7 +7,7 @@
#include <rte_eal.h>
#include <rte_dev.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_ethdev.h>
#include <rte_bus_pci.h>
#include <ethdev_pci.h>
@@ -2690,12 +2690,12 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
- NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
+ const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_noiommu_is_enabled();
+ NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_dev_vfio_no_io_mmu_enabled);
- if (n_rte_vfio_no_io_mmu_enabled) {
+ if (n_dev_vfio_no_io_mmu_enabled) {
NT_LOG(ERR, NTNIC, "vfio_no_iommu_enabled=%d: this PMD needs VFIO IOMMU",
- n_rte_vfio_no_io_mmu_enabled);
+ n_dev_vfio_no_io_mmu_enabled);
return -1;
}
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..8489a59859 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -6,7 +6,7 @@
#include <rte_malloc.h>
#include <rte_memory.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_dev.h>
#include <rte_bus_pci.h>
#include <rte_spinlock.h>
@@ -70,16 +70,16 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
if (ret <= 0)
return -1;
if (vf_num == 0) {
/* use default container for pf0 */
- vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
+ vfio->container_fd = DEV_VFIO_DEFAULT_CONTAINER_FD;
} else {
- vfio->container_fd = rte_vfio_container_create();
+ vfio->container_fd = dev_vfio_container_create();
if (vfio->container_fd < 0) {
NT_LOG(ERR, NTNIC,
@@ -88,7 +88,7 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
}
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
+ vfio->group_fd = dev_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
if (vfio->group_fd < 0) {
NT_LOG(ERR, NTNIC,
@@ -115,8 +115,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
err:
- if (vfio->container_fd != RTE_VFIO_DEFAULT_CONTAINER_FD)
- rte_vfio_container_destroy(vfio->container_fd);
+ if (vfio->container_fd != DEV_VFIO_DEFAULT_CONTAINER_FD)
+ dev_vfio_container_destroy(vfio->container_fd);
return -1;
}
@@ -135,7 +135,7 @@ nthw_vfio_remove(int vf_num)
return -1;
}
- rte_vfio_container_destroy(vfio->container_fd);
+ dev_vfio_container_destroy(vfio->container_fd);
return 0;
}
@@ -169,14 +169,14 @@ nthw_vfio_dma_map(int vf_num, void *virt_addr, uint64_t *iova_addr, uint64_t siz
vf_num, virt_addr, rte_malloc_virt2iova(virt_addr), gp_virt_base, vfio->iova_addr,
size);
- int res = rte_vfio_container_dma_map(vfio->container_fd, gp_virt_base, vfio->iova_addr,
+ int res = dev_vfio_container_dma_map(vfio->container_fd, gp_virt_base, vfio->iova_addr,
size);
NT_LOG(DBG, NTNIC, "VFIO MMAP res %i, container_fd %i, vf_num %i", res,
vfio->container_fd, vf_num);
if (res) {
- NT_LOG(ERR, NTNIC, "rte_vfio_container_dma_map failed: res %d", res);
+ NT_LOG(ERR, NTNIC, "dev_vfio_container_dma_map failed: res %d", res);
return -1;
}
@@ -210,7 +210,7 @@ nthw_vfio_dma_unmap(int vf_num, void *virt_addr, uint64_t iova_addr, uint64_t si
return -1;
}
- int res = rte_vfio_container_dma_unmap(vfio->container_fd, gp_virt_base, iova_addr, size);
+ int res = dev_vfio_container_dma_unmap(vfio->container_fd, gp_virt_base, iova_addr, size);
if (res != 0) {
NT_LOG(ERR, NTNIC,
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..36d06f1e3f 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -20,7 +20,7 @@
#include <rte_vhost.h>
#include <rte_vdpa.h>
#include <vdpa_driver.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_spinlock.h>
#include <rte_log.h>
#include <rte_kvargs.h>
@@ -182,18 +182,18 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
&iommu_group_num);
if (ret <= 0) {
DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
return -1;
}
- internal->vfio_container_fd = rte_vfio_container_create();
+ internal->vfio_container_fd = dev_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
+ internal->vfio_group_fd = dev_vfio_container_group_bind(
internal->vfio_container_fd, iommu_group_num);
if (internal->vfio_group_fd < 0)
goto err;
@@ -216,7 +216,7 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
return 0;
err:
- rte_vfio_container_destroy(internal->vfio_container_fd);
+ dev_vfio_container_destroy(internal->vfio_container_fd);
return -1;
}
@@ -246,7 +246,7 @@ ifcvf_dma_map(struct ifcvf_internal *internal, bool do_map)
reg->host_user_addr, reg->guest_phys_addr, reg->size);
if (do_map) {
- ret = rte_vfio_container_dma_map(vfio_container_fd,
+ ret = dev_vfio_container_dma_map(vfio_container_fd,
reg->host_user_addr, reg->guest_phys_addr,
reg->size);
if (ret < 0) {
@@ -254,7 +254,7 @@ ifcvf_dma_map(struct ifcvf_internal *internal, bool do_map)
goto exit;
}
} else {
- ret = rte_vfio_container_dma_unmap(vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(vfio_container_fd,
reg->host_user_addr, reg->guest_phys_addr,
reg->size);
if (ret < 0) {
@@ -388,7 +388,7 @@ vdpa_ifcvf_stop(struct ifcvf_internal *internal)
if (RTE_VHOST_NEED_LOG(features)) {
ifcvf_disable_logging(hw);
rte_vhost_get_log_base(internal->vid, &log_base, &log_size);
- rte_vfio_container_dma_unmap(internal->vfio_container_fd,
+ dev_vfio_container_dma_unmap(internal->vfio_container_fd,
log_base, IFCVF_LOG_BASE, log_size);
/*
* IFCVF marks dirty memory pages for only packet buffer,
@@ -791,7 +791,7 @@ m_ifcvf_start(struct ifcvf_internal *internal)
vring_init(&internal->m_vring[i], vq.size, vring_buf,
rte_mem_page_size());
- ret = rte_vfio_container_dma_map(internal->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(internal->vfio_container_fd,
(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
if (ret < 0) {
DRV_LOG(ERR, "mediated vring DMA map failed.");
@@ -900,7 +900,7 @@ m_ifcvf_stop(struct ifcvf_internal *internal)
size = RTE_ALIGN_CEIL(vring_size(vq.size, rte_mem_page_size()),
rte_mem_page_size());
- rte_vfio_container_dma_unmap(internal->vfio_container_fd,
+ dev_vfio_container_dma_unmap(internal->vfio_container_fd,
(uint64_t)(uintptr_t)internal->m_vring[i].desc,
m_vring_iova, size);
@@ -1205,7 +1205,7 @@ ifcvf_set_features(int vid)
ifcvf_sw_fallback_switchover(internal);
} else {
rte_vhost_get_log_base(vid, &log_base, &log_size);
- rte_vfio_container_dma_map(internal->vfio_container_fd,
+ dev_vfio_container_dma_map(internal->vfio_container_fd,
log_base, IFCVF_LOG_BASE, log_size);
ifcvf_enable_logging(&internal->hw, IFCVF_LOG_BASE, log_size);
}
@@ -1824,7 +1824,7 @@ ifcvf_pci_remove(struct rte_pci_device *pci_dev)
DRV_LOG(ERR, "failed to update datapath %s", pci_dev->name);
rte_pci_unmap_device(internal->pdev);
- rte_vfio_container_destroy(internal->vfio_container_fd);
+ dev_vfio_container_destroy(internal->vfio_container_fd);
rte_vdpa_unregister_device(internal->vdev);
pthread_mutex_lock(&internal_list_lock);
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4c0ab3ff30..94740e768d 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -12,7 +12,7 @@
#include <nfp_common_pci.h>
#include <nfp_dev.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
#include <vdpa_driver.h>
@@ -120,16 +120,16 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
&device->iommu_group);
if (ret <= 0)
return -1;
- device->vfio_container_fd = rte_vfio_container_create();
+ device->vfio_container_fd = dev_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
+ device->vfio_group_fd = dev_vfio_container_group_bind(
device->vfio_container_fd, device->iommu_group);
if (device->vfio_group_fd < 0)
goto container_destroy;
@@ -146,9 +146,9 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
return 0;
group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
+ dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
- rte_vfio_container_destroy(device->vfio_container_fd);
+ dev_vfio_container_destroy(device->vfio_container_fd);
return -1;
}
@@ -157,8 +157,8 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
- rte_vfio_container_destroy(device->vfio_container_fd);
+ dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
+ dev_vfio_container_destroy(device->vfio_container_fd);
}
static int
@@ -173,7 +173,7 @@ nfp_vdpa_dma_do_unmap(struct rte_vhost_memory *mem,
for (i = 0; i < times; i++) {
region = &mem->regions[i];
- ret = rte_vfio_container_dma_unmap(vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(vfio_container_fd,
region->host_user_addr, region->guest_phys_addr,
region->size);
if (ret < 0) {
@@ -197,7 +197,7 @@ nfp_vdpa_dma_do_map(struct rte_vhost_memory *mem,
for (i = 0; i < times; i++) {
region = &mem->regions[i];
- ret = rte_vfio_container_dma_map(vfio_container_fd,
+ ret = dev_vfio_container_dma_map(vfio_container_fd,
region->host_user_addr, region->guest_phys_addr,
region->size);
if (ret < 0) {
@@ -282,7 +282,7 @@ nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
rte_mem_page_size());
- rte_vfio_container_dma_unmap(device->vfio_container_fd,
+ dev_vfio_container_dma_unmap(device->vfio_container_fd,
(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
m_vring_iova, size);
@@ -315,7 +315,7 @@ nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
- ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(device->vfio_container_fd,
(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
if (ret != 0) {
DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..c65f716a15 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -12,7 +12,7 @@
#include <rte_errno.h>
#include <rte_kvargs.h>
#include <rte_string_fns.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_vhost.h>
#include "efx.h"
@@ -74,13 +74,13 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
rte_pci_device_name(&dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- sva->vfio_container_fd = rte_vfio_container_create();
+ sva->vfio_container_fd = dev_vfio_container_create();
if (sva->vfio_container_fd < 0) {
sfc_vdpa_err(sva, "failed to create VFIO container");
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
+ rc = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
&sva->iommu_group_num);
if (rc <= 0) {
sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
@@ -89,7 +89,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
}
sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
+ dev_vfio_container_group_bind(sva->vfio_container_fd,
sva->iommu_group_num);
if (sva->vfio_group_fd < 0) {
sfc_vdpa_err(sva,
@@ -109,7 +109,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
+ if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
sva->iommu_group_num) != 0) {
sfc_vdpa_err(sva,
"failed to unbind IOMMU group %d from container %d",
@@ -118,7 +118,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
fail_group_bind:
fail_get_group_num:
- if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
+ if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
}
@@ -132,14 +132,14 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
+ if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
sva->iommu_group_num) != 0) {
sfc_vdpa_err(sva,
"failed to unbind IOMMU group %d from container %d",
sva->iommu_group_num, sva->vfio_container_fd);
}
- if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
+ if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
sva->vfio_container_fd);
diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c
index 7e43719f53..65050b9640 100644
--- a/drivers/vdpa/sfc/sfc_vdpa_hw.c
+++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c
@@ -6,7 +6,7 @@
#include <rte_common.h>
#include <rte_errno.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_vhost.h>
#include "efx.h"
@@ -64,7 +64,7 @@ sfc_vdpa_dma_alloc(struct sfc_vdpa_adapter *sva, const char *name,
mcdi_iova = SFC_VDPA_DEFAULT_MCDI_IOVA;
for (;;) {
- ret = rte_vfio_container_dma_map(sva->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(sva->vfio_container_fd,
(uint64_t)mz->addr, mcdi_iova,
mcdi_buff_size);
if (ret == 0)
@@ -98,7 +98,7 @@ sfc_vdpa_dma_free(struct sfc_vdpa_adapter *sva, efsys_mem_t *esmp)
sfc_vdpa_log_init(sva, "name=%s", esmp->esm_mz->name);
- ret = rte_vfio_container_dma_unmap(sva->vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(sva->vfio_container_fd,
(uint64_t)esmp->esm_base,
esmp->esm_addr, sva->mcdi_buff_size);
if (ret < 0)
@@ -140,7 +140,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
mem_reg = &vhost_mem->regions[i];
if (do_map) {
- rc = rte_vfio_container_dma_map(vfio_container_fd,
+ rc = dev_vfio_container_dma_map(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
@@ -151,7 +151,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
goto failed_vfio_dma_map;
}
} else {
- rc = rte_vfio_container_dma_unmap(vfio_container_fd,
+ rc = dev_vfio_container_dma_unmap(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
@@ -171,7 +171,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
failed_vfio_dma_map:
for (j = 0; j < i; j++) {
mem_reg = &vhost_mem->regions[j];
- rte_vfio_container_dma_unmap(vfio_container_fd,
+ dev_vfio_container_dma_unmap(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 991ca45064..d6f75af20b 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -41,7 +41,7 @@
#include <rte_dev.h>
#include <rte_devargs.h>
#include <rte_version.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <malloc_heap.h>
#include <telemetry_internal.h>
@@ -819,8 +819,8 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
-int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
+int dev_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
__rte_unused struct vfio_device_info *device_info)
@@ -829,8 +829,8 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int rte_vfio_release_device(__rte_unused const char *sysfs_base,
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
+int dev_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
{
@@ -838,41 +838,41 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
-int rte_vfio_enable(__rte_unused const char *modname)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
+int dev_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
-rte_vfio_cleanup(void)
+dev_vfio_cleanup(void)
{
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
-int rte_vfio_is_enabled(__rte_unused const char *modname)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
+int dev_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
+int dev_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
+int dev_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
+dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *iommu_group_num)
{
@@ -880,59 +880,59 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
-rte_vfio_get_container_fd(void)
+dev_vfio_get_container_fd(void)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
+dev_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
-rte_vfio_container_create(void)
+dev_vfio_container_create(void)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
-rte_vfio_container_destroy(__rte_unused int container_fd)
+dev_vfio_container_destroy(__rte_unused int container_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
+dev_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
+dev_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(__rte_unused int container_fd,
+dev_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
@@ -941,9 +941,9 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(__rte_unused int container_fd,
+dev_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/dev_vfio.h
similarity index 82%
rename from lib/eal/include/rte_vfio.h
rename to lib/eal/include/dev_vfio.h
index e76181c83c..712ceed058 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -2,14 +2,14 @@
* Copyright(c) 2017 6WIND S.A.
*/
-#ifndef _RTE_VFIO_H_
-#define _RTE_VFIO_H_
+#ifndef _DEV_VFIO_H_
+#define _DEV_VFIO_H_
/**
* @file
* @internal
*
- * RTE VFIO internal API.
+ * VFIO device API.
*
* This library provides VFIO related utility functions for use by drivers.
*/
@@ -25,11 +25,11 @@ extern "C" {
#ifdef RTE_EXEC_ENV_LINUX
-#define RTE_VFIO_DIR "/dev/vfio"
-#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
-#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
-#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
+#define DEV_VFIO_DIR "/dev/vfio"
+#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
+#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
+#define DEV_VFIO_NOIOMMU_MODE \
"/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -37,7 +37,7 @@ extern "C" {
/* we don't need an actual definition, only pointer is used */
struct vfio_device_info;
-#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+#define DEV_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
* @internal
@@ -66,7 +66,7 @@ struct vfio_device_info;
* >1 if the device cannot be managed this way.
*/
__rte_internal
-int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
+int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
@@ -90,7 +90,7 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* <0 on failure.
*/
__rte_internal
-int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
+int dev_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
@@ -107,7 +107,7 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* <0 on failure.
*/
__rte_internal
-int rte_vfio_enable(const char *modname);
+int dev_vfio_enable(const char *modname);
/**
* @internal
@@ -116,7 +116,7 @@ int rte_vfio_enable(const char *modname);
* This function is only relevant to Linux.
*/
__rte_internal
-void rte_vfio_cleanup(void);
+void dev_vfio_cleanup(void);
/**
* @internal
@@ -132,7 +132,7 @@ void rte_vfio_cleanup(void);
* 0 otherwise.
*/
__rte_internal
-int rte_vfio_is_enabled(const char *modname);
+int dev_vfio_is_enabled(const char *modname);
/**
* @internal
@@ -146,7 +146,7 @@ int rte_vfio_is_enabled(const char *modname);
* <0 for errors.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+int dev_vfio_noiommu_is_enabled(void);
/**
* @internal
@@ -164,7 +164,7 @@ int rte_vfio_noiommu_is_enabled(void);
*/
__rte_internal
int
-rte_vfio_clear_group(int vfio_group_fd);
+dev_vfio_clear_group(int vfio_group_fd);
/**
* @internal
@@ -189,7 +189,7 @@ rte_vfio_clear_group(int vfio_group_fd);
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
+dev_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
@@ -216,7 +216,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
+dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
@@ -232,7 +232,7 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
*/
__rte_internal
int
-rte_vfio_get_container_fd(void);
+dev_vfio_get_container_fd(void);
/**
* @internal
@@ -250,7 +250,7 @@ rte_vfio_get_container_fd(void);
*/
__rte_internal
int
-rte_vfio_get_group_fd(int iommu_group_num);
+dev_vfio_get_group_fd(int iommu_group_num);
/**
* @internal
@@ -270,7 +270,7 @@ rte_vfio_get_group_fd(int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_create(void);
+dev_vfio_container_create(void);
/**
* @internal
@@ -285,7 +285,7 @@ rte_vfio_container_create(void);
*/
__rte_internal
int
-rte_vfio_container_destroy(int container_fd);
+dev_vfio_container_destroy(int container_fd);
/**
* @internal
@@ -303,7 +303,7 @@ rte_vfio_container_destroy(int container_fd);
*/
__rte_internal
int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
+dev_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
* @internal
@@ -321,14 +321,14 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
+dev_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
+ * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
* use the default container.
*
* @param vaddr
@@ -346,7 +346,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
@@ -354,7 +354,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
+ * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
* use the default container.
*
* @param vaddr
@@ -372,11 +372,11 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
*/
__rte_internal
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
#ifdef __cplusplus
}
#endif
-#endif /* _RTE_VFIO_H_ */
+#endif /* _DEV_VFIO_H_ */
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index 9282143918..8d3a6d5e00 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -60,7 +60,7 @@ headers += files(
driver_sdk_headers = files(
'bus_driver.h',
'dev_driver.h',
- 'rte_vfio.h',
+ 'dev_vfio.h',
)
# special case install the generic headers, since they go in a subdir
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 5577bab24a..77691a7908 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -41,7 +41,7 @@
#include <rte_bus.h>
#include <rte_version.h>
#include <malloc_heap.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <telemetry_internal.h>
#include <eal_export.h>
@@ -771,7 +771,7 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (rte_vfio_enable("vfio")) {
+ if (dev_vfio_enable("vfio")) {
rte_eal_init_alert("Cannot init VFIO");
rte_errno = EAGAIN;
goto err_out;
@@ -985,7 +985,7 @@ rte_eal_cleanup(void)
rte_service_finalize();
eal_bus_cleanup();
- rte_vfio_cleanup();
+ dev_vfio_cleanup();
rte_mp_channel_cleanup();
rte_eal_alarm_cleanup();
rte_trace_save();
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 84bb845e96..162202d32e 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -15,7 +15,7 @@
#include <rte_log.h>
#include <rte_memory.h>
#include <rte_eal_memconfig.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include "eal_filesystem.h"
@@ -366,7 +366,7 @@ vfio_open_group_fd(int iommu_group_num, bool mp_request)
/* if not requesting via mp, open the group locally */
if (!mp_request) {
/* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
if (vfio_group_fd < 0) {
/* if file not found, it's not an error */
@@ -377,7 +377,7 @@ vfio_open_group_fd(int iommu_group_num, bool mp_request)
}
/* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
+ snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT,
iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
if (vfio_group_fd < 0) {
@@ -524,7 +524,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
{
unsigned int i;
- if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
+ if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD)
return default_vfio_cfg;
for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
@@ -535,9 +535,9 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+dev_vfio_get_group_fd(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -688,7 +688,7 @@ vfio_sync_default_container(void)
if (rte_eal_process_type() != RTE_PROC_SECONDARY)
return -1;
- /* default container fd should have been opened in rte_vfio_enable() */
+ /* default container fd should have been opened in dev_vfio_enable() */
if (!vfio_enabled ||
default_vfio_cfg->vfio_container_fd < 0) {
EAL_LOG(ERR, "VFIO support is not initialized");
@@ -734,9 +734,9 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
int
-rte_vfio_clear_group(int vfio_group_fd)
+dev_vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -758,9 +758,9 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
-rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
+dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
{
struct vfio_group_status group_status = {
@@ -777,7 +777,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
eal_get_internal_configuration();
/* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
if (ret == 0) {
EAL_LOG(NOTICE,
"%s not managed by VFIO driver, skipping",
@@ -790,7 +790,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -816,14 +816,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -844,7 +844,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -868,7 +868,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -885,7 +885,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -954,7 +954,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -991,7 +991,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1004,7 +1004,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
strerror(errno));
close(*vfio_dev_fd);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
vfio_group_device_get(vfio_group_fd);
@@ -1012,9 +1012,9 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
+dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
{
struct vfio_config *vfio_cfg;
@@ -1029,7 +1029,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
rte_mcfg_mem_read_lock();
/* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
if (ret <= 0) {
EAL_LOG(WARNING, "%s not managed by VFIO driver",
dev_addr);
@@ -1039,9 +1039,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "dev_vfio_get_group_fd failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1077,7 +1077,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (dev_vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -1101,9 +1101,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
int
-rte_vfio_enable(const char *modname)
+dev_vfio_enable(const char *modname)
{
/* initialize group list */
unsigned int i, j;
@@ -1147,7 +1147,7 @@ rte_vfio_enable(const char *modname)
}
/* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
+ dir = opendir(DEV_VFIO_DIR);
if (dir == NULL) {
EAL_LOG(DEBUG,
"VFIO directory does not exist, skipping VFIO support...");
@@ -1180,9 +1180,9 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
-rte_vfio_is_enabled(const char *modname)
+dev_vfio_is_enabled(const char *modname)
{
const int mod_available = rte_eal_check_module(modname) > 0;
return vfio_enabled && mod_available;
@@ -1220,9 +1220,9 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
+dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
@@ -1231,7 +1231,7 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
return -1;
if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
+ ret = dev_vfio_setup_device(sysfs_base, dev_addr,
vfio_dev_fd, device_info);
if (ret)
return -1;
@@ -1299,10 +1299,10 @@ vfio_open_container_fd(bool mp_request)
/* if not requesting via mp, open a new container locally */
if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
if (vfio_container_fd < 0) {
EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
return -1;
}
@@ -1354,12 +1354,12 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
-rte_vfio_get_container_fd(void)
+dev_vfio_get_container_fd(void)
{
/* Return the default container fd if VFIO is enabled.
- * The default container is set up during rte_vfio_enable().
+ * The default container is set up during dev_vfio_enable().
* This function does not create a new container.
*/
if (!vfio_enabled)
@@ -1368,9 +1368,9 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
+dev_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
{
char linkname[PATH_MAX];
@@ -2039,15 +2039,15 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
int
-rte_vfio_noiommu_is_enabled(void)
+dev_vfio_noiommu_is_enabled(void)
{
int fd;
ssize_t cnt;
char c;
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
if (fd < 0) {
if (errno != ENOENT) {
EAL_LOG(ERR, "Cannot open VFIO noiommu file "
@@ -2072,9 +2072,9 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
-rte_vfio_container_create(void)
+dev_vfio_container_create(void)
{
unsigned int i;
@@ -2100,9 +2100,9 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
-rte_vfio_container_destroy(int container_fd)
+dev_vfio_container_destroy(int container_fd)
{
struct vfio_config *vfio_cfg;
unsigned int i;
@@ -2115,7 +2115,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ dev_vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2127,9 +2127,9 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2142,9 +2142,9 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+dev_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
@@ -2183,9 +2183,9 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct vfio_config *vfio_cfg;
@@ -2204,9 +2204,9 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct vfio_config *vfio_cfg;
@@ -2273,9 +2273,9 @@ vfio_cleanup_config(struct vfio_config *vfio_cfg)
memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
-rte_vfio_cleanup(void)
+dev_vfio_cleanup(void)
{
unsigned int i;
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..ae1c8b40b9 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -8,7 +8,7 @@
#include <rte_errno.h>
#include <rte_log.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal.h>
#include "eal_private.h"
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = dev_vfio_get_group_fd(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
@@ -49,7 +49,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
break;
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
- fd = rte_vfio_get_container_fd();
+ fd = dev_vfio_get_container_fd();
if (fd < 0)
r->result = SOCKET_ERR;
else {
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index b8771d4d85..d59ce9331f 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -24,7 +24,7 @@
#include <eal_options.h>
#include <eal_private.h>
#include <rte_service_component.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "eal_firmware.h"
#include "eal_hugepages.h"
@@ -152,9 +152,9 @@ rte_eal_cleanup(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
-rte_vfio_cleanup(void)
+dev_vfio_cleanup(void)
{
}
@@ -459,9 +459,9 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(__rte_unused int container_fd,
+dev_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
@@ -470,9 +470,9 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(__rte_unused int container_fd,
+dev_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 020c993b29..35aa70dffc 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -42,7 +42,7 @@
#include <rte_common.h>
#include <rte_malloc.h>
#include <rte_log.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_errno.h>
#include "iotlb.h"
@@ -191,7 +191,7 @@ async_dma_map_region(struct virtio_net *dev, struct rte_vhost_mem_region *reg, b
continue;
if (do_map) {
- ret = rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ ret = dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD,
page->host_user_addr,
page->host_iova,
page->size);
@@ -216,7 +216,7 @@ async_dma_map_region(struct virtio_net *dev, struct rte_vhost_mem_region *reg, b
return -1;
}
} else {
- ret = rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ ret = dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD,
page->host_user_addr,
page->host_iova,
page->size);
@@ -267,7 +267,7 @@ free_all_mem_regions(struct virtio_net *dev)
if (!dev || !dev->mem)
return;
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map(dev, false);
for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
@@ -1564,7 +1564,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
dev->mem->nregions++;
}
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map(dev, true);
if (vhost_user_postcopy_register(dev, main_fd, ctx) < 0)
@@ -1750,7 +1750,7 @@ vhost_user_add_mem_reg(struct virtio_net **pdev,
dev->mem->nregions++;
- if (dev->async_copy && rte_vfio_is_enabled("vfio")) {
+ if (dev->async_copy && dev_vfio_is_enabled("vfio")) {
if (async_dma_map_region(dev, reg, true) < 0)
goto free_new_region_no_dma;
}
@@ -1791,7 +1791,7 @@ vhost_user_add_mem_reg(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_OK;
free_new_region:
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map_region(dev, reg, false);
free_new_region_no_dma:
remove_guest_pages(dev, reg);
@@ -1827,7 +1827,7 @@ vhost_user_rem_mem_reg(struct virtio_net **pdev,
if (region->userspace_addr == current_region->guest_user_addr
&& region->guest_phys_addr == current_region->guest_phys_addr
&& region->memory_size == current_region->size) {
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map_region(dev, current_region, false);
if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
vhost_user_iotlb_cache_remove(dev,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 07/24] bus/pci: rename mismatching error labels
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (5 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 06/24] vfio: rename API to reflect internal status Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 08/24] vfio: split get device info from setup Anatoly Burakov
` (16 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Currently, there are two VFIO map functions - primary and secondary -
that use a very similar goto error pattern. However, the label names are
confusing.
Specifically, for primary, we have three error labels: vfio_res (release
VFIO resource), vfio_dev_fd (close the fd), and map (unmap all the mapped
regions). For secondary, there is vfio_dev (close the fd), and
vfio_dev_fd (unmap all mapped regions).
For consistency, align secondary process function's error labels to match
semantics of how they work in primary process function.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 86334d34e6..2d8144deb0 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -968,7 +968,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
- goto err_vfio_dev;
+ goto err_vfio_dev_fd;
/* map BARs */
maps = vfio_res->maps;
@@ -979,14 +979,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
if (ret < 0) {
PCI_LOG(ERR, "%s sparse mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
} else {
ret = pci_vfio_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
if (ret < 0) {
PCI_LOG(ERR, "%s mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
}
@@ -995,17 +995,17 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
/* we need save vfio_dev_fd, so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
if (rte_intr_dev_fd_set(dev->vfio_req_intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
return 0;
-err_vfio_dev_fd:
+err_map:
for (j = 0; j < i; j++) {
if (maps[j].addr)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
-err_vfio_dev:
+err_vfio_dev_fd:
dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 08/24] vfio: split get device info from setup
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (6 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 07/24] bus/pci: rename mismatching error labels Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 09/24] vfio: add container device assignment API Anatoly Burakov
` (15 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 3 +--
lib/eal/include/dev_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 59 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 02a34d6f18..531e4ccc8d 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 2d8144deb0..3cc7458419 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (dev_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (dev_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 9d7df9be80..950306e2d9 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = dev_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index 053af0d227..ad244d9756 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = dev_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = dev_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = dev_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index d6f75af20b..2990cf14cd 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -822,8 +822,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int dev_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 712ceed058..3719cc6adc 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -196,19 +193,16 @@ dev_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with `dev_vfio_setup_device()` first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -216,8 +210,7 @@ dev_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 162202d32e..a64173fe7f 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -761,7 +761,7 @@ dev_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -978,7 +978,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -995,18 +995,8 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1222,26 +1212,18 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
int
-dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = dev_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 09/24] vfio: add container device assignment API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (7 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 08/24] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 10/24] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (14 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `dev_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/dev_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 2990cf14cd..0650abf406 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -950,3 +950,13 @@ dev_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 3719cc6adc..85b4dbd3fc 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -280,6 +280,32 @@ __rte_internal
int
dev_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `dev_vfio_setup_device()` call to set up the device with
+ * the VFIO container specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index a64173fe7f..29c03d01b4 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2109,6 +2109,38 @@ dev_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = dev_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 10/24] net/nbl: do not use VFIO group bind API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (8 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 09/24] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 11/24] net/ntnic: use container device assignment API Anatoly Burakov
` (13 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 24 ++++++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index f1522772f5..b3b6548bb8 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,15 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+
+ snprintf(path, sizeof(path), DEV_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +433,16 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = dev_vfio_container_group_bind(container, common->iommu_group_num);
+ /*
+ * This assumes one device per group, as kernel will return -EBUSY on
+ * attempting to open the same group fd for a different device.
+ */
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +549,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
dev_vfio_container_destroy(container);
@@ -550,17 +563,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
rte_mcfg_mem_read_lock();
TAILQ_REMOVE(&nbl_adapter_list, adapter, next);
close(common->devfd);
- vfio_group_fd = dev_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = dev_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 11/24] net/ntnic: use container device assignment API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (9 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 10/24] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 12/24] vdpa/ifc: " Anatoly Burakov
` (12 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 8489a59859..cacd81643f 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = DEV_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = dev_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = dev_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 12/24] vdpa/ifc: use container device assignment API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (10 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 11/24] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 13/24] vdpa/nfp: " Anatoly Burakov
` (11 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 36d06f1e3f..04ed2052d3 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = dev_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = dev_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (dev_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 13/24] vdpa/nfp: use container device assignment API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (11 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 12/24] vdpa/ifc: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 14/24] vdpa/sfc: " Anatoly Burakov
` (10 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 94740e768d..76d7d616a6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -120,33 +118,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = dev_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = dev_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = dev_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
dev_vfio_container_destroy(device->vfio_container_fd);
@@ -157,7 +148,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
dev_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1023,22 +1013,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1190,7 +1164,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 14/24] vdpa/sfc: use container device assignment API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (12 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 13/24] vdpa/nfp: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 15/24] vdpa/mlx5: remove group-related API Anatoly Burakov
` (9 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index c65f716a15..bd5420cfa5 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- dev_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = dev_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 15/24] vdpa/mlx5: remove group-related API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (13 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 14/24] vdpa/sfc: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 16/24] vhost: remove group-related API from driver Anatoly Burakov
` (8 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 16/24] vhost: remove group-related API from driver
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (14 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 15/24] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 17/24] vfio: remove group-based API Anatoly Burakov
` (7 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
lib/vhost/vdpa_driver.h | 3 ---
3 files changed, 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 345a621716..99355b8afd 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to respond to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 8a82c815e3..e4a754d037 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,10 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 17/24] vfio: remove group-based API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (15 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 16/24] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 18/24] vfio: cleanup and refactor Anatoly Burakov
` (6 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 33 ---------------
lib/eal/include/dev_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 130 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0650abf406..44b5e820fe 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -862,13 +862,6 @@ int dev_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
-int dev_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -887,14 +880,6 @@ dev_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
-int
-dev_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
@@ -911,24 +896,6 @@ dev_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
-int
-dev_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
-int
-dev_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
dev_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 85b4dbd3fc..98b7d7b442 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -145,24 +145,6 @@ int dev_vfio_is_enabled(const char *modname);
__rte_internal
int dev_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-dev_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -227,24 +209,6 @@ __rte_internal
int
dev_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-dev_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -306,42 +270,6 @@ int
dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-dev_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-dev_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 29c03d01b4..345558f43a 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -70,6 +70,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -535,9 +538,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-dev_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -734,9 +736,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
-int
-dev_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -790,7 +791,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -816,14 +817,14 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -844,7 +845,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -868,7 +869,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -885,7 +886,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -954,7 +955,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -991,7 +992,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1029,9 +1030,9 @@ dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "dev_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1067,7 +1068,7 @@ dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (dev_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2097,7 +2098,7 @@ dev_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- dev_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2129,7 +2130,7 @@ dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = dev_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2141,9 +2142,8 @@ dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
-int
-dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2156,9 +2156,8 @@ dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
-int
-dev_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index ae1c8b40b9..66341a701a 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = dev_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 18/24] vfio: cleanup and refactor
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (16 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 17/24] vfio: remove group-based API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 19/24] bus/pci: use the new VFIO mode API Anatoly Burakov
` (5 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple dev_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, dev_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Add device deduplication based on sysfs path and device address.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
- Rename internal structs and macros to use vfio_* namespace
This will change behavior of the following functions:
- `dev_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `dev_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `dev_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
All users of `dev_vfio_setup_device` and `dev_vfio_get_group_num` have been
adjusted to account for API change.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/dev_vfio.h | 254 ++-
lib/eal/linux/eal_vfio.c | 2497 ++++++++--------------
lib/eal/linux/eal_vfio.h | 159 +-
lib/eal/linux/eal_vfio_group.c | 983 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 62 +-
lib/eal/linux/meson.build | 1 +
14 files changed, 2250 insertions(+), 1772 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 531e4ccc8d..7ba0937950 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <dev_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 153e6112db..c2b6ac1f3f 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -376,9 +376,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = dev_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 3cc7458419..0279c6bae1 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <dev_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index ad244d9756..0e16af0365 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <dev_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = dev_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index b3b6548bb8..10dd179ac6 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -415,7 +415,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = dev_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 44b5e820fe..55e1ee63b2 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -927,3 +927,19 @@ dev_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
+int
+dev_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_mode)
+enum dev_vfio_mode
+dev_vfio_get_mode(void)
+{
+ return DEV_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 98b7d7b442..d833e7878e 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define DEV_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define DEV_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,50 @@ struct vfio_device_info;
#define DEV_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum dev_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - DEV_VFIO_MODE_NONE: VFIO is not enabled.
+ * - DEV_VFIO_MODE_GROUP: Legacy group mode.
+ * - DEV_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum dev_vfio_mode {
+ DEV_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ DEV_VFIO_MODE_GROUP, /**< Group mode */
+ DEV_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `dev_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ * - ENOMEM - Memory allocation failed for device tracking.
*/
__rte_internal
int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,40 +90,54 @@ int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int dev_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Enable VFIO subsystem and check if specified kernel module is loaded.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `dev_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int dev_vfio_enable(const char *modname);
@@ -117,59 +153,73 @@ void dev_vfio_cleanup(void);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int dev_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int dev_vfio_noiommu_is_enabled(void);
+enum dev_vfio_mode
+dev_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+dev_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-dev_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -188,7 +238,12 @@ dev_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -196,14 +251,17 @@ dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -211,7 +269,9 @@ dev_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -222,8 +282,14 @@ dev_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -231,14 +297,22 @@ dev_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -263,33 +337,47 @@ dev_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ * - ENOMEM - Memory allocation failed for device tracking.
*/
__rte_internal
int
-dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+dev_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use DEV_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -300,22 +388,26 @@ dev_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use DEV_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 345558f43a..ab808ed0a9 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,85 +24,43 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - ENOMEM: memory allocation failed
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
+ VFIO_NO_MEM,
};
-struct vfio_config {
- bool enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct vfio_container vfio_containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_global_cfg = {
+ .mode = DEV_VFIO_MODE_NONE,
+ .default_cfg = &vfio_containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-/* whether VFIO is enabled (usable) in this process */
-static bool vfio_enabled;
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
-is_null_map(const struct user_mem_map *map)
+is_null_map(const struct vfio_user_mem_map *map)
{
return map->addr == 0 && map->iova == 0 &&
map->len == 0 && map->chunk == 0;
@@ -114,8 +72,8 @@ is_null_map(const struct user_mem_map *map)
static int
user_mem_map_cmp(const void *a, const void *b)
{
- const struct user_mem_map *umm_a = a;
- const struct user_mem_map *umm_b = b;
+ const struct vfio_user_mem_map *umm_a = a;
+ const struct vfio_user_mem_map *umm_b = b;
/* move null entries to end */
if (is_null_map(umm_a))
@@ -152,14 +110,14 @@ user_mem_map_cmp(const void *a, const void *b)
* mappings that will be kept.
*/
static int
-process_maps(struct user_mem_map *src, size_t src_len,
- struct user_mem_map newmap[2], uint64_t vaddr, uint64_t len)
+process_maps(struct vfio_user_mem_map *src, size_t src_len,
+ struct vfio_user_mem_map newmap[2], uint64_t vaddr, uint64_t len)
{
- struct user_mem_map *src_first = &src[0];
- struct user_mem_map *src_last = &src[src_len - 1];
- struct user_mem_map *dst_first = &newmap[0];
+ struct vfio_user_mem_map *src_first = &src[0];
+ struct vfio_user_mem_map *src_last = &src[src_len - 1];
+ struct vfio_user_mem_map *dst_first = &newmap[0];
/* we can get at most two new segments */
- struct user_mem_map *dst_last = &newmap[1];
+ struct vfio_user_mem_map *dst_last = &newmap[1];
uint64_t first_off = vaddr - src_first->addr;
uint64_t last_off = (src_last->addr + src_last->len) - (vaddr + len);
int newmap_len = 0;
@@ -174,7 +132,7 @@ process_maps(struct user_mem_map *src, size_t src_len,
}
if (last_off != 0) {
/* if we had start offset, we have two segments */
- struct user_mem_map *last =
+ struct vfio_user_mem_map *last =
first_off == 0 ? dst_first : dst_last;
last->addr = (src_last->addr + src_last->len) - last_off;
last->iova = (src_last->iova + src_last->len) - last_off;
@@ -188,50 +146,50 @@ process_maps(struct user_mem_map *src, size_t src_len,
/* erase certain maps from the list */
static void
-delete_maps(struct user_mem_maps *user_mem_maps, struct user_mem_map *del_maps,
+delete_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, struct vfio_user_mem_map *del_maps,
size_t n_del)
{
unsigned int i;
size_t j;
- for (i = 0, j = 0; i < RTE_DIM(user_mem_maps->maps) && j < n_del; i++) {
- struct user_mem_map *left = &user_mem_maps->maps[i];
- struct user_mem_map *right = &del_maps[j];
+ for (i = 0, j = 0; i < RTE_DIM(vfio_user_mem_maps->maps) && j < n_del; i++) {
+ struct vfio_user_mem_map *left = &vfio_user_mem_maps->maps[i];
+ struct vfio_user_mem_map *right = &del_maps[j];
if (user_mem_map_cmp(left, right) == 0) {
memset(left, 0, sizeof(*left));
j++;
- user_mem_maps->n_maps--;
+ vfio_user_mem_maps->n_maps--;
}
}
}
static void
-copy_maps(struct user_mem_maps *user_mem_maps, struct user_mem_map *add_maps,
+copy_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, struct vfio_user_mem_map *add_maps,
size_t n_add)
{
unsigned int i;
size_t j;
- for (i = 0, j = 0; i < RTE_DIM(user_mem_maps->maps) && j < n_add; i++) {
- struct user_mem_map *left = &user_mem_maps->maps[i];
- struct user_mem_map *right = &add_maps[j];
+ for (i = 0, j = 0; i < RTE_DIM(vfio_user_mem_maps->maps) && j < n_add; i++) {
+ struct vfio_user_mem_map *left = &vfio_user_mem_maps->maps[i];
+ struct vfio_user_mem_map *right = &add_maps[j];
/* insert into empty space */
if (is_null_map(left)) {
memcpy(left, right, sizeof(*left));
j++;
- user_mem_maps->n_maps++;
+ vfio_user_mem_maps->n_maps++;
}
}
}
/* try merging two maps into one, return 1 if succeeded */
static int
-merge_map(struct user_mem_map *left, struct user_mem_map *right)
+merge_map(struct vfio_user_mem_map *left, struct vfio_user_mem_map *right)
{
/* merge the same maps into one */
- if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
+ if (memcmp(left, right, sizeof(struct vfio_user_mem_map)) == 0)
goto out;
if (left->addr + left->len != right->addr)
@@ -249,13 +207,13 @@ merge_map(struct user_mem_map *left, struct user_mem_map *right)
}
static bool
-addr_is_chunk_aligned(struct user_mem_map *maps, size_t n_maps,
+addr_is_chunk_aligned(struct vfio_user_mem_map *maps, size_t n_maps,
uint64_t vaddr, uint64_t iova)
{
unsigned int i;
for (i = 0; i < n_maps; i++) {
- struct user_mem_map *map = &maps[i];
+ struct vfio_user_mem_map *map = &maps[i];
uint64_t map_va_end = map->addr + map->len;
uint64_t map_iova_end = map->iova + map->len;
uint64_t map_va_off = vaddr - map->addr;
@@ -276,8 +234,8 @@ addr_is_chunk_aligned(struct user_mem_map *maps, size_t n_maps,
}
static int
-find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
- uint64_t iova, uint64_t len, struct user_mem_map *dst,
+find_user_mem_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, uint64_t addr,
+ uint64_t iova, uint64_t len, struct vfio_user_mem_map *dst,
size_t dst_len)
{
uint64_t va_end = addr + len;
@@ -286,8 +244,8 @@ find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
size_t j;
int i, ret;
- for (i = 0, j = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map = &user_mem_maps->maps[i];
+ for (i = 0, j = 0; i < vfio_user_mem_maps->n_maps; i++) {
+ struct vfio_user_mem_map *map = &vfio_user_mem_maps->maps[i];
uint64_t map_va_end = map->addr + map->len;
uint64_t map_iova_end = map->iova + map->len;
@@ -327,308 +285,158 @@ find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
/* this will sort all user maps, and merge/compact any adjacent maps */
static void
-compact_user_maps(struct user_mem_maps *user_mem_maps)
+compact_user_maps(struct vfio_user_mem_maps *vfio_user_mem_maps)
{
unsigned int i;
- qsort(user_mem_maps->maps, RTE_DIM(user_mem_maps->maps),
- sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
+ qsort(vfio_user_mem_maps->maps, RTE_DIM(vfio_user_mem_maps->maps),
+ sizeof(vfio_user_mem_maps->maps[0]), user_mem_map_cmp);
/* we'll go over the list backwards when merging */
- for (i = RTE_DIM(user_mem_maps->maps) - 2; i != 0; i--) {
- struct user_mem_map *l, *r;
+ for (i = RTE_DIM(vfio_user_mem_maps->maps) - 2; i != 0; i--) {
+ struct vfio_user_mem_map *l, *r;
- l = &user_mem_maps->maps[i];
- r = &user_mem_maps->maps[i + 1];
+ l = &vfio_user_mem_maps->maps[i];
+ r = &vfio_user_mem_maps->maps[i + 1];
if (is_null_map(l) || is_null_map(r))
continue;
/* try and merge the maps */
if (merge_map(l, r))
- user_mem_maps->n_maps--;
+ vfio_user_mem_maps->n_maps--;
}
/* the entries are still sorted, but now they have holes in them, so
* sort the list again.
*/
- qsort(user_mem_maps->maps, RTE_DIM(user_mem_maps->maps),
- sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
+ qsort(vfio_user_mem_maps->maps, RTE_DIM(vfio_user_mem_maps->maps),
+ sizeof(vfio_user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct vfio_container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_global_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct vfio_container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct vfio_container *cfg;
if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_global_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct vfio_container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
+ struct vfio_container *cfg;
+ struct vfio_group *grp;
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
+ }
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct vfio_container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct vfio_container *cfg;
+
+ /* find an unused container config */
+ VFIO_CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = VFIO_CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct vfio_container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = VFIO_CONTAINER_INITIALIZER;
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct vfio_container *cfg, enum vfio_device_type type)
{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
+ struct vfio_device *dev;
+
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
+
+ VFIO_DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ dev->type = type;
+ /* set to invalid fd */
+ dev->fd = -1;
+
+ cfg->n_devices++;
+ return dev;
}
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
}
-static int
-vfio_group_device_count(int vfio_group_fd)
+static void
+vfio_device_erase(struct vfio_container *cfg, struct vfio_device *dev)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ if (dev->type == VFIO_DEVICE_TYPE_GROUP) {
+ free(dev->sysfs_base);
+ free(dev->dev_addr);
}
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
+static struct vfio_device *
+vfio_device_get_by_addr(struct vfio_container *cfg,
+ const char *sysfs_base,
+ const char *dev_addr)
+{
+ struct vfio_device *dev;
+
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ /* only group mode have sysfs_base and dev_addr */
+ if (dev->type != VFIO_DEVICE_TYPE_GROUP)
+ continue;
+ if (strcmp(dev->sysfs_base, sysfs_base) == 0 &&
+ strcmp(dev->dev_addr, dev_addr) == 0)
+ return dev;
}
-
- return vfio_cfg->vfio_groups[i].devices;
+ return NULL;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -643,11 +451,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -665,11 +471,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -677,448 +481,565 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in dev_vfio_enable() */
- if (!vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct vfio_container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct vfio_user_mem_maps *vfio_user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_global_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
return -1;
}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < vfio_user_mem_maps->n_maps; i++) {
+ struct vfio_user_mem_map *map;
+ map = &vfio_user_mem_maps->maps[i];
+
+ ret = vfio_global_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
return 0;
}
+static enum vfio_result
+vfio_group_assign_device(struct vfio_container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ char *sysfs_dup, *dev_dup;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* if this device is already assigned, reuse it instead of opening a second fd */
+ dev = vfio_device_get_by_addr(cfg, sysfs_base, dev_addr);
+ if (dev != NULL) {
+ *out_dev = dev;
+ return VFIO_EXISTS;
+ }
+
+ sysfs_dup = strdup(sysfs_base);
+ dev_dup = strdup(dev_addr);
+ if (sysfs_dup == NULL || dev_dup == NULL) {
+ EAL_LOG(ERR, "Cannot allocate memory for device %s", dev_addr);
+ free(sysfs_dup);
+ free(dev_dup);
+ return VFIO_NO_MEM;
+ }
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg, VFIO_DEVICE_TYPE_GROUP);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* store sysfs path and device address */
+ dev->sysfs_base = sysfs_dup;
+ dev->dev_addr = dev_dup;
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we do not need to look in other configs: if we were to attempt to use
+ * a different container, the kernel wouldn't have allowed us to bind the
+ * group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct vfio_container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ return -1;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ return -1;
+ case VFIO_NO_MEM:
+ EAL_LOG(ERR, "Memory allocation failed for device %s", dev_addr);
+ rte_errno = ENOMEM;
+ return -1;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- /* get group number */
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
-int
-dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_global_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ case VFIO_NO_MEM:
+ EAL_LOG(ERR, "Memory allocation failed for device %s", dev_addr);
+ rte_errno = ENOMEM;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
+int
+dev_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct vfio_container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ VFIO_CONTAINER_FOREACH_ACTIVE(icfg) {
+ VFIO_DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct vfio_container *cfg, enum dev_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request container from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = VFIO_SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum dev_vfio_mode
+vfio_select_mode(void)
+{
+ struct vfio_container *cfg;
+ enum dev_vfio_mode mode = DEV_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_global_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return DEV_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ cfg->dma_setup_done = true;
+ return (enum dev_vfio_mode)mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return DEV_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return DEV_VFIO_MODE_NOIOMMU;
+ return DEV_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return DEV_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum dev_vfio_mode mode)
+{
+ switch (mode) {
+ case DEV_VFIO_MODE_GROUP: return "group";
+ case DEV_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
int
dev_vfio_enable(const char *modname)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum dev_vfio_mode mode = DEV_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].enabled = false;
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
+ if (modname == NULL) {
+ rte_errno = EINVAL;
+ return -1;
}
+ /* if already enabled, nothing to do */
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_NONE)
+ return 0;
+
EAL_LOG(DEBUG, "Probing VFIO support...");
/* check if vfio module is loaded */
@@ -1136,37 +1057,16 @@ dev_vfio_enable(const char *modname)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", modname);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(DEV_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->enabled = true;
- vfio_enabled = true;
- } else {
+ /* have we initialized anything? */
+ if (mode == DEV_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_global_cfg.mode = mode;
return 0;
}
@@ -1175,40 +1075,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
dev_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_global_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (default_vfio_cfg->vfio_iommu_type == NULL)
+ if (vfio_global_cfg.ops == NULL)
return -1;
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_global_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
@@ -1217,126 +1094,27 @@ dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
dev_vfio_get_container_fd(void)
@@ -1345,511 +1123,54 @@ dev_vfio_get_container_fd(void)
* The default container is set up during dev_vfio_enable().
* This function does not create a new container.
*/
- if (!vfio_enabled)
- return -1;
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_NONE)
+ return vfio_global_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-dev_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
}
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP && vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
return -1;
}
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
return -1;
}
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_global_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1857,33 +1178,30 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map *new_map;
- struct user_mem_maps *user_mem_maps;
+ struct vfio_user_mem_map *new_map;
+ struct vfio_user_mem_maps *vfio_user_mem_maps;
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
- if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
+ vfio_user_mem_maps = &cfg->mem_maps;
+ rte_spinlock_recursive_lock(&vfio_user_mem_maps->lock);
+ if (vfio_user_mem_maps->n_maps == RTE_DIM(vfio_user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1896,35 +1214,35 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_global_cfg.ops->partial_unmap;
/* create new user mem map entry */
- new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
+ new_map = &vfio_user_mem_maps->maps[vfio_user_mem_maps->n_maps++];
new_map->addr = vaddr;
new_map->iova = iova;
new_map->len = len;
/* for IOMMU types supporting partial unmap, we don't need chunking */
new_map->chunk = has_partial_unmap ? 0 : len;
- compact_user_maps(user_mem_maps);
+ compact_user_maps(vfio_user_mem_maps);
out:
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
+ rte_spinlock_recursive_unlock(&vfio_user_mem_maps->lock);
return ret;
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
- struct user_mem_map new_maps[2]; /* can be at most 2 */
- struct user_mem_maps *user_mem_maps;
+ struct vfio_user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
+ struct vfio_user_mem_map new_maps[2]; /* can be at most 2 */
+ struct vfio_user_mem_maps *vfio_user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
+ vfio_user_mem_maps = &cfg->mem_maps;
+ rte_spinlock_recursive_lock(&vfio_user_mem_maps->lock);
/*
* Previously, we had adjacent mappings entirely contained within one
@@ -1944,18 +1262,17 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
* the start and the end of our requested unmap. We need to collect all
* maps that include our unmapped region.
*/
- n_orig = find_user_mem_maps(user_mem_maps, vaddr, iova, len,
+ n_orig = find_user_mem_maps(vfio_user_mem_maps, vaddr, iova, len,
orig_maps, RTE_DIM(orig_maps));
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_global_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -1971,7 +1288,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -1986,39 +1302,31 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
n_new = process_maps(orig_maps, n_orig, new_maps, vaddr, len);
/* can we store the new maps in our list? */
- newlen = (user_mem_maps->n_maps - n_orig) + n_new;
- if (newlen >= RTE_DIM(user_mem_maps->maps)) {
+ newlen = (vfio_user_mem_maps->n_maps - n_orig) + n_new;
+ if (newlen >= RTE_DIM(vfio_user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
- delete_maps(user_mem_maps, orig_maps, n_orig);
- copy_maps(user_mem_maps, new_maps, n_new);
- compact_user_maps(user_mem_maps);
+ delete_maps(vfio_user_mem_maps, orig_maps, n_orig);
+ copy_maps(vfio_user_mem_maps, new_maps, n_new);
+ compact_user_maps(vfio_user_mem_maps);
out:
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
+ rte_spinlock_recursive_unlock(&vfio_user_mem_maps->lock);
return ret;
}
@@ -2026,281 +1334,206 @@ RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
int
dev_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
{
- unsigned int i;
+ struct vfio_container *cfg;
+ int container_fd;
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- vfio_cfgs[i].enabled = true;
- return vfio_cfgs[i].vfio_container_fd;
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
+ }
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
dev_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->enabled = false;
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
-int
-dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
+ rte_errno = ENODEV;
return -1;
}
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ rte_errno = EINVAL;
return -1;
}
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct vfio_container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
+ struct vfio_container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
-static void
-vfio_cleanup_config(struct vfio_config *vfio_cfg)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_mode)
+enum dev_vfio_mode
+dev_vfio_get_mode(void)
{
- unsigned int i;
-
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- struct vfio_group *group = &vfio_cfg->vfio_groups[i];
-
- if (group->group_num == -1)
- continue;
- if (group->devices != 0) {
- EAL_LOG(ERR, "Cannot cleanup VFIO group %d with %d devices",
- group->group_num, group->devices);
- return;
- }
- if (group->fd >= 0 && close(group->fd) < 0) {
- EAL_LOG(ERR, "Cannot close VFIO group %d: %s",
- group->group_num, strerror(errno));
- return;
- }
-
- group->group_num = -1;
- group->fd = -1;
- group->devices = 0;
- vfio_cfg->vfio_active_groups--;
- }
-
- /* if there are still active groups, we cannot cleanup the container */
- if (vfio_cfg->vfio_active_groups != 0) {
- EAL_LOG(ERR, "Cannot cleanup VFIO container with %d active groups",
- vfio_cfg->vfio_active_groups);
- return;
- }
-
- if (vfio_cfg->vfio_container_fd >= 0 &&
- close(vfio_cfg->vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Cannot close VFIO container: %s", strerror(errno));
- return;
- }
-
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->enabled = false;
- vfio_cfg->vfio_iommu_type = NULL;
-
- vfio_cfg->mem_maps.n_maps = 0;
- memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
+ return vfio_global_cfg.mode;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
dev_vfio_cleanup(void)
{
- unsigned int i;
+ struct vfio_container *cfg;
vfio_mp_sync_cleanup();
- /* mem events can only be unregistered from the primary process */
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_device *dev;
+ struct vfio_group *grp;
- /* cleanup all enabled configs, ignore failures */
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].enabled)
- vfio_cleanup_config(&vfio_cfgs[i]);
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev)
+ vfio_device_erase(cfg, dev);
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp)
+ vfio_group_erase(cfg, grp);
+
+ /* callback is only registered for the default container in the primary */
+ if (group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+
+ vfio_container_erase(cfg);
}
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..418f24f603 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,34 +6,95 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <dev_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct vfio_user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct vfio_user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct vfio_user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+enum vfio_device_type {
+ VFIO_DEVICE_TYPE_NONE = 0,
+ VFIO_DEVICE_TYPE_GROUP,
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ enum vfio_device_type type;
+ int fd;
+ int group; /**< back-reference to group list */
+ char *sysfs_base; /**< sysfs path prefix */
+ char *dev_addr; /**< device address */
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct vfio_container {
+ bool active;
+ bool dma_setup_done;
+ int container_fd;
+ struct vfio_user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*vfio_dma_func_t)(struct vfio_container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*vfio_dma_user_func_t)(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
@@ -41,33 +102,82 @@ struct vfio_iommu_type {
vfio_dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct vfio_container *default_cfg;
+ enum dev_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+/* per-process, per-container data */
+extern struct vfio_container vfio_containers[RTE_MAX_VFIO_CONTAINERS];
-int
-vfio_get_iommu_type(void);
+/* current configuration */
+extern struct vfio_config vfio_global_cfg;
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define VFIO_CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &vfio_containers[0]; \
+ (cfg) < &vfio_containers[RTE_DIM(vfio_containers)]; \
+ (cfg)++)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+#define VFIO_CONTAINER_FOREACH_ACTIVE(cfg) \
+ VFIO_CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+#define VFIO_GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
+
+#define VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) \
+ VFIO_GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
+
+#define VFIO_DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
+
+#define VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ VFIO_DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
+
+/* for containers, we need to initialize the fd, and mem maps lock */
+#define VFIO_CONTAINER_INITIALIZER \
+ ((struct vfio_container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ .container_fd = -1, \
+ })
+
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct vfio_container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct vfio_container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct vfio_container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct vfio_container *cfg, int iommu_group);
+void vfio_group_erase(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct vfio_container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
-#define SOCKET_REQ_CONTAINER 0x100
-#define SOCKET_REQ_GROUP 0x200
-#define SOCKET_REQ_IOMMU_TYPE 0x400
-#define SOCKET_OK 0x0
-#define SOCKET_NO_FD 0x1
-#define SOCKET_ERR 0xFF
+#define VFIO_SOCKET_REQ_CONTAINER 0x100
+#define VFIO_SOCKET_REQ_GROUP 0x200
+#define VFIO_SOCKET_REQ_IOMMU_TYPE 0x400
+#define VFIO_SOCKET_OK 0x0
+#define VFIO_SOCKET_NO_FD 0x1
+#define VFIO_SOCKET_ERR 0xFF
struct vfio_mp_param {
int req;
@@ -75,6 +185,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum dev_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..c06cd50731
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,983 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <dev_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct vfio_container *);
+static int vfio_type1_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct vfio_container *);
+static int vfio_spapr_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct vfio_container *);
+static int vfio_noiommu_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct vfio_container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+static uint64_t spapr_dma_win_len;
+static uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct vfio_container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct vfio_container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct vfio_container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct vfio_container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct vfio_container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ VFIO_GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct vfio_container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = VFIO_SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == VFIO_SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = VFIO_SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct vfio_container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(DEV_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct vfio_container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_global_cfg.ops == NULL) {
+ vfio_global_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_global_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_global_cfg.ops->type_id, vfio_global_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 66341a701a..e1f82ad77b 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -31,46 +31,65 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
memset(&reply, 0, sizeof(reply));
switch (m->req) {
- case SOCKET_REQ_GROUP:
- r->req = SOCKET_REQ_GROUP;
+ case VFIO_SOCKET_REQ_GROUP:
+ {
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP &&
+ vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
- r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
- r->result = SOCKET_OK;
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
+ r->result = VFIO_SOCKET_NO_FD;
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
+ r->result = VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
- case SOCKET_REQ_CONTAINER:
- r->req = SOCKET_REQ_CONTAINER;
+ }
+ case VFIO_SOCKET_REQ_CONTAINER:
+ r->req = VFIO_SOCKET_REQ_CONTAINER;
fd = dev_vfio_get_container_fd();
if (fd < 0)
- r->result = SOCKET_ERR;
+ r->result = VFIO_SOCKET_ERR;
else {
- r->result = SOCKET_OK;
+ r->result = VFIO_SOCKET_OK;
+ r->mode = vfio_global_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
- case SOCKET_REQ_IOMMU_TYPE:
+ case VFIO_SOCKET_REQ_IOMMU_TYPE:
{
int iommu_type_id;
- r->req = SOCKET_REQ_IOMMU_TYPE;
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP &&
+ vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
if (iommu_type_id < 0)
- r->result = SOCKET_ERR;
+ r->result = VFIO_SOCKET_ERR;
else {
r->iommu_type_id = iommu_type_id;
- r->result = SOCKET_OK;
+ r->result = VFIO_SOCKET_OK;
}
break;
}
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 19/24] bus/pci: use the new VFIO mode API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (17 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 18/24] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 20/24] bus/fslmc: " Anatoly Burakov
` (4 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status. This requires moving
the VFIO init earlier in the init sequence, as now checking no-IOMMU mode
is not a sysfs check but instead relies on full VFIO subsystem to be
initialized and available.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
lib/eal/linux/eal.c | 15 ++++++++++-----
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index af37a65c45..b9fe4797fb 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (dev_vfio_noiommu_is_enabled() == 1)
+ if (dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 77691a7908..11e6217826 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -676,6 +676,16 @@ rte_eal_init(int argc, char **argv)
}
}
+ /*
+ * VFIO must be initialized before bus scan because buses need to know
+ * about no-IOMMU mode status.
+ */
+ if (dev_vfio_enable("vfio")) {
+ rte_eal_init_alert("Cannot init VFIO");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
if (rte_bus_scan()) {
rte_eal_init_alert("Cannot scan the buses for devices");
rte_errno = ENODEV;
@@ -771,11 +781,6 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (dev_vfio_enable("vfio")) {
- rte_eal_init_alert("Cannot init VFIO");
- rte_errno = EAGAIN;
- goto err_out;
- }
/* in secondary processes, memory init may allocate additional fbarrays
* not present in primary processes, so to avoid any potential issues,
* initialize memzones first.
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 20/24] bus/fslmc: use the new VFIO mode API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (18 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 19/24] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 21/24] net/hinic3: " Anatoly Burakov
` (3 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 11 ++++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index d55c414ead..8bfbdac52e 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,14 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (dev_vfio_get_mode() != DEV_VFIO_MODE_GROUP &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU) {
+ DPAA2_BUS_DEBUG("DPAA2: VFIO not in group mode");
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +508,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && dev_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index c2b6ac1f3f..c416ef9f68 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -204,7 +204,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (dev_vfio_noiommu_is_enabled() > 0)
+ if (dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 21/24] net/hinic3: use the new VFIO mode API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (19 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 20/24] bus/fslmc: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 22/24] net/ntnic: " Anatoly Burakov
` (2 subsequent siblings)
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 6a6cf44752..3486160ba1 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && dev_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 22/24] net/ntnic: use the new VFIO mode API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (20 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 21/24] net/hinic3: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 23/24] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 24/24] vfio: introduce cdev mode Anatoly Burakov
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index d9744a7d05..45629a567d 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_noiommu_is_enabled();
+ const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_dev_vfio_no_io_mmu_enabled);
if (n_dev_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 23/24] vfio: remove no-IOMMU check API
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (21 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 22/24] net/ntnic: " Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 24/24] vfio: introduce cdev mode Anatoly Burakov
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson
The `dev_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ------
lib/eal/include/dev_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 27 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 55e1ee63b2..b3bf97b4ef 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -856,12 +856,6 @@ int dev_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
-int dev_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index d833e7878e..a3e4b63743 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -180,20 +180,6 @@ __rte_internal
enum dev_vfio_mode
dev_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-dev_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index ab808ed0a9..86498a3571 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1330,13 +1330,6 @@ container_dma_unmap(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
-int
-dev_vfio_noiommu_is_enabled(void)
-{
- return vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v13 24/24] vfio: introduce cdev mode
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
` (22 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 23/24] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-27 15:14 ` Anatoly Burakov
23 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-27 15:14 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
The implementation largely follows the kernel reference code, with most
differences from group mode concentrated in how the container and the
devices are set up. The VFIO container allocates an "IOAS" (IO address
space), maps the memory into that space, and attaches devices to IOAS -
there is no IOMMU type discovery at device attach time.
Unlike group mode, the cdev mode is device-centric, and all DMA mappings
are made for an IOAS and not into VFIO container itself. As a result, while
group mode only maps memory during first device attach, cdev mode can map
memory without having any devices attached. However, current VFIO init has
moved to before bus scan, because we need VFIO mode information for buses
to decide on IOVA mode. DMA memory mapping, however, must be performed
after the DPDK memory was already initialized, so we add another internal
API to trigger VFIO DMA maps for cdev mode right after memory init.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/linux_gsg/linux_drivers.rst | 30 ++
doc/guides/rel_notes/release_26_11.rst | 5 +
lib/eal/freebsd/eal.c | 16 +-
lib/eal/include/dev_vfio.h | 45 +++
lib/eal/linux/eal.c | 7 +
lib/eal/linux/eal_vfio.c | 259 +++++++++++++++-
lib/eal/linux/eal_vfio.h | 39 ++-
lib/eal/linux/eal_vfio_cdev.c | 396 +++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 +++
lib/eal/linux/meson.build | 1 +
10 files changed, 830 insertions(+), 10 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 12e9faf4a2..7dd38a69f8 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -133,6 +133,31 @@ For proper operation of VFIO when running DPDK applications as a non-privileged
For more information, please refer to :ref:`linux_gsg_running_without_root_privileges`.
+.. _linux_gsg_vfio_modes:
+
+VFIO group and cdev modes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Linux kernel provides two different interfaces for VFIO:
+
+group mode
+ The original interface, based on IOMMU groups.
+ It is accessed through the container device ``/dev/vfio/vfio`` and the per-group devices under ``/dev/vfio/``.
+
+cdev mode
+ The newer interface, based on iommufd.
+ It is accessed through ``/dev/iommu`` and the per-device character devices under ``/dev/vfio/devices/``.
+
+DPDK selects the mode automatically; there is no EAL option to force a particular mode.
+Currently, group mode is preferred and is used whenever the container device ``/dev/vfio/vfio`` is usable.
+Using cdev mode requires ``/dev/iommu`` to be present and accessible.
+
+.. note::
+
+ Using VF tokens in cdev mode requires Linux kernel version 6.17 or later.
+ On older kernels the device is bound without the token, so the kernel should be configured to use group mode if VF token support is required.
+
+
.. _linux_gsg_vfio_noiommu:
VFIO no-IOMMU mode
@@ -265,6 +290,11 @@ The token will be used for all PF and VF ports within the application.
Linux versions earlier than version 5.7 do not support the creation of
virtual functions within the VFIO framework.
+.. note::
+
+ When VFIO uses cdev mode, VF tokens require a newer kernel.
+ See :ref:`linux_gsg_vfio_modes` for details.
+
Troubleshooting VFIO
~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index d5899b8d66..72ea8060c4 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added VFIO cdev (IOMMUFD) mode.**
+
+ Added support for the VFIO character device (cdev) API, also known as IOMMUFD.
+ DPDK now supports both the group mode and the new cdev mode.
+
Removed Items
-------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index b3bf97b4ef..9494574526 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -844,10 +844,10 @@ int dev_vfio_enable(__rte_unused const char *modname)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
-void
-dev_vfio_cleanup(void)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_init_mem)
+int dev_vfio_init_mem(void)
{
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
@@ -937,3 +937,13 @@ dev_vfio_get_mode(void)
{
return DEV_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_num)
+int
+dev_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index a3e4b63743..efdc0de491 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define DEV_VFIO_DIR "/dev/vfio"
#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define DEV_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define DEV_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define DEV_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - DEV_VFIO_MODE_NONE: VFIO is not enabled.
* - DEV_VFIO_MODE_GROUP: Legacy group mode.
* - DEV_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - DEV_VFIO_MODE_CDEV: Character device mode.
*/
enum dev_vfio_mode {
DEV_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
DEV_VFIO_MODE_GROUP, /**< Group mode */
DEV_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ DEV_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -151,6 +155,20 @@ int dev_vfio_enable(const char *modname);
__rte_internal
void dev_vfio_cleanup(void);
+/**
+ * @internal
+ * Initialize VFIO memory mapping. Must be called after `dev_vfio_enable()` and
+ * after EAL memory initialization has completed.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int dev_vfio_init_mem(void);
+
/**
* @internal
* Check if VFIO subsystem is initialized and a specified kernel module is loaded.
@@ -207,6 +225,33 @@ __rte_internal
int
dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+dev_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 11e6217826..4dd82e5935 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -818,6 +818,13 @@ rte_eal_init(int argc, char **argv)
goto err_out;
}
+ /* VFIO memory setup that requires DPDK memory to be available. */
+ if (dev_vfio_init_mem() < 0) {
+ rte_eal_init_alert("Cannot init VFIO memory");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
/* register multi-process action callbacks for hotplug after memory init */
if (eal_mp_dev_hotplug_init() < 0) {
rte_eal_init_alert("failed to register mp callback for hotplug");
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 86498a3571..796e291c97 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -349,6 +349,24 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct vfio_container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ /* only cdev mode has dev_num */
+ if (dev->type != VFIO_DEVICE_TYPE_CDEV)
+ continue;
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ }
+ return NULL;
+}
+
static struct vfio_container *
vfio_container_create(void)
{
@@ -541,6 +559,55 @@ vfio_setup_dma_mem(struct vfio_container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct vfio_container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(DEBUG, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg, VFIO_DEVICE_TYPE_CDEV);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct vfio_container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -704,6 +771,49 @@ dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when dev_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when dev_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * There is one downside for group mode when adding duplicate devices: to get to device fd,
+ * we need to go through the entire codepath before we arrive at fd only to realize it was
+ * already opened earlier, but this is acceptable compromise for unifying the API around
+ * device assignment.
+ */
+
if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -724,6 +834,9 @@ dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case DEV_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case DEV_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -800,6 +913,24 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_global_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -924,6 +1055,12 @@ dev_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -988,9 +1125,25 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- cfg->dma_setup_done = true;
- return (enum dev_vfio_mode)mode;
+ if (mode == DEV_VFIO_MODE_CDEV) {
+ /* set up ops and sync IOAS */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else if (mode == DEV_VFIO_MODE_GROUP ||
+ mode == DEV_VFIO_MODE_NOIOMMU) {
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
+
+ return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
if (vfio_mp_sync_setup() < 0)
@@ -1006,10 +1159,17 @@ vfio_select_mode(void)
return DEV_VFIO_MODE_NOIOMMU;
return DEV_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* cdev ops, IOAS and DMA setup are deferred to dev_vfio_init_mem() */
+ if (vfio_cdev_enable(cfg) == 0)
+ return DEV_VFIO_MODE_CDEV;
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
vfio_container_erase(cfg);
+ /* reset ops as well */
+ vfio_global_cfg.ops = NULL;
return DEV_VFIO_MODE_NONE;
}
@@ -1020,6 +1180,7 @@ vfio_mode_to_str(enum dev_vfio_mode mode)
switch (mode) {
case DEV_VFIO_MODE_GROUP: return "group";
case DEV_VFIO_MODE_NOIOMMU: return "noiommu";
+ case DEV_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1071,6 +1232,36 @@ dev_vfio_enable(const char *modname)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_init_mem)
+int
+dev_vfio_init_mem(void)
+{
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
+
+ /* secondary already synced everything during dev_vfio_enable() */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_CDEV:
+ /* iommufd IOAS is populated once DPDK memory is available */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ return -1;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ return -1;
+ if (vfio_register_mem_event_callback() < 0)
+ return -1;
+ cfg->dma_setup_done = true;
+ break;
+ default:
+ /* group/noiommu map memory at device attach, or no VFIO */
+ break;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
dev_vfio_is_enabled(const char *modname)
@@ -1163,6 +1354,40 @@ dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_num)
+int
+dev_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1362,6 +1587,27 @@ dev_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ /* we only need to set up IOAS for DMA to work */
+ cfg->dma_setup_done = true;
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1420,6 +1666,13 @@ dev_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case DEV_VFIO_MODE_CDEV:
+ /* erase all devices */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 418f24f603..363c8fcf89 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,6 +48,7 @@ struct vfio_group {
enum vfio_device_type {
VFIO_DEVICE_TYPE_NONE = 0,
VFIO_DEVICE_TYPE_GROUP,
+ VFIO_DEVICE_TYPE_CDEV,
};
/* device tracking (common for group and cdev modes) */
@@ -55,9 +56,16 @@ struct vfio_device {
bool active;
enum vfio_device_type type;
int fd;
- int group; /**< back-reference to group list */
- char *sysfs_base; /**< sysfs path prefix */
- char *dev_addr; /**< device address */
+ union {
+ struct {
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX */
+ };
+ struct {
+ int group; /**< back-reference to group list */
+ char *sysfs_base; /**< sysfs path prefix */
+ char *dev_addr; /**< device address */
+ };
+ };
};
/* group mode specific configuration */
@@ -68,13 +76,21 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct vfio_container {
bool active;
bool dma_setup_done;
int container_fd;
struct vfio_user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -169,12 +185,25 @@ int vfio_group_setup_iommu(struct vfio_container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct vfio_container *cfg);
+void vfio_cdev_setup_ops(void);
+int vfio_cdev_setup_ioas(struct vfio_container *cfg);
+int vfio_cdev_sync_ioas(struct vfio_container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct vfio_container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct vfio_container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
#define VFIO_SOCKET_REQ_CONTAINER 0x100
#define VFIO_SOCKET_REQ_GROUP 0x200
#define VFIO_SOCKET_REQ_IOMMU_TYPE 0x400
+#define VFIO_SOCKET_REQ_CDEV 0x800
+#define VFIO_SOCKET_REQ_IOAS_ID 0x1000
#define VFIO_SOCKET_OK 0x0
#define VFIO_SOCKET_NO_FD 0x1
#define VFIO_SOCKET_ERR 0xFF
@@ -185,6 +214,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum dev_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..360d3b9306
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,396 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct vfio_container *cfg);
+static int vfio_cdev_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+void
+vfio_cdev_setup_ops(void)
+{
+ vfio_global_cfg.ops = &iommufd_ops;
+}
+
+static int
+vfio_cdev_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct vfio_container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct vfio_container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = VFIO_SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct vfio_container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(DEV_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ DEV_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct vfio_container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(DEV_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct vfio_container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ DEV_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct vfio_container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = VFIO_SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct vfio_container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0) {
+ close(device_fd);
+ dev->fd = -1;
+ return -1;
+ }
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index e1f82ad77b..8e989d17ab 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case VFIO_SOCKET_REQ_CDEV:
+ {
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_global_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = VFIO_SOCKET_NO_FD;
+ } else {
+ r->result = VFIO_SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case VFIO_SOCKET_REQ_IOAS_ID:
+ {
+ struct vfio_container *cfg;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_IOAS_ID;
+ cfg = vfio_global_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = VFIO_SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 00/25] Support VFIO cdev API in DPDK
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
` (7 preceding siblings ...)
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 01/25] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
` (25 more replies)
8 siblings, 26 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev
This patchset depends on the following:
* https://patches.dpdk.org/project/dpdk/list/?series=39072 ("net/nbl: fix use-after-free")
This patchset introduces a major refactor of the VFIO subsystem in DPDK to
support character device (cdev) interface introduced in Linux kernel, as well as
make the API more streamlined and useful. The goal is to simplify device
management, improve compatibility, make the code readable, and clarify API.
The following sections outline the key issues addressed by this patchset and the
corresponding changes introduced.
1. Only group mode is supported
===============================
Since kernel version 4.14.327 (LTS), VFIO supports the new character device
(cdev)-based way of working with VFIO devices (otherwise known as IOMMUFD). This
is a device-centric mode and does away with all the complexity regarding groups
and IOMMU types, delegating it all to the kernel, and exposes a much simpler
interface to userspace. The old group-based implementation will still be around,
and will need to be kept in DPDK for compatibility reasons.
To enable this, VFIO is heavily refactored, so that the code can support both
modes while relying on (mostly) common infrastructure.
Additionally, a new `rte_vfio_get_mode` API is added for those cases that need
some introspection into VFIO's internals, with three new modes: group
(old-style), no-iommu (old-style but without IOMMU), and cdev (the new mode).
Although no-IOMMU is technically a variant of group mode, the distinction is
largely irrelevant to the user, as all usages of noiommu checks in our codebase
are for deciding whether to use IOVA or PA, not anything to do with managing
groups. The current plan for kernel community is to *not* introduce no-IOMMU
cdev implementation, and IOMMUFD's own group API compatibility layer also does
not implement no-IOMMU mode, which is why a no-IOMMU mode looks like a distinct
VFIO mode to the outside observer, even though internally it uses group mode.
2. Custom container assignment API does not map to cdev mode
============================================================
The existing `rte_vfio_device_setup/release` model is fundamentally incompatible
with cdev mode, because for custom container cases, the expected flow is that
the user binds the IOMMU group (and thus, implicitly, the device itself) to a
specific container using `rte_vfio_container_group_bind`, whereas this step is
not needed for cdev as the device fd is assigned to the container straight away.
Therefore, what we do instead is introduce a new API for container device
assignment which, semantically, will assign a device to specified container, so
that when it is mapped using `rte_pci_map_device`, the appropriate container is
selected. Under the hood though, we essentially transition to getting device fd
straight away at assign stage, so that by the time the PCI bus attempts to map
the device, it is already mapped and we just return an fd. There is no
"unassign" API because `release_device` already performs that function.
Because the API is now unified around device assignment, the old group-specific
API's can be removed and, where appropriate, reimplemented using new API. There
were other users of VFIO which relied on group API but only for convenience
purposes; no actual VFIO functionality depended on those API's.
List of removed API's:
* `rte_vfio_get_group_fd`
* `rte_vfio_clear_group`
* `rte_vfio_container_group_bind` (replaced by container assign API)
* `rte_vfio_container_group_unbind`
* `rte_vfio_noiommu_is_enabled` (replaced by new mode API)
3. The API responsibilities aren't clear and bleed into each other
==================================================================
Some API's do multiple things at once. In particular:
* `rte_vfio_get_device_info` will setup the device
* `rte_vfio_setup_device` will get device info
These API's have been adjusted to do one thing only.
4. The API does not need to be public
=====================================
The initial idea for exposing VFIO API was to enable userspace applications to
directly map memory for DMA, but it turns out that in practice only drivers use
this API. Therefore, the entire VFIO API is made internal, driver-only, and is
renamed from `rte_vfio` to `dev_vfio`.
v14:
- Hardened the transitional patches against attempting to use VFIO without
having done initialization
- Fixed a bunch of typos, moved a few hunks around to where they belong, and
added a more release note entries
v13:
- Addressed feedback from Stephen's AI review:
- Added strdup-based deduplication for group mode to avoid leaking fd's when
container assignment API is used
- Added more documentation about cdev mode to Linux GSG
- Fixed comments and log levels in a couple of places
- Renamed the API from `rte_vfio_*` to `dev_vfio_*` on account of it no longer
being a publicly visible API
- Moved `dev_vfio.h` into driver SDK headers
- Decoupled VFIO internals from FSLMC and EAL
- Added proper teardown for VFIO subsystem on cleanup
v12:
- Addressed feedback from Stephen's AI review:
- Add release notes updates to patch 2 and 20
- Fix ENXIO typos
- Added a new init step after memory init to enable DMA mapping for cdev
v11:
- Addressed feedback from Stephen's AI review:
- Use CONTAINER_INITIALIZER for reset
- Set container fd to -1 in CONTAINER_INITIALIZER
- Fixed double close()
- Moved VFIO init to earlier in init sequence to account for
bus drivers needing no-IOMMU mode status
- Fixed missing ops set for cdev mode, and missing ops reset
- Fixed fd leak on failed attach in cdev
v10:
- Added a patch that renames confusing error labels
- Fixed compiler warning about unused variable
v9:
- Moved erroneous rte_errno-related comments to later in the patchset
- Moved removal of vDPA group fd API's to their respective patches
- Fixed typo in errno comments (ENXIO vs ENOXIO)
- Fixed corruption of group config in secondary process (v8 AI review)
v8:
- Rebase
- Fixed build errors due to variable shadowing
- Removed duplicate fd check as kernel does not provide a way to distinguish
between device fd's
v7:
- Rebase
- Added removal of deprecation notices
- Fixed implicit numeric comparison in patch 12
v6:
- Fixed missing header include in vfio cdev file
v5:
- Added back missing uapi patch
v4:
- Fixed issues with documenting rte_vfio_mode enum
- Separated deprecation notices into a separate patchset
v3:
- Make API removal cleaner
- Fix `get_group_num` usages to align with new API
- Fix issues with function exports
- Fix issues with `setup_device` returning old-style values in some cases
v2:
- Make the entire API internal
- More aggressive API pruning, complete removal of group API
- Fixed a bug in group mode where device could not be used
- Better documentation and deprecation notice patches
- Moved doc patches to beginning of patchset
Anatoly Burakov (25):
uapi: update to v6.17 and add iommufd.h
vfio: make all functions internal
bus/fslmc: decouple from EAL VFIO
vfio: decouple EAL from VFIO internals
vfio: do proper teardown on VFIO cleanup
vfio: remove modname parameter from init
vfio: rename API to reflect internal status
bus/pci: rename mismatching error labels
vfio: split get device info from setup
vfio: add container device assignment API
net/nbl: do not use VFIO group bind API
net/ntnic: use container device assignment API
vdpa/ifc: use container device assignment API
vdpa/nfp: use container device assignment API
vdpa/sfc: use container device assignment API
vdpa/mlx5: remove group-related API
vhost: remove group-related API from driver
vfio: remove group-based API
vfio: cleanup and refactor
bus/pci: use the new VFIO mode API
bus/fslmc: use the new VFIO mode API
net/hinic3: use the new VFIO mode API
net/ntnic: use the new VFIO mode API
vfio: remove no-IOMMU check API
vfio: introduce cdev mode
config/arm/meson.build | 1 +
config/meson.build | 1 +
doc/api/doxy-api-index.md | 3 +-
doc/guides/linux_gsg/linux_drivers.rst | 30 +
doc/guides/prog_guide/vhost_lib.rst | 4 -
doc/guides/rel_notes/deprecation.rst | 10 -
doc/guides/rel_notes/release_26_11.rst | 16 +
drivers/bus/cdx/cdx.c | 6 +-
drivers/bus/cdx/cdx_vfio.c | 37 +-
drivers/bus/fslmc/fslmc_bus.c | 13 +-
drivers/bus/fslmc/fslmc_vfio.c | 66 +-
drivers/bus/fslmc/meson.build | 1 -
drivers/bus/pci/linux/pci.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 79 +-
drivers/bus/pci/pci_common.c | 10 +-
drivers/bus/platform/platform.c | 17 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 20 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 5 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 34 +-
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
drivers/net/ntnic/ntnic_ethdev.c | 10 +-
drivers/net/ntnic/ntnic_vfio.c | 48 +-
drivers/vdpa/ifc/ifcvf_vdpa.c | 54 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
drivers/vdpa/nfp/nfp_vdpa.c | 53 +-
drivers/vdpa/sfc/sfc_vdpa.c | 47 +-
drivers/vdpa/sfc/sfc_vdpa.h | 2 -
drivers/vdpa/sfc/sfc_vdpa_hw.c | 12 +-
kernel/linux/uapi/linux/iommufd.h | 1292 ++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
lib/eal/freebsd/eal.c | 100 +-
lib/eal/include/dev_vfio.h | 449 ++++
lib/eal/include/meson.build | 2 +-
lib/eal/include/rte_vfio.h | 338 ---
lib/eal/linux/eal.c | 27 +-
lib/eal/linux/eal_vfio.c | 2635 +++++++++------------
lib/eal/linux/eal_vfio.h | 191 +-
lib/eal/linux/eal_vfio_cdev.c | 396 ++++
lib/eal/linux/eal_vfio_group.c | 983 ++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 108 +-
lib/eal/linux/meson.build | 2 +
lib/eal/windows/eal.c | 16 +-
lib/vhost/vdpa_driver.h | 3 -
lib/vhost/vhost_user.c | 16 +-
46 files changed, 4885 insertions(+), 2274 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
create mode 100644 lib/eal/include/dev_vfio.h
delete mode 100644 lib/eal/include/rte_vfio.h
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
create mode 100644 lib/eal/linux/eal_vfio_group.c
--
2.52.0
^ permalink raw reply [flat|nested] 203+ messages in thread
* [PATCH v14 01/25] uapi: update to v6.17 and add iommufd.h
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 02/25] vfio: make all functions internal Anatoly Burakov
` (24 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Maxime Coquelin
In order to support VF tokens for cdev-based VFIO mode, kernel v6.17 is
required. Update internal headers to version v6.17 and include iommufd.h.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Stephen Hemminger <stephen@networkplumber.org>
---
Depends-on: series-39072 ("net/nbl: fix use-after-free")
kernel/linux/uapi/linux/iommufd.h | 1292 +++++++++++++++++++++++++++++
kernel/linux/uapi/linux/vduse.h | 2 +-
kernel/linux/uapi/linux/vfio.h | 12 +-
kernel/linux/uapi/version | 2 +-
4 files changed, 1305 insertions(+), 3 deletions(-)
create mode 100644 kernel/linux/uapi/linux/iommufd.h
diff --git a/kernel/linux/uapi/linux/iommufd.h b/kernel/linux/uapi/linux/iommufd.h
new file mode 100644
index 0000000000..2105a03955
--- /dev/null
+++ b/kernel/linux/uapi/linux/iommufd.h
@@ -0,0 +1,1292 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/* Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES.
+ */
+#ifndef _IOMMUFD_H
+#define _IOMMUFD_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+#define IOMMUFD_TYPE (';')
+
+/**
+ * DOC: General ioctl format
+ *
+ * The ioctl interface follows a general format to allow for extensibility. Each
+ * ioctl is passed in a structure pointer as the argument providing the size of
+ * the structure in the first u32. The kernel checks that any structure space
+ * beyond what it understands is 0. This allows userspace to use the backward
+ * compatible portion while consistently using the newer, larger, structures.
+ *
+ * ioctls use a standard meaning for common errnos:
+ *
+ * - ENOTTY: The IOCTL number itself is not supported at all
+ * - E2BIG: The IOCTL number is supported, but the provided structure has
+ * non-zero in a part the kernel does not understand.
+ * - EOPNOTSUPP: The IOCTL number is supported, and the structure is
+ * understood, however a known field has a value the kernel does not
+ * understand or support.
+ * - EINVAL: Everything about the IOCTL was understood, but a field is not
+ * correct.
+ * - ENOENT: An ID or IOVA provided does not exist.
+ * - ENOMEM: Out of memory.
+ * - EOVERFLOW: Mathematics overflowed.
+ *
+ * As well as additional errnos, within specific ioctls.
+ */
+enum {
+ IOMMUFD_CMD_BASE = 0x80,
+ IOMMUFD_CMD_DESTROY = IOMMUFD_CMD_BASE,
+ IOMMUFD_CMD_IOAS_ALLOC = 0x81,
+ IOMMUFD_CMD_IOAS_ALLOW_IOVAS = 0x82,
+ IOMMUFD_CMD_IOAS_COPY = 0x83,
+ IOMMUFD_CMD_IOAS_IOVA_RANGES = 0x84,
+ IOMMUFD_CMD_IOAS_MAP = 0x85,
+ IOMMUFD_CMD_IOAS_UNMAP = 0x86,
+ IOMMUFD_CMD_OPTION = 0x87,
+ IOMMUFD_CMD_VFIO_IOAS = 0x88,
+ IOMMUFD_CMD_HWPT_ALLOC = 0x89,
+ IOMMUFD_CMD_GET_HW_INFO = 0x8a,
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING = 0x8b,
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP = 0x8c,
+ IOMMUFD_CMD_HWPT_INVALIDATE = 0x8d,
+ IOMMUFD_CMD_FAULT_QUEUE_ALLOC = 0x8e,
+ IOMMUFD_CMD_IOAS_MAP_FILE = 0x8f,
+ IOMMUFD_CMD_VIOMMU_ALLOC = 0x90,
+ IOMMUFD_CMD_VDEVICE_ALLOC = 0x91,
+ IOMMUFD_CMD_IOAS_CHANGE_PROCESS = 0x92,
+ IOMMUFD_CMD_VEVENTQ_ALLOC = 0x93,
+ IOMMUFD_CMD_HW_QUEUE_ALLOC = 0x94,
+};
+
+/**
+ * struct iommu_destroy - ioctl(IOMMU_DESTROY)
+ * @size: sizeof(struct iommu_destroy)
+ * @id: iommufd object ID to destroy. Can be any destroyable object type.
+ *
+ * Destroy any object held within iommufd.
+ */
+struct iommu_destroy {
+ __u32 size;
+ __u32 id;
+};
+#define IOMMU_DESTROY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_DESTROY)
+
+/**
+ * struct iommu_ioas_alloc - ioctl(IOMMU_IOAS_ALLOC)
+ * @size: sizeof(struct iommu_ioas_alloc)
+ * @flags: Must be 0
+ * @out_ioas_id: Output IOAS ID for the allocated object
+ *
+ * Allocate an IO Address Space (IOAS) which holds an IO Virtual Address (IOVA)
+ * to memory mapping.
+ */
+struct iommu_ioas_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_ioas_id;
+};
+#define IOMMU_IOAS_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOC)
+
+/**
+ * struct iommu_iova_range - ioctl(IOMMU_IOVA_RANGE)
+ * @start: First IOVA
+ * @last: Inclusive last IOVA
+ *
+ * An interval in IOVA space.
+ */
+struct iommu_iova_range {
+ __aligned_u64 start;
+ __aligned_u64 last;
+};
+
+/**
+ * struct iommu_ioas_iova_ranges - ioctl(IOMMU_IOAS_IOVA_RANGES)
+ * @size: sizeof(struct iommu_ioas_iova_ranges)
+ * @ioas_id: IOAS ID to read ranges from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to the output array of struct iommu_iova_range
+ * @out_iova_alignment: Minimum alignment required for mapping IOVA
+ *
+ * Query an IOAS for ranges of allowed IOVAs. Mapping IOVA outside these ranges
+ * is not allowed. num_iovas will be set to the total number of iovas and
+ * the allowed_iovas[] will be filled in as space permits.
+ *
+ * The allowed ranges are dependent on the HW path the DMA operation takes, and
+ * can change during the lifetime of the IOAS. A fresh empty IOAS will have a
+ * full range, and each attached device will narrow the ranges based on that
+ * device's HW restrictions. Detaching a device can widen the ranges. Userspace
+ * should query ranges after every attach/detach to know what IOVAs are valid
+ * for mapping.
+ *
+ * On input num_iovas is the length of the allowed_iovas array. On output it is
+ * the total number of iovas filled in. The ioctl will return -EMSGSIZE and set
+ * num_iovas to the required value if num_iovas is too small. In this case the
+ * caller should allocate a larger output array and re-issue the ioctl.
+ *
+ * out_iova_alignment returns the minimum IOVA alignment that can be given
+ * to IOMMU_IOAS_MAP/COPY. IOVA's must satisfy::
+ *
+ * starting_iova % out_iova_alignment == 0
+ * (starting_iova + length) % out_iova_alignment == 0
+ *
+ * out_iova_alignment can be 1 indicating any IOVA is allowed. It cannot
+ * be higher than the system PAGE_SIZE.
+ */
+struct iommu_ioas_iova_ranges {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+ __aligned_u64 out_iova_alignment;
+};
+#define IOMMU_IOAS_IOVA_RANGES _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_IOVA_RANGES)
+
+/**
+ * struct iommu_ioas_allow_iovas - ioctl(IOMMU_IOAS_ALLOW_IOVAS)
+ * @size: sizeof(struct iommu_ioas_allow_iovas)
+ * @ioas_id: IOAS ID to allow IOVAs from
+ * @num_iovas: Input/Output total number of ranges in the IOAS
+ * @__reserved: Must be 0
+ * @allowed_iovas: Pointer to array of struct iommu_iova_range
+ *
+ * Ensure a range of IOVAs are always available for allocation. If this call
+ * succeeds then IOMMU_IOAS_IOVA_RANGES will never return a list of IOVA ranges
+ * that are narrower than the ranges provided here. This call will fail if
+ * IOMMU_IOAS_IOVA_RANGES is currently narrower than the given ranges.
+ *
+ * When an IOAS is first created the IOVA_RANGES will be maximally sized, and as
+ * devices are attached the IOVA will narrow based on the device restrictions.
+ * When an allowed range is specified any narrowing will be refused, ie device
+ * attachment can fail if the device requires limiting within the allowed range.
+ *
+ * Automatic IOVA allocation is also impacted by this call. MAP will only
+ * allocate within the allowed IOVAs if they are present.
+ *
+ * This call replaces the entire allowed list with the given list.
+ */
+struct iommu_ioas_allow_iovas {
+ __u32 size;
+ __u32 ioas_id;
+ __u32 num_iovas;
+ __u32 __reserved;
+ __aligned_u64 allowed_iovas;
+};
+#define IOMMU_IOAS_ALLOW_IOVAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_ALLOW_IOVAS)
+
+/**
+ * enum iommufd_ioas_map_flags - Flags for map and copy
+ * @IOMMU_IOAS_MAP_FIXED_IOVA: If clear the kernel will compute an appropriate
+ * IOVA to place the mapping at
+ * @IOMMU_IOAS_MAP_WRITEABLE: DMA is allowed to write to this mapping
+ * @IOMMU_IOAS_MAP_READABLE: DMA is allowed to read from this mapping
+ */
+enum iommufd_ioas_map_flags {
+ IOMMU_IOAS_MAP_FIXED_IOVA = 1 << 0,
+ IOMMU_IOAS_MAP_WRITEABLE = 1 << 1,
+ IOMMU_IOAS_MAP_READABLE = 1 << 2,
+};
+
+/**
+ * struct iommu_ioas_map - ioctl(IOMMU_IOAS_MAP)
+ * @size: sizeof(struct iommu_ioas_map)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @ioas_id: IOAS ID to change the mapping of
+ * @__reserved: Must be 0
+ * @user_va: Userspace pointer to start mapping from
+ * @length: Number of bytes to map
+ * @iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is set
+ * then this must be provided as input.
+ *
+ * Set an IOVA mapping from a user pointer. If FIXED_IOVA is specified then the
+ * mapping will be established at iova, otherwise a suitable location based on
+ * the reserved and allowed lists will be automatically selected and returned in
+ * iova.
+ *
+ * If IOMMU_IOAS_MAP_FIXED_IOVA is specified then the iova range must currently
+ * be unused, existing IOVA cannot be replaced.
+ */
+struct iommu_ioas_map {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __u32 __reserved;
+ __aligned_u64 user_va;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP)
+
+/**
+ * struct iommu_ioas_map_file - ioctl(IOMMU_IOAS_MAP_FILE)
+ * @size: sizeof(struct iommu_ioas_map_file)
+ * @flags: same as for iommu_ioas_map
+ * @ioas_id: same as for iommu_ioas_map
+ * @fd: the memfd to map
+ * @start: byte offset from start of file to map from
+ * @length: same as for iommu_ioas_map
+ * @iova: same as for iommu_ioas_map
+ *
+ * Set an IOVA mapping from a memfd file. All other arguments and semantics
+ * match those of IOMMU_IOAS_MAP.
+ */
+struct iommu_ioas_map_file {
+ __u32 size;
+ __u32 flags;
+ __u32 ioas_id;
+ __s32 fd;
+ __aligned_u64 start;
+ __aligned_u64 length;
+ __aligned_u64 iova;
+};
+#define IOMMU_IOAS_MAP_FILE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_MAP_FILE)
+
+/**
+ * struct iommu_ioas_copy - ioctl(IOMMU_IOAS_COPY)
+ * @size: sizeof(struct iommu_ioas_copy)
+ * @flags: Combination of enum iommufd_ioas_map_flags
+ * @dst_ioas_id: IOAS ID to change the mapping of
+ * @src_ioas_id: IOAS ID to copy from
+ * @length: Number of bytes to copy and map
+ * @dst_iova: IOVA the mapping was placed at. If IOMMU_IOAS_MAP_FIXED_IOVA is
+ * set then this must be provided as input.
+ * @src_iova: IOVA to start the copy
+ *
+ * Copy an already existing mapping from src_ioas_id and establish it in
+ * dst_ioas_id. The src iova/length must exactly match a range used with
+ * IOMMU_IOAS_MAP.
+ *
+ * This may be used to efficiently clone a subset of an IOAS to another, or as a
+ * kind of 'cache' to speed up mapping. Copy has an efficiency advantage over
+ * establishing equivalent new mappings, as internal resources are shared, and
+ * the kernel will pin the user memory only once.
+ */
+struct iommu_ioas_copy {
+ __u32 size;
+ __u32 flags;
+ __u32 dst_ioas_id;
+ __u32 src_ioas_id;
+ __aligned_u64 length;
+ __aligned_u64 dst_iova;
+ __aligned_u64 src_iova;
+};
+#define IOMMU_IOAS_COPY _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_COPY)
+
+/**
+ * struct iommu_ioas_unmap - ioctl(IOMMU_IOAS_UNMAP)
+ * @size: sizeof(struct iommu_ioas_unmap)
+ * @ioas_id: IOAS ID to change the mapping of
+ * @iova: IOVA to start the unmapping at
+ * @length: Number of bytes to unmap, and return back the bytes unmapped
+ *
+ * Unmap an IOVA range. The iova/length must be a superset of a previously
+ * mapped range used with IOMMU_IOAS_MAP or IOMMU_IOAS_COPY. Splitting or
+ * truncating ranges is not allowed. The values 0 to U64_MAX will unmap
+ * everything.
+ */
+struct iommu_ioas_unmap {
+ __u32 size;
+ __u32 ioas_id;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+};
+#define IOMMU_IOAS_UNMAP _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_UNMAP)
+
+/**
+ * enum iommufd_option - ioctl(IOMMU_OPTION_RLIMIT_MODE) and
+ * ioctl(IOMMU_OPTION_HUGE_PAGES)
+ * @IOMMU_OPTION_RLIMIT_MODE:
+ * Change how RLIMIT_MEMLOCK accounting works. The caller must have privilege
+ * to invoke this. Value 0 (default) is user based accounting, 1 uses process
+ * based accounting. Global option, object_id must be 0
+ * @IOMMU_OPTION_HUGE_PAGES:
+ * Value 1 (default) allows contiguous pages to be combined when generating
+ * iommu mappings. Value 0 disables combining, everything is mapped to
+ * PAGE_SIZE. This can be useful for benchmarking. This is a per-IOAS
+ * option, the object_id must be the IOAS ID.
+ */
+enum iommufd_option {
+ IOMMU_OPTION_RLIMIT_MODE = 0,
+ IOMMU_OPTION_HUGE_PAGES = 1,
+};
+
+/**
+ * enum iommufd_option_ops - ioctl(IOMMU_OPTION_OP_SET) and
+ * ioctl(IOMMU_OPTION_OP_GET)
+ * @IOMMU_OPTION_OP_SET: Set the option's value
+ * @IOMMU_OPTION_OP_GET: Get the option's value
+ */
+enum iommufd_option_ops {
+ IOMMU_OPTION_OP_SET = 0,
+ IOMMU_OPTION_OP_GET = 1,
+};
+
+/**
+ * struct iommu_option - iommu option multiplexer
+ * @size: sizeof(struct iommu_option)
+ * @option_id: One of enum iommufd_option
+ * @op: One of enum iommufd_option_ops
+ * @__reserved: Must be 0
+ * @object_id: ID of the object if required
+ * @val64: Option value to set or value returned on get
+ *
+ * Change a simple option value. This multiplexor allows controlling options
+ * on objects. IOMMU_OPTION_OP_SET will load an option and IOMMU_OPTION_OP_GET
+ * will return the current value.
+ */
+struct iommu_option {
+ __u32 size;
+ __u32 option_id;
+ __u16 op;
+ __u16 __reserved;
+ __u32 object_id;
+ __aligned_u64 val64;
+};
+#define IOMMU_OPTION _IO(IOMMUFD_TYPE, IOMMUFD_CMD_OPTION)
+
+/**
+ * enum iommufd_vfio_ioas_op - IOMMU_VFIO_IOAS_* ioctls
+ * @IOMMU_VFIO_IOAS_GET: Get the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_SET: Change the current compatibility IOAS
+ * @IOMMU_VFIO_IOAS_CLEAR: Disable VFIO compatibility
+ */
+enum iommufd_vfio_ioas_op {
+ IOMMU_VFIO_IOAS_GET = 0,
+ IOMMU_VFIO_IOAS_SET = 1,
+ IOMMU_VFIO_IOAS_CLEAR = 2,
+};
+
+/**
+ * struct iommu_vfio_ioas - ioctl(IOMMU_VFIO_IOAS)
+ * @size: sizeof(struct iommu_vfio_ioas)
+ * @ioas_id: For IOMMU_VFIO_IOAS_SET the input IOAS ID to set
+ * For IOMMU_VFIO_IOAS_GET will output the IOAS ID
+ * @op: One of enum iommufd_vfio_ioas_op
+ * @__reserved: Must be 0
+ *
+ * The VFIO compatibility support uses a single ioas because VFIO APIs do not
+ * support the ID field. Set or Get the IOAS that VFIO compatibility will use.
+ * When VFIO_GROUP_SET_CONTAINER is used on an iommufd it will get the
+ * compatibility ioas, either by taking what is already set, or auto creating
+ * one. From then on VFIO will continue to use that ioas and is not effected by
+ * this ioctl. SET or CLEAR does not destroy any auto-created IOAS.
+ */
+struct iommu_vfio_ioas {
+ __u32 size;
+ __u32 ioas_id;
+ __u16 op;
+ __u16 __reserved;
+};
+#define IOMMU_VFIO_IOAS _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VFIO_IOAS)
+
+/**
+ * enum iommufd_hwpt_alloc_flags - Flags for HWPT allocation
+ * @IOMMU_HWPT_ALLOC_NEST_PARENT: If set, allocate a HWPT that can serve as
+ * the parent HWPT in a nesting configuration.
+ * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOMMU is
+ * enforced on device attachment
+ * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data is
+ * valid.
+ * @IOMMU_HWPT_ALLOC_PASID: Requests a domain that can be used with PASID. The
+ * domain can be attached to any PASID on the device.
+ * Any domain attached to the non-PASID part of the
+ * device must also be flagged, otherwise attaching a
+ * PASID will blocked.
+ * For the user that wants to attach PASID, ioas is
+ * not recommended for both the non-PASID part
+ * and PASID part of the device.
+ * If IOMMU does not support PASID it will return
+ * error (-EOPNOTSUPP).
+ */
+enum iommufd_hwpt_alloc_flags {
+ IOMMU_HWPT_ALLOC_NEST_PARENT = 1 << 0,
+ IOMMU_HWPT_ALLOC_DIRTY_TRACKING = 1 << 1,
+ IOMMU_HWPT_FAULT_ID_VALID = 1 << 2,
+ IOMMU_HWPT_ALLOC_PASID = 1 << 3,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_flags - Intel VT-d stage-1 page table
+ * entry attributes
+ * @IOMMU_VTD_S1_SRE: Supervisor request
+ * @IOMMU_VTD_S1_EAFE: Extended access enable
+ * @IOMMU_VTD_S1_WPE: Write protect enable
+ */
+enum iommu_hwpt_vtd_s1_flags {
+ IOMMU_VTD_S1_SRE = 1 << 0,
+ IOMMU_VTD_S1_EAFE = 1 << 1,
+ IOMMU_VTD_S1_WPE = 1 << 2,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1 - Intel VT-d stage-1 page table
+ * info (IOMMU_HWPT_DATA_VTD_S1)
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_flags
+ * @pgtbl_addr: The base address of the stage-1 page table.
+ * @addr_width: The address width of the stage-1 page table
+ * @__reserved: Must be 0
+ */
+struct iommu_hwpt_vtd_s1 {
+ __aligned_u64 flags;
+ __aligned_u64 pgtbl_addr;
+ __u32 addr_width;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_hwpt_arm_smmuv3 - ARM SMMUv3 nested STE
+ * (IOMMU_HWPT_DATA_ARM_SMMUV3)
+ *
+ * @ste: The first two double words of the user space Stream Table Entry for
+ * the translation. Must be little-endian.
+ * Allowed fields: (Refer to "5.2 Stream Table Entry" in SMMUv3 HW Spec)
+ * - word-0: V, Cfg, S1Fmt, S1ContextPtr, S1CDMax
+ * - word-1: EATS, S1DSS, S1CIR, S1COR, S1CSH, S1STALLD
+ *
+ * -EIO will be returned if @ste is not legal or contains any non-allowed field.
+ * Cfg can be used to select a S1, Bypass or Abort configuration. A Bypass
+ * nested domain will translate the same as the nesting parent. The S1 will
+ * install a Context Descriptor Table pointing at userspace memory translated
+ * by the nesting parent.
+ */
+struct iommu_hwpt_arm_smmuv3 {
+ __aligned_le64 ste[2];
+};
+
+/**
+ * enum iommu_hwpt_data_type - IOMMU HWPT Data Type
+ * @IOMMU_HWPT_DATA_NONE: no data
+ * @IOMMU_HWPT_DATA_VTD_S1: Intel VT-d stage-1 page table
+ * @IOMMU_HWPT_DATA_ARM_SMMUV3: ARM SMMUv3 Context Descriptor Table
+ */
+enum iommu_hwpt_data_type {
+ IOMMU_HWPT_DATA_NONE = 0,
+ IOMMU_HWPT_DATA_VTD_S1 = 1,
+ IOMMU_HWPT_DATA_ARM_SMMUV3 = 2,
+};
+
+/**
+ * struct iommu_hwpt_alloc - ioctl(IOMMU_HWPT_ALLOC)
+ * @size: sizeof(struct iommu_hwpt_alloc)
+ * @flags: Combination of enum iommufd_hwpt_alloc_flags
+ * @dev_id: The device to allocate this HWPT for
+ * @pt_id: The IOAS or HWPT or vIOMMU to connect this HWPT to
+ * @out_hwpt_id: The ID of the new HWPT
+ * @__reserved: Must be 0
+ * @data_type: One of enum iommu_hwpt_data_type
+ * @data_len: Length of the type specific data
+ * @data_uptr: User pointer to the type specific data
+ * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of
+ * IOMMU_HWPT_FAULT_ID_VALID is set.
+ * @__reserved2: Padding to 64-bit alignment. Must be 0.
+ *
+ * Explicitly allocate a hardware page table object. This is the same object
+ * type that is returned by iommufd_device_attach() and represents the
+ * underlying iommu driver's iommu_domain kernel object.
+ *
+ * A kernel-managed HWPT will be created with the mappings from the given
+ * IOAS via the @pt_id. The @data_type for this allocation must be set to
+ * IOMMU_HWPT_DATA_NONE. The HWPT can be allocated as a parent HWPT for a
+ * nesting configuration by passing IOMMU_HWPT_ALLOC_NEST_PARENT via @flags.
+ *
+ * A user-managed nested HWPT will be created from a given vIOMMU (wrapping a
+ * parent HWPT) or a parent HWPT via @pt_id, in which the parent HWPT must be
+ * allocated previously via the same ioctl from a given IOAS (@pt_id). In this
+ * case, the @data_type must be set to a pre-defined type corresponding to an
+ * I/O page table type supported by the underlying IOMMU hardware. The device
+ * via @dev_id and the vIOMMU via @pt_id must be associated to the same IOMMU
+ * instance.
+ *
+ * If the @data_type is set to IOMMU_HWPT_DATA_NONE, @data_len and
+ * @data_uptr should be zero. Otherwise, both @data_len and @data_uptr
+ * must be given.
+ */
+struct iommu_hwpt_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pt_id;
+ __u32 out_hwpt_id;
+ __u32 __reserved;
+ __u32 data_type;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ __u32 fault_id;
+ __u32 __reserved2;
+};
+#define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC)
+
+/**
+ * enum iommu_hw_info_vtd_flags - Flags for VT-d hw_info
+ * @IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17: If set, disallow read-only mappings
+ * on a nested_parent domain.
+ * https://www.intel.com/content/www/us/en/content-details/772415/content-details.html
+ */
+enum iommu_hw_info_vtd_flags {
+ IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17 = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info_vtd - Intel VT-d hardware information
+ *
+ * @flags: Combination of enum iommu_hw_info_vtd_flags
+ * @__reserved: Must be 0
+ *
+ * @cap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.2 Capability Register.
+ * @ecap_reg: Value of Intel VT-d capability register defined in VT-d spec
+ * section 11.4.3 Extended Capability Register.
+ *
+ * User needs to understand the Intel VT-d specification to decode the
+ * register value.
+ */
+struct iommu_hw_info_vtd {
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 cap_reg;
+ __aligned_u64 ecap_reg;
+};
+
+/**
+ * struct iommu_hw_info_arm_smmuv3 - ARM SMMUv3 hardware information
+ * (IOMMU_HW_INFO_TYPE_ARM_SMMUV3)
+ *
+ * @flags: Must be set to 0
+ * @__reserved: Must be 0
+ * @idr: Implemented features for ARM SMMU Non-secure programming interface
+ * @iidr: Information about the implementation and implementer of ARM SMMU,
+ * and architecture version supported
+ * @aidr: ARM SMMU architecture version
+ *
+ * For the details of @idr, @iidr and @aidr, please refer to the chapters
+ * from 6.3.1 to 6.3.6 in the SMMUv3 Spec.
+ *
+ * This reports the raw HW capability, and not all bits are meaningful to be
+ * read by userspace. Only the following fields should be used:
+ *
+ * idr[0]: ST_LEVEL, TERM_MODEL, STALL_MODEL, TTENDIAN , CD2L, ASID16, TTF
+ * idr[1]: SIDSIZE, SSIDSIZE
+ * idr[3]: BBML, RIL
+ * idr[5]: VAX, GRAN64K, GRAN16K, GRAN4K
+ *
+ * - S1P should be assumed to be true if a NESTED HWPT can be created
+ * - VFIO/iommufd only support platforms with COHACC, it should be assumed to be
+ * true.
+ * - ATS is a per-device property. If the VMM describes any devices as ATS
+ * capable in ACPI/DT it should set the corresponding idr.
+ *
+ * This list may expand in future (eg E0PD, AIE, PBHA, D128, DS etc). It is
+ * important that VMMs do not read bits outside the list to allow for
+ * compatibility with future kernels. Several features in the SMMUv3
+ * architecture are not currently supported by the kernel for nesting: HTTU,
+ * BTM, MPAM and others.
+ */
+struct iommu_hw_info_arm_smmuv3 {
+ __u32 flags;
+ __u32 __reserved;
+ __u32 idr[6];
+ __u32 iidr;
+ __u32 aidr;
+};
+
+/**
+ * struct iommu_hw_info_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Hardware
+ * Information (IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV)
+ *
+ * @flags: Must be 0
+ * @version: Version number for the CMDQ-V HW for PARAM bits[03:00]
+ * @log2vcmdqs: Log2 of the total number of VCMDQs for PARAM bits[07:04]
+ * @log2vsids: Log2 of the total number of SID replacements for PARAM bits[15:12]
+ * @__reserved: Must be 0
+ *
+ * VMM can use these fields directly in its emulated global PARAM register. Note
+ * that only one Virtual Interface (VINTF) should be exposed to a VM, i.e. PARAM
+ * bits[11:08] should be set to 0 for log2 of the total number of VINTFs.
+ */
+struct iommu_hw_info_tegra241_cmdqv {
+ __u32 flags;
+ __u8 version;
+ __u8 log2vcmdqs;
+ __u8 log2vsids;
+ __u8 __reserved;
+};
+
+/**
+ * enum iommu_hw_info_type - IOMMU Hardware Info Types
+ * @IOMMU_HW_INFO_TYPE_NONE: Output by the drivers that do not report hardware
+ * info
+ * @IOMMU_HW_INFO_TYPE_DEFAULT: Input to request for a default type
+ * @IOMMU_HW_INFO_TYPE_INTEL_VTD: Intel VT-d iommu info type
+ * @IOMMU_HW_INFO_TYPE_ARM_SMMUV3: ARM SMMUv3 iommu info type
+ * @IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) info type
+ */
+enum iommu_hw_info_type {
+ IOMMU_HW_INFO_TYPE_NONE = 0,
+ IOMMU_HW_INFO_TYPE_DEFAULT = 0,
+ IOMMU_HW_INFO_TYPE_INTEL_VTD = 1,
+ IOMMU_HW_INFO_TYPE_ARM_SMMUV3 = 2,
+ IOMMU_HW_INFO_TYPE_TEGRA241_CMDQV = 3,
+};
+
+/**
+ * enum iommufd_hw_capabilities
+ * @IOMMU_HW_CAP_DIRTY_TRACKING: IOMMU hardware support for dirty tracking
+ * If available, it means the following APIs
+ * are supported:
+ *
+ * IOMMU_HWPT_GET_DIRTY_BITMAP
+ * IOMMU_HWPT_SET_DIRTY_TRACKING
+ *
+ * @IOMMU_HW_CAP_PCI_PASID_EXEC: Execute Permission Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ * @IOMMU_HW_CAP_PCI_PASID_PRIV: Privileged Mode Supported, user ignores it
+ * when the struct
+ * iommu_hw_info::out_max_pasid_log2 is zero.
+ */
+enum iommufd_hw_capabilities {
+ IOMMU_HW_CAP_DIRTY_TRACKING = 1 << 0,
+ IOMMU_HW_CAP_PCI_PASID_EXEC = 1 << 1,
+ IOMMU_HW_CAP_PCI_PASID_PRIV = 1 << 2,
+};
+
+/**
+ * enum iommufd_hw_info_flags - Flags for iommu_hw_info
+ * @IOMMU_HW_INFO_FLAG_INPUT_TYPE: If set, @in_data_type carries an input type
+ * for user space to request for a specific info
+ */
+enum iommufd_hw_info_flags {
+ IOMMU_HW_INFO_FLAG_INPUT_TYPE = 1 << 0,
+};
+
+/**
+ * struct iommu_hw_info - ioctl(IOMMU_GET_HW_INFO)
+ * @size: sizeof(struct iommu_hw_info)
+ * @flags: Must be 0
+ * @dev_id: The device bound to the iommufd
+ * @data_len: Input the length of a user buffer in bytes. Output the length of
+ * data that kernel supports
+ * @data_uptr: User pointer to a user-space buffer used by the kernel to fill
+ * the iommu type specific hardware information data
+ * @in_data_type: This shares the same field with @out_data_type, making it be
+ * a bidirectional field. When IOMMU_HW_INFO_FLAG_INPUT_TYPE is
+ * set, an input type carried via this @in_data_type field will
+ * be valid, requesting for the info data to the given type. If
+ * IOMMU_HW_INFO_FLAG_INPUT_TYPE is unset, any input value will
+ * be seen as IOMMU_HW_INFO_TYPE_DEFAULT
+ * @out_data_type: Output the iommu hardware info type as defined in the enum
+ * iommu_hw_info_type.
+ * @out_capabilities: Output the generic iommu capability info type as defined
+ * in the enum iommu_hw_capabilities.
+ * @out_max_pasid_log2: Output the width of PASIDs. 0 means no PASID support.
+ * PCI devices turn to out_capabilities to check if the
+ * specific capabilities is supported or not.
+ * @__reserved: Must be 0
+ *
+ * Query an iommu type specific hardware information data from an iommu behind
+ * a given device that has been bound to iommufd. This hardware info data will
+ * be used to sync capabilities between the virtual iommu and the physical
+ * iommu, e.g. a nested translation setup needs to check the hardware info, so
+ * a guest stage-1 page table can be compatible with the physical iommu.
+ *
+ * To capture an iommu type specific hardware information data, @data_uptr and
+ * its length @data_len must be provided. Trailing bytes will be zeroed if the
+ * user buffer is larger than the data that kernel has. Otherwise, kernel only
+ * fills the buffer using the given length in @data_len. If the ioctl succeeds,
+ * @data_len will be updated to the length that kernel actually supports,
+ * @out_data_type will be filled to decode the data filled in the buffer
+ * pointed by @data_uptr. Input @data_len == zero is allowed.
+ */
+struct iommu_hw_info {
+ __u32 size;
+ __u32 flags;
+ __u32 dev_id;
+ __u32 data_len;
+ __aligned_u64 data_uptr;
+ union {
+ __u32 in_data_type;
+ __u32 out_data_type;
+ };
+ __u8 out_max_pasid_log2;
+ __u8 __reserved[3];
+ __aligned_u64 out_capabilities;
+};
+#define IOMMU_GET_HW_INFO _IO(IOMMUFD_TYPE, IOMMUFD_CMD_GET_HW_INFO)
+
+/*
+ * enum iommufd_hwpt_set_dirty_tracking_flags - Flags for steering dirty
+ * tracking
+ * @IOMMU_HWPT_DIRTY_TRACKING_ENABLE: Enable dirty tracking
+ */
+enum iommufd_hwpt_set_dirty_tracking_flags {
+ IOMMU_HWPT_DIRTY_TRACKING_ENABLE = 1,
+};
+
+/**
+ * struct iommu_hwpt_set_dirty_tracking - ioctl(IOMMU_HWPT_SET_DIRTY_TRACKING)
+ * @size: sizeof(struct iommu_hwpt_set_dirty_tracking)
+ * @flags: Combination of enum iommufd_hwpt_set_dirty_tracking_flags
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @__reserved: Must be 0
+ *
+ * Toggle dirty tracking on an HW pagetable.
+ */
+struct iommu_hwpt_set_dirty_tracking {
+ __u32 size;
+ __u32 flags;
+ __u32 hwpt_id;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_SET_DIRTY_TRACKING _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING)
+
+/**
+ * enum iommufd_hwpt_get_dirty_bitmap_flags - Flags for getting dirty bits
+ * @IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR: Just read the PTEs without clearing
+ * any dirty bits metadata. This flag
+ * can be passed in the expectation
+ * where the next operation is an unmap
+ * of the same IOVA range.
+ *
+ */
+enum iommufd_hwpt_get_dirty_bitmap_flags {
+ IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR = 1,
+};
+
+/**
+ * struct iommu_hwpt_get_dirty_bitmap - ioctl(IOMMU_HWPT_GET_DIRTY_BITMAP)
+ * @size: sizeof(struct iommu_hwpt_get_dirty_bitmap)
+ * @hwpt_id: HW pagetable ID that represents the IOMMU domain
+ * @flags: Combination of enum iommufd_hwpt_get_dirty_bitmap_flags
+ * @__reserved: Must be 0
+ * @iova: base IOVA of the bitmap first bit
+ * @length: IOVA range size
+ * @page_size: page size granularity of each bit in the bitmap
+ * @data: bitmap where to set the dirty bits. The bitmap bits each
+ * represent a page_size which you deviate from an arbitrary iova.
+ *
+ * Checking a given IOVA is dirty:
+ *
+ * data[(iova / page_size) / 64] & (1ULL << ((iova / page_size) % 64))
+ *
+ * Walk the IOMMU pagetables for a given IOVA range to return a bitmap
+ * with the dirty IOVAs. In doing so it will also by default clear any
+ * dirty bit metadata set in the IOPTE.
+ */
+struct iommu_hwpt_get_dirty_bitmap {
+ __u32 size;
+ __u32 hwpt_id;
+ __u32 flags;
+ __u32 __reserved;
+ __aligned_u64 iova;
+ __aligned_u64 length;
+ __aligned_u64 page_size;
+ __aligned_u64 data;
+};
+#define IOMMU_HWPT_GET_DIRTY_BITMAP _IO(IOMMUFD_TYPE, \
+ IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP)
+
+/**
+ * enum iommu_hwpt_invalidate_data_type - IOMMU HWPT Cache Invalidation
+ * Data Type
+ * @IOMMU_HWPT_INVALIDATE_DATA_VTD_S1: Invalidation data for VTD_S1
+ * @IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3: Invalidation data for ARM SMMUv3
+ */
+enum iommu_hwpt_invalidate_data_type {
+ IOMMU_HWPT_INVALIDATE_DATA_VTD_S1 = 0,
+ IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3 = 1,
+};
+
+/**
+ * enum iommu_hwpt_vtd_s1_invalidate_flags - Flags for Intel VT-d
+ * stage-1 cache invalidation
+ * @IOMMU_VTD_INV_FLAGS_LEAF: Indicates whether the invalidation applies
+ * to all-levels page structure cache or just
+ * the leaf PTE cache.
+ */
+enum iommu_hwpt_vtd_s1_invalidate_flags {
+ IOMMU_VTD_INV_FLAGS_LEAF = 1 << 0,
+};
+
+/**
+ * struct iommu_hwpt_vtd_s1_invalidate - Intel VT-d cache invalidation
+ * (IOMMU_HWPT_INVALIDATE_DATA_VTD_S1)
+ * @addr: The start address of the range to be invalidated. It needs to
+ * be 4KB aligned.
+ * @npages: Number of contiguous 4K pages to be invalidated.
+ * @flags: Combination of enum iommu_hwpt_vtd_s1_invalidate_flags
+ * @__reserved: Must be 0
+ *
+ * The Intel VT-d specific invalidation data for user-managed stage-1 cache
+ * invalidation in nested translation. Userspace uses this structure to
+ * tell the impacted cache scope after modifying the stage-1 page table.
+ *
+ * Invalidating all the caches related to the page table by setting @addr
+ * to be 0 and @npages to be U64_MAX.
+ *
+ * The device TLB will be invalidated automatically if ATS is enabled.
+ */
+struct iommu_hwpt_vtd_s1_invalidate {
+ __aligned_u64 addr;
+ __aligned_u64 npages;
+ __u32 flags;
+ __u32 __reserved;
+};
+
+/**
+ * struct iommu_viommu_arm_smmuv3_invalidate - ARM SMMUv3 cache invalidation
+ * (IOMMU_VIOMMU_INVALIDATE_DATA_ARM_SMMUV3)
+ * @cmd: 128-bit cache invalidation command that runs in SMMU CMDQ.
+ * Must be little-endian.
+ *
+ * Supported command list only when passing in a vIOMMU via @hwpt_id:
+ * CMDQ_OP_TLBI_NSNH_ALL
+ * CMDQ_OP_TLBI_NH_VA
+ * CMDQ_OP_TLBI_NH_VAA
+ * CMDQ_OP_TLBI_NH_ALL
+ * CMDQ_OP_TLBI_NH_ASID
+ * CMDQ_OP_ATC_INV
+ * CMDQ_OP_CFGI_CD
+ * CMDQ_OP_CFGI_CD_ALL
+ *
+ * -EIO will be returned if the command is not supported.
+ */
+struct iommu_viommu_arm_smmuv3_invalidate {
+ __aligned_le64 cmd[2];
+};
+
+/**
+ * struct iommu_hwpt_invalidate - ioctl(IOMMU_HWPT_INVALIDATE)
+ * @size: sizeof(struct iommu_hwpt_invalidate)
+ * @hwpt_id: ID of a nested HWPT or a vIOMMU, for cache invalidation
+ * @data_uptr: User pointer to an array of driver-specific cache invalidation
+ * data.
+ * @data_type: One of enum iommu_hwpt_invalidate_data_type, defining the data
+ * type of all the entries in the invalidation request array. It
+ * should be a type supported by the hwpt pointed by @hwpt_id.
+ * @entry_len: Length (in bytes) of a request entry in the request array
+ * @entry_num: Input the number of cache invalidation requests in the array.
+ * Output the number of requests successfully handled by kernel.
+ * @__reserved: Must be 0.
+ *
+ * Invalidate iommu cache for user-managed page table or vIOMMU. Modifications
+ * on a user-managed page table should be followed by this operation, if a HWPT
+ * is passed in via @hwpt_id. Other caches, such as device cache or descriptor
+ * cache can be flushed if a vIOMMU is passed in via the @hwpt_id field.
+ *
+ * Each ioctl can support one or more cache invalidation requests in the array
+ * that has a total size of @entry_len * @entry_num.
+ *
+ * An empty invalidation request array by setting @entry_num==0 is allowed, and
+ * @entry_len and @data_uptr would be ignored in this case. This can be used to
+ * check if the given @data_type is supported or not by kernel.
+ */
+struct iommu_hwpt_invalidate {
+ __u32 size;
+ __u32 hwpt_id;
+ __aligned_u64 data_uptr;
+ __u32 data_type;
+ __u32 entry_len;
+ __u32 entry_num;
+ __u32 __reserved;
+};
+#define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE)
+
+/**
+ * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is
+ * valid.
+ * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group.
+ */
+enum iommu_hwpt_pgfault_flags {
+ IOMMU_PGFAULT_FLAGS_PASID_VALID = (1 << 0),
+ IOMMU_PGFAULT_FLAGS_LAST_PAGE = (1 << 1),
+};
+
+/**
+ * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault
+ * @IOMMU_PGFAULT_PERM_READ: request for read permission
+ * @IOMMU_PGFAULT_PERM_WRITE: request for write permission
+ * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has the
+ * Execute Requested bit set in PASID TLP Prefix.
+ * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has the
+ * Privileged Mode Requested bit set in PASID TLP
+ * Prefix.
+ */
+enum iommu_hwpt_pgfault_perm {
+ IOMMU_PGFAULT_PERM_READ = (1 << 0),
+ IOMMU_PGFAULT_PERM_WRITE = (1 << 1),
+ IOMMU_PGFAULT_PERM_EXEC = (1 << 2),
+ IOMMU_PGFAULT_PERM_PRIV = (1 << 3),
+};
+
+/**
+ * struct iommu_hwpt_pgfault - iommu page fault data
+ * @flags: Combination of enum iommu_hwpt_pgfault_flags
+ * @dev_id: id of the originated device
+ * @pasid: Process Address Space ID
+ * @grpid: Page Request Group Index
+ * @perm: Combination of enum iommu_hwpt_pgfault_perm
+ * @__reserved: Must be 0.
+ * @addr: Fault address
+ * @length: a hint of how much data the requestor is expecting to fetch. For
+ * example, if the PRI initiator knows it is going to do a 10MB
+ * transfer, it could fill in 10MB and the OS could pre-fault in
+ * 10MB of IOVA. It's default to 0 if there's no such hint.
+ * @cookie: kernel-managed cookie identifying a group of fault messages. The
+ * cookie number encoded in the last page fault of the group should
+ * be echoed back in the response message.
+ */
+struct iommu_hwpt_pgfault {
+ __u32 flags;
+ __u32 dev_id;
+ __u32 pasid;
+ __u32 grpid;
+ __u32 perm;
+ __u32 __reserved;
+ __aligned_u64 addr;
+ __u32 length;
+ __u32 cookie;
+};
+
+/**
+ * enum iommufd_page_response_code - Return status of fault handlers
+ * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables
+ * populated, retry the access. This is the
+ * "Success" defined in PCI 10.4.2.1.
+ * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the
+ * access. This is the "Invalid Request" in PCI
+ * 10.4.2.1.
+ */
+enum iommufd_page_response_code {
+ IOMMUFD_PAGE_RESP_SUCCESS = 0,
+ IOMMUFD_PAGE_RESP_INVALID = 1,
+};
+
+/**
+ * struct iommu_hwpt_page_response - IOMMU page fault response
+ * @cookie: The kernel-managed cookie reported in the fault message.
+ * @code: One of response code in enum iommufd_page_response_code.
+ */
+struct iommu_hwpt_page_response {
+ __u32 cookie;
+ __u32 code;
+};
+
+/**
+ * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_fault_alloc)
+ * @flags: Must be 0
+ * @out_fault_id: The ID of the new FAULT
+ * @out_fault_fd: The fd of the new FAULT
+ *
+ * Explicitly allocate a fault handling object.
+ */
+struct iommu_fault_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 out_fault_id;
+ __u32 out_fault_fd;
+};
+#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_ALLOC)
+
+/**
+ * enum iommu_viommu_type - Virtual IOMMU Type
+ * @IOMMU_VIOMMU_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VIOMMU_TYPE_ARM_SMMUV3: ARM SMMUv3 driver specific type
+ * @IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) enabled ARM SMMUv3 type
+ */
+enum iommu_viommu_type {
+ IOMMU_VIOMMU_TYPE_DEFAULT = 0,
+ IOMMU_VIOMMU_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_viommu_tegra241_cmdqv - NVIDIA Tegra241 CMDQV Virtual Interface
+ * (IOMMU_VIOMMU_TYPE_TEGRA241_CMDQV)
+ * @out_vintf_mmap_offset: mmap offset argument for VINTF's page0
+ * @out_vintf_mmap_length: mmap length argument for VINTF's page0
+ *
+ * Both @out_vintf_mmap_offset and @out_vintf_mmap_length are reported by kernel
+ * for user space to mmap the VINTF page0 from the host physical address space
+ * to the guest physical address space so that a guest kernel can directly R/W
+ * access to the VINTF page0 in order to control its virtual command queues.
+ */
+struct iommu_viommu_tegra241_cmdqv {
+ __aligned_u64 out_vintf_mmap_offset;
+ __aligned_u64 out_vintf_mmap_length;
+};
+
+/**
+ * struct iommu_viommu_alloc - ioctl(IOMMU_VIOMMU_ALLOC)
+ * @size: sizeof(struct iommu_viommu_alloc)
+ * @flags: Must be 0
+ * @type: Type of the virtual IOMMU. Must be defined in enum iommu_viommu_type
+ * @dev_id: The device's physical IOMMU will be used to back the virtual IOMMU
+ * @hwpt_id: ID of a nesting parent HWPT to associate to
+ * @out_viommu_id: Output virtual IOMMU ID for the allocated object
+ * @data_len: Length of the type specific data
+ * @__reserved: Must be 0
+ * @data_uptr: User pointer to a driver-specific virtual IOMMU data
+ *
+ * Allocate a virtual IOMMU object, representing the underlying physical IOMMU's
+ * virtualization support that is a security-isolated slice of the real IOMMU HW
+ * that is unique to a specific VM. Operations global to the IOMMU are connected
+ * to the vIOMMU, such as:
+ * - Security namespace for guest owned ID, e.g. guest-controlled cache tags
+ * - Non-device-affiliated event reporting, e.g. invalidation queue errors
+ * - Access to a sharable nesting parent pagetable across physical IOMMUs
+ * - Virtualization of various platforms IDs, e.g. RIDs and others
+ * - Delivery of paravirtualized invalidation
+ * - Direct assigned invalidation queues
+ * - Direct assigned interrupts
+ */
+struct iommu_viommu_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 type;
+ __u32 dev_id;
+ __u32 hwpt_id;
+ __u32 out_viommu_id;
+ __u32 data_len;
+ __u32 __reserved;
+ __aligned_u64 data_uptr;
+};
+#define IOMMU_VIOMMU_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VIOMMU_ALLOC)
+
+/**
+ * struct iommu_vdevice_alloc - ioctl(IOMMU_VDEVICE_ALLOC)
+ * @size: sizeof(struct iommu_vdevice_alloc)
+ * @viommu_id: vIOMMU ID to associate with the virtual device
+ * @dev_id: The physical device to allocate a virtual instance on the vIOMMU
+ * @out_vdevice_id: Object handle for the vDevice. Pass to IOMMU_DESTORY
+ * @virt_id: Virtual device ID per vIOMMU, e.g. vSID of ARM SMMUv3, vDeviceID
+ * of AMD IOMMU, and vRID of Intel VT-d
+ *
+ * Allocate a virtual device instance (for a physical device) against a vIOMMU.
+ * This instance holds the device's information (related to its vIOMMU) in a VM.
+ * User should use IOMMU_DESTROY to destroy the virtual device before
+ * destroying the physical device (by closing vfio_cdev fd). Otherwise the
+ * virtual device would be forcibly destroyed on physical device destruction,
+ * its vdevice_id would be permanently leaked (unremovable & unreusable) until
+ * iommu fd closed.
+ */
+struct iommu_vdevice_alloc {
+ __u32 size;
+ __u32 viommu_id;
+ __u32 dev_id;
+ __u32 out_vdevice_id;
+ __aligned_u64 virt_id;
+};
+#define IOMMU_VDEVICE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VDEVICE_ALLOC)
+
+/**
+ * struct iommu_ioas_change_process - ioctl(VFIO_IOAS_CHANGE_PROCESS)
+ * @size: sizeof(struct iommu_ioas_change_process)
+ * @__reserved: Must be 0
+ *
+ * This transfers pinned memory counts for every memory map in every IOAS
+ * in the context to the current process. This only supports maps created
+ * with IOMMU_IOAS_MAP_FILE, and returns EINVAL if other maps are present.
+ * If the ioctl returns a failure status, then nothing is changed.
+ *
+ * This API is useful for transferring operation of a device from one process
+ * to another, such as during userland live update.
+ */
+struct iommu_ioas_change_process {
+ __u32 size;
+ __u32 __reserved;
+};
+
+#define IOMMU_IOAS_CHANGE_PROCESS \
+ _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS)
+
+/**
+ * enum iommu_veventq_flag - flag for struct iommufd_vevent_header
+ * @IOMMU_VEVENTQ_FLAG_LOST_EVENTS: vEVENTQ has lost vEVENTs
+ */
+enum iommu_veventq_flag {
+ IOMMU_VEVENTQ_FLAG_LOST_EVENTS = (1U << 0),
+};
+
+/**
+ * struct iommufd_vevent_header - Virtual Event Header for a vEVENTQ Status
+ * @flags: Combination of enum iommu_veventq_flag
+ * @sequence: The sequence index of a vEVENT in the vEVENTQ, with a range of
+ * [0, INT_MAX] where the following index of INT_MAX is 0
+ *
+ * Each iommufd_vevent_header reports a sequence index of the following vEVENT:
+ *
+ * +----------------------+-------+----------------------+-------+---+-------+
+ * | header0 {sequence=0} | data0 | header1 {sequence=1} | data1 |...| dataN |
+ * +----------------------+-------+----------------------+-------+---+-------+
+ *
+ * And this sequence index is expected to be monotonic to the sequence index of
+ * the previous vEVENT. If two adjacent sequence indexes has a delta larger than
+ * 1, it means that delta - 1 number of vEVENTs has lost, e.g. two lost vEVENTs:
+ *
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ * | ... | header3 {sequence=3} | data3 | header6 {sequence=6} | data6 | ... |
+ * +-----+----------------------+-------+----------------------+-------+-----+
+ *
+ * If a vEVENT lost at the tail of the vEVENTQ and there is no following vEVENT
+ * providing the next sequence index, an IOMMU_VEVENTQ_FLAG_LOST_EVENTS header
+ * would be added to the tail, and no data would follow this header:
+ *
+ * +--+----------------------+-------+-----------------------------------------+
+ * |..| header3 {sequence=3} | data3 | header4 {flags=LOST_EVENTS, sequence=4} |
+ * +--+----------------------+-------+-----------------------------------------+
+ */
+struct iommufd_vevent_header {
+ __u32 flags;
+ __u32 sequence;
+};
+
+/**
+ * enum iommu_veventq_type - Virtual Event Queue Type
+ * @IOMMU_VEVENTQ_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_VEVENTQ_TYPE_ARM_SMMUV3: ARM SMMUv3 Virtual Event Queue
+ * @IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV Extension IRQ
+ */
+enum iommu_veventq_type {
+ IOMMU_VEVENTQ_TYPE_DEFAULT = 0,
+ IOMMU_VEVENTQ_TYPE_ARM_SMMUV3 = 1,
+ IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV = 2,
+};
+
+/**
+ * struct iommu_vevent_arm_smmuv3 - ARM SMMUv3 Virtual Event
+ * (IOMMU_VEVENTQ_TYPE_ARM_SMMUV3)
+ * @evt: 256-bit ARM SMMUv3 Event record, little-endian.
+ * Reported event records: (Refer to "7.3 Event records" in SMMUv3 HW Spec)
+ * - 0x04 C_BAD_STE
+ * - 0x06 F_STREAM_DISABLED
+ * - 0x08 C_BAD_SUBSTREAMID
+ * - 0x0a C_BAD_CD
+ * - 0x10 F_TRANSLATION
+ * - 0x11 F_ADDR_SIZE
+ * - 0x12 F_ACCESS
+ * - 0x13 F_PERMISSION
+ *
+ * StreamID field reports a virtual device ID. To receive a virtual event for a
+ * device, a vDEVICE must be allocated via IOMMU_VDEVICE_ALLOC.
+ */
+struct iommu_vevent_arm_smmuv3 {
+ __aligned_le64 evt[4];
+};
+
+/**
+ * struct iommu_vevent_tegra241_cmdqv - Tegra241 CMDQV IRQ
+ * (IOMMU_VEVENTQ_TYPE_TEGRA241_CMDQV)
+ * @lvcmdq_err_map: 128-bit logical vcmdq error map, little-endian.
+ * (Refer to register LVCMDQ_ERR_MAPs per VINTF )
+ *
+ * The 128-bit register value from HW exclusively reflect the error bits for a
+ * Virtual Interface represented by a vIOMMU object. Read and report directly.
+ */
+struct iommu_vevent_tegra241_cmdqv {
+ __aligned_le64 lvcmdq_err_map[2];
+};
+
+/**
+ * struct iommu_veventq_alloc - ioctl(IOMMU_VEVENTQ_ALLOC)
+ * @size: sizeof(struct iommu_veventq_alloc)
+ * @flags: Must be 0
+ * @viommu_id: virtual IOMMU ID to associate the vEVENTQ with
+ * @type: Type of the vEVENTQ. Must be defined in enum iommu_veventq_type
+ * @veventq_depth: Maximum number of events in the vEVENTQ
+ * @out_veventq_id: The ID of the new vEVENTQ
+ * @out_veventq_fd: The fd of the new vEVENTQ. User space must close the
+ * successfully returned fd after using it
+ * @__reserved: Must be 0
+ *
+ * Explicitly allocate a virtual event queue interface for a vIOMMU. A vIOMMU
+ * can have multiple FDs for different types, but is confined to one per @type.
+ * User space should open the @out_veventq_fd to read vEVENTs out of a vEVENTQ,
+ * if there are vEVENTs available. A vEVENTQ will lose events due to overflow,
+ * if the number of the vEVENTs hits @veventq_depth.
+ *
+ * Each vEVENT in a vEVENTQ encloses a struct iommufd_vevent_header followed by
+ * a type-specific data structure, in a normal case:
+ *
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ * | | header0 | data0 | header1 | data1 | ... | headerN | dataN | |
+ * +-+---------+-------+---------+-------+-----+---------+-------+-+
+ *
+ * unless a tailing IOMMU_VEVENTQ_FLAG_LOST_EVENTS header is logged (refer to
+ * struct iommufd_vevent_header).
+ */
+struct iommu_veventq_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 veventq_depth;
+ __u32 out_veventq_id;
+ __u32 out_veventq_fd;
+ __u32 __reserved;
+};
+#define IOMMU_VEVENTQ_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_VEVENTQ_ALLOC)
+
+/**
+ * enum iommu_hw_queue_type - HW Queue Type
+ * @IOMMU_HW_QUEUE_TYPE_DEFAULT: Reserved for future use
+ * @IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV: NVIDIA Tegra241 CMDQV (extension for ARM
+ * SMMUv3) Virtual Command Queue (VCMDQ)
+ */
+enum iommu_hw_queue_type {
+ IOMMU_HW_QUEUE_TYPE_DEFAULT = 0,
+ /*
+ * TEGRA241_CMDQV requirements (otherwise, allocation will fail)
+ * - alloc starts from the lowest @index=0 in ascending order
+ * - destroy starts from the last allocated @index in descending order
+ * - @base_addr must be aligned to @length in bytes and mapped in IOAS
+ * - @length must be a power of 2, with a minimum 32 bytes and a maximum
+ * 2 ^ idr[1].CMDQS * 16 bytes (use GET_HW_INFO call to read idr[1]
+ * from struct iommu_hw_info_arm_smmuv3)
+ * - suggest to back the queue memory with contiguous physical pages or
+ * a single huge page with alignment of the queue size, and limit the
+ * emulated vSMMU's IDR1.CMDQS to log2(huge page size / 16 bytes)
+ */
+ IOMMU_HW_QUEUE_TYPE_TEGRA241_CMDQV = 1,
+};
+
+/**
+ * struct iommu_hw_queue_alloc - ioctl(IOMMU_HW_QUEUE_ALLOC)
+ * @size: sizeof(struct iommu_hw_queue_alloc)
+ * @flags: Must be 0
+ * @viommu_id: Virtual IOMMU ID to associate the HW queue with
+ * @type: One of enum iommu_hw_queue_type
+ * @index: The logical index to the HW queue per virtual IOMMU for a multi-queue
+ * model
+ * @out_hw_queue_id: The ID of the new HW queue
+ * @nesting_parent_iova: Base address of the queue memory in the guest physical
+ * address space
+ * @length: Length of the queue memory
+ *
+ * Allocate a HW queue object for a vIOMMU-specific HW-accelerated queue, which
+ * allows HW to access a guest queue memory described using @nesting_parent_iova
+ * and @length.
+ *
+ * A vIOMMU can allocate multiple queues, but it must use a different @index per
+ * type to separate each allocation, e.g::
+ *
+ * Type1 HW queue0, Type1 HW queue1, Type2 HW queue0, ...
+ */
+struct iommu_hw_queue_alloc {
+ __u32 size;
+ __u32 flags;
+ __u32 viommu_id;
+ __u32 type;
+ __u32 index;
+ __u32 out_hw_queue_id;
+ __aligned_u64 nesting_parent_iova;
+ __aligned_u64 length;
+};
+#define IOMMU_HW_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HW_QUEUE_ALLOC)
+#endif
diff --git a/kernel/linux/uapi/linux/vduse.h b/kernel/linux/uapi/linux/vduse.h
index f46269af34..da6ac89af1 100644
--- a/kernel/linux/uapi/linux/vduse.h
+++ b/kernel/linux/uapi/linux/vduse.h
@@ -237,7 +237,7 @@ struct vduse_iova_umem {
* struct vduse_iova_info - information of one IOVA region
* @start: start of the IOVA region
* @last: last of the IOVA region
- * @capability: capability of the IOVA regsion
+ * @capability: capability of the IOVA region
* @reserved: for future use, needs to be initialized to zero
*
* Structure used by VDUSE_IOTLB_GET_INFO ioctl to get information of
diff --git a/kernel/linux/uapi/linux/vfio.h b/kernel/linux/uapi/linux/vfio.h
index 79bf8c0cc5..4d96d1fc12 100644
--- a/kernel/linux/uapi/linux/vfio.h
+++ b/kernel/linux/uapi/linux/vfio.h
@@ -905,10 +905,12 @@ struct vfio_device_feature {
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
- * @flags: Must be 0.
+ * @flags: Must be 0 or a bit flags of VFIO_DEVICE_BIND_*
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
+ * @token_uuid_ptr: Valid if VFIO_DEVICE_BIND_FLAG_TOKEN. Points to a 16 byte
+ * UUID in the same format as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN.
*
* Bind a vfio_device to the specified iommufd.
*
@@ -917,13 +919,21 @@ struct vfio_device_feature {
*
* Unbind is automatically conducted when device fd is closed.
*
+ * A token is sometimes required to open the device, unless this is known to be
+ * needed VFIO_DEVICE_BIND_FLAG_TOKEN should not be set and token_uuid_ptr is
+ * ignored. The only case today is a PF/VF relationship where the VF bind must
+ * be provided the same token as VFIO_DEVICE_FEATURE_PCI_VF_TOKEN provided to
+ * the PF.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
+#define VFIO_DEVICE_BIND_FLAG_TOKEN (1 << 0)
__s32 iommufd;
__u32 out_devid;
+ __aligned_u64 token_uuid_ptr;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
diff --git a/kernel/linux/uapi/version b/kernel/linux/uapi/version
index 966a998301..d9e789dade 100644
--- a/kernel/linux/uapi/version
+++ b/kernel/linux/uapi/version
@@ -1 +1 @@
-v6.16
+v6.17
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 02/25] vfio: make all functions internal
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 01/25] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 03/25] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
` (23 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
The VFIO API is an externally exported API because the original intent was
to offer DMA mapping facilities to applications. However, practical usage
of this API seems to be centered around drivers, so keeping this API
exported to applications only creates needless API/ABI stability surface
that has no added value.
Make the entire VFIO API internal-only and visible only to drivers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/api/doxy-api-index.md | 3 +-
doc/guides/rel_notes/deprecation.rst | 6 ----
doc/guides/rel_notes/release_26_11.rst | 8 +++++
lib/eal/freebsd/eal.c | 30 ++++++++--------
lib/eal/include/meson.build | 2 +-
lib/eal/include/rte_vfio.h | 47 ++++++++++++++++++++++----
lib/eal/linux/eal_vfio.c | 32 +++++++++---------
lib/eal/windows/eal.c | 4 +--
8 files changed, 84 insertions(+), 48 deletions(-)
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 9296042119..4216b1b9e4 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -37,8 +37,7 @@ The public API headers are grouped by topics:
[latency](@ref rte_latencystats.h),
[devargs](@ref rte_devargs.h),
[PCI](@ref rte_pci.h),
- [vdev](@ref rte_bus_vdev.h),
- [vfio](@ref rte_vfio.h)
+ [vdev](@ref rte_bus_vdev.h)
- **device specific**:
[softnic](@ref rte_eth_softnic.h),
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6ad7698c6b..8a82c815e3 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,12 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* eal: The entire VFIO API (``rte_vfio_*``) will be made internal only,
- and will only be available to EAL and drivers.
- Group-based API (``rte_vfio_*_group_*``) will be removed
- and replaced with unified container device assignment API.
- This change will be made in 26.11 release.
-
* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
as part of its internal API. All drivers will be adjusted
to use the new unified VFIO container device assignment API.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index c8cc86295d..b5793b5c94 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -93,6 +93,10 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public API.
+
ABI Changes
-----------
@@ -109,6 +113,10 @@ ABI Changes
Also, make sure to start the actual text at the margin.
=======================================================
+* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
+ These functions are now available only to EAL and drivers,
+ and are no longer part of the public ABI.
+
Known Issues
------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 8b1ba5b99b..0fe54a9dd7 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -819,7 +819,7 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
@@ -829,7 +829,7 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int rte_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
@@ -838,33 +838,33 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int rte_vfio_enable(__rte_unused const char *modname)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int rte_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
@@ -874,7 +874,7 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -882,7 +882,7 @@ rte_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
@@ -890,7 +890,7 @@ rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -898,7 +898,7 @@ rte_vfio_container_create(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(__rte_unused int container_fd)
{
@@ -906,7 +906,7 @@ rte_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -915,7 +915,7 @@ rte_vfio_container_group_bind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
@@ -924,7 +924,7 @@ rte_vfio_container_group_unbind(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -935,7 +935,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index aef5824e5f..9282143918 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -55,12 +55,12 @@ headers += files(
'rte_trace_point_register.h',
'rte_uuid.h',
'rte_version.h',
- 'rte_vfio.h',
)
driver_sdk_headers = files(
'bus_driver.h',
'dev_driver.h',
+ 'rte_vfio.h',
)
# special case install the generic headers, since they go in a subdir
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index d1e8bce56b..0ddeb08f94 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -7,7 +7,11 @@
/**
* @file
- * RTE VFIO. This library provides various VFIO related utility functions.
+ * @internal
+ *
+ * RTE VFIO internal API.
+ *
+ * This library provides VFIO related utility functions for use by drivers.
*/
#include <stdbool.h>
@@ -36,6 +40,7 @@ struct vfio_device_info;
#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
+ * @internal
* Setup vfio_cfg for the device identified by its address.
* It discovers the configured I/O MMU groups or sets a new one for the device.
* If a new groups is assigned, the DMA mapping is performed.
@@ -60,10 +65,12 @@ struct vfio_device_info;
* <0 on failure.
* >1 if the device cannot be managed this way.
*/
+__rte_internal
int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Release a device mapped to a VFIO-managed I/O MMU group.
*
* This function is only relevant to linux and will return
@@ -82,9 +89,11 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
+ * @internal
* Enable a VFIO-related kmod.
*
* This function is only relevant to linux and will return
@@ -97,9 +106,11 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int rte_vfio_enable(const char *modname);
/**
+ * @internal
* Check whether a VFIO-related kmod is enabled.
*
* This function is only relevant to Linux.
@@ -111,9 +122,11 @@ int rte_vfio_enable(const char *modname);
* 1 if true.
* 0 otherwise.
*/
+__rte_internal
int rte_vfio_is_enabled(const char *modname);
/**
+ * @internal
* Whether VFIO NOIOMMU mode is enabled.
*
* This function is only relevant to Linux.
@@ -123,10 +136,12 @@ int rte_vfio_is_enabled(const char *modname);
* 0 if false.
* <0 for errors.
*/
+__rte_internal
int rte_vfio_noiommu_is_enabled(void);
/**
- * Remove group fd from internal VFIO group fd array/
+ * @internal
+ * Remove group fd from internal VFIO group fd array.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -138,11 +153,13 @@ int rte_vfio_noiommu_is_enabled(void);
* 0 on success.
* <0 on failure.
*/
+__rte_internal
int
rte_vfio_clear_group(int vfio_group_fd);
/**
- * Parse IOMMU group number for a device
+ * @internal
+ * Parse IOMMU group number for a device.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -161,12 +178,14 @@ rte_vfio_clear_group(int vfio_group_fd);
* 0 for non-existent group or VFIO
* <0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
- * Get device information
+ * @internal
+ * Get device information.
*
* This function is only relevant to Linux and will return an error on BSD.
*
@@ -186,12 +205,13 @@ rte_vfio_get_group_num(const char *sysfs_base,
* 0 on success.
* <0 on failure.
*/
-__rte_experimental
+__rte_internal
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
+ * @internal
* Get the default VFIO container fd
*
* This function is only relevant to linux and will return
@@ -201,11 +221,13 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
* > 0 default container fd
* < 0 if VFIO is not enabled or not supported
*/
+__rte_internal
int
rte_vfio_get_container_fd(void);
/**
- * Open VFIO group fd or get an existing one
+ * @internal
+ * Open VFIO group fd or get an existing one.
*
* This function is only relevant to linux and will return
* an error on BSD.
@@ -217,10 +239,12 @@ rte_vfio_get_container_fd(void);
* > 0 group fd
* < 0 for errors
*/
+__rte_internal
int
rte_vfio_get_group_fd(int iommu_group_num);
/**
+ * @internal
* Create a new container for device binding.
*
* @note Any newly allocated DPDK memory will not be mapped into these
@@ -235,10 +259,12 @@ rte_vfio_get_group_fd(int iommu_group_num);
* the container fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_create(void);
/**
+ * @internal
* Destroy the container, unbind all vfio groups within it.
*
* @param container_fd
@@ -248,10 +274,12 @@ rte_vfio_container_create(void);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_destroy(int container_fd);
/**
+ * @internal
* Bind a IOMMU group to a container.
*
* @param container_fd
@@ -264,10 +292,12 @@ rte_vfio_container_destroy(int container_fd);
* group fd if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Unbind a IOMMU group from a container.
*
* @param container_fd
@@ -280,10 +310,12 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
+ * @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
@@ -303,11 +335,13 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
+ * @internal
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
@@ -327,6 +361,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* 0 if successful
* <0 if failed
*/
+__rte_internal
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index f1050ffa60..33fa04feaf 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -532,7 +532,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
int
rte_vfio_get_group_fd(int iommu_group_num)
{
@@ -731,7 +731,7 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
int
rte_vfio_clear_group(int vfio_group_fd)
{
@@ -755,7 +755,7 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
int
rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1009,7 +1009,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
int
rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
rte_vfio_enable(const char *modname)
{
@@ -1175,7 +1175,7 @@ rte_vfio_enable(const char *modname)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int
rte_vfio_is_enabled(const char *modname)
{
@@ -1215,7 +1215,7 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_vfio_get_device_info, 24.03)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
int
rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
@@ -1349,7 +1349,7 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
int
rte_vfio_get_container_fd(void)
{
@@ -1363,7 +1363,7 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
int
rte_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
@@ -2034,7 +2034,7 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
int
rte_vfio_noiommu_is_enabled(void)
{
@@ -2067,7 +2067,7 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
int
rte_vfio_container_create(void)
{
@@ -2094,7 +2094,7 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
int
rte_vfio_container_destroy(int container_fd)
{
@@ -2120,7 +2120,7 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
int
rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
@@ -2135,7 +2135,7 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
int
rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
@@ -2176,7 +2176,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
@@ -2197,7 +2197,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index 6dacae7235..de7a89a829 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -453,7 +453,7 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
int
rte_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
@@ -464,7 +464,7 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
int
rte_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 03/25] bus/fslmc: decouple from EAL VFIO
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 01/25] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 02/25] vfio: make all functions internal Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 04/25] vfio: decouple EAL from VFIO internals Anatoly Burakov
` (22 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus has its own VFIO implementation which is completely separate from
one in EAL or rte_vfio. Yet, it references a couple of macros and uses a
struct definition. Fully decoupling them is not a huge lift, so do that.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/fslmc/fslmc_vfio.c | 50 +++++++++++++++++++++-------------
drivers/bus/fslmc/meson.build | 1 -
2 files changed, 31 insertions(+), 20 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 412b70e5ae..8d39895bac 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -35,7 +35,6 @@
#include <dev_driver.h>
#include <rte_eal_memconfig.h>
#include <rte_vfio.h>
-#include <eal_vfio.h>
#include "private.h"
#include "fslmc_vfio.h"
@@ -73,8 +72,21 @@ struct fslmc_dmaseg_list fslmc_iosegs =
static uint64_t fslmc_mem_va2iova = RTE_BAD_IOVA;
static int fslmc_mem_map_num;
+#define FSLMC_VFIO_SOCKET_OK 0x0
+#define FSLMC_VFIO_SOCKET_NO_FD 0x1
+#define FSLMC_VFIO_SOCKET_ERR 0xFF
+
+struct fslmc_vfio_mp_param {
+ int req;
+ int result;
+ union {
+ int group_num;
+ int iommu_type_id;
+ };
+};
+
struct fslmc_mem_param {
- struct vfio_mp_param mp_param;
+ struct fslmc_vfio_mp_param mp_param;
struct fslmc_dmaseg_list memsegs;
struct fslmc_dmaseg_list iosegs;
uint64_t mem_va2iova;
@@ -386,7 +398,7 @@ fslmc_vfio_open_group_fd(const char *group_name)
struct rte_mp_msg mp_req, *mp_rep;
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ struct fslmc_vfio_mp_param *p = (struct fslmc_vfio_mp_param *)mp_req.param;
int iommu_group_num, ret;
vfio_group_fd = fslmc_vfio_group_fd_by_name(group_name);
@@ -417,10 +429,10 @@ fslmc_vfio_open_group_fd(const char *group_name)
if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
mp_reply.nb_received == 1) {
mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1)
+ p = (struct fslmc_vfio_mp_param *)mp_rep->param;
+ if (p->result == FSLMC_VFIO_SOCKET_OK && mp_rep->num_fds == 1)
vfio_group_fd = mp_rep->fds[0];
- else if (p->result == SOCKET_NO_FD)
+ else if (p->result == FSLMC_VFIO_SOCKET_NO_FD)
DPAA2_BUS_ERR("Bad VFIO group fd");
}
@@ -490,7 +502,7 @@ fslmc_vfio_open_container_fd(void)
struct rte_mp_msg mp_req, *mp_rep;
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (void *)mp_req.param;
+ struct fslmc_vfio_mp_param *p = (void *)mp_req.param;
if (fslmc_vfio_container_fd() > 0)
return fslmc_vfio_container_fd();
@@ -551,7 +563,7 @@ fslmc_vfio_open_container_fd(void)
mp_rep = &mp_reply.msgs[0];
p = (void *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
+ if (p->result == FSLMC_VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
vfio_container_fd = mp_rep->fds[0];
free(mp_reply.msgs);
}
@@ -594,8 +606,8 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
int fd = -1;
int ret;
struct rte_mp_msg reply;
- struct vfio_mp_param *r = (void *)reply.param;
- const struct vfio_mp_param *m = (const void *)msg->param;
+ struct fslmc_vfio_mp_param *r = (void *)reply.param;
+ const struct fslmc_vfio_mp_param *m = (const void *)msg->param;
struct fslmc_mem_param *map;
if (msg->len_param != sizeof(*m)) {
@@ -612,13 +624,13 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
r->group_num = m->group_num;
fd = fslmc_vfio_group_fd_by_id(m->group_num);
if (fd < 0) {
- r->result = SOCKET_ERR;
+ r->result = FSLMC_VFIO_SOCKET_ERR;
} else if (!fd) {
/* if group exists but isn't bound to VFIO driver */
- r->result = SOCKET_NO_FD;
+ r->result = FSLMC_VFIO_SOCKET_NO_FD;
} else {
/* if group exists and is bound to VFIO driver */
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -628,9 +640,9 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
r->req = FSLMC_VFIO_SOCKET_REQ_CONTAINER;
fd = fslmc_vfio_container_fd();
if (fd <= 0) {
- r->result = SOCKET_ERR;
+ r->result = FSLMC_VFIO_SOCKET_ERR;
} else {
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
@@ -640,7 +652,7 @@ fslmc_vfio_mp_primary(const struct rte_mp_msg *msg,
map = (void *)reply.param;
r = &map->mp_param;
r->req = FSLMC_VFIO_SOCKET_REQ_MEM;
- r->result = SOCKET_OK;
+ r->result = FSLMC_VFIO_SOCKET_OK;
map->memsegs = fslmc_memsegs;
map->iosegs = fslmc_iosegs;
map->mem_va2iova = fslmc_mem_va2iova;
@@ -666,19 +678,19 @@ fslmc_vfio_mp_sync_mem_req(void)
struct rte_mp_reply mp_reply = {0};
struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
int ret = 0;
- struct vfio_mp_param *mp_param;
+ struct fslmc_vfio_mp_param *mp_param;
struct fslmc_mem_param *mem_rsp;
mp_param = (void *)mp_req.param;
memset(&mp_req, 0, sizeof(struct rte_mp_msg));
mp_param->req = FSLMC_VFIO_SOCKET_REQ_MEM;
rte_strscpy(mp_req.name, FSLMC_VFIO_MP, sizeof(mp_req.name));
- mp_req.len_param = sizeof(struct vfio_mp_param);
+ mp_req.len_param = sizeof(struct fslmc_vfio_mp_param);
if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
mp_reply.nb_received == 1) {
mp_rep = &mp_reply.msgs[0];
mem_rsp = (struct fslmc_mem_param *)mp_rep->param;
- if (mem_rsp->mp_param.result == SOCKET_OK) {
+ if (mem_rsp->mp_param.result == FSLMC_VFIO_SOCKET_OK) {
fslmc_memsegs = mem_rsp->memsegs;
fslmc_mem_va2iova = mem_rsp->mem_va2iova;
fslmc_mem_map_num = mem_rsp->mem_map_num;
diff --git a/drivers/bus/fslmc/meson.build b/drivers/bus/fslmc/meson.build
index 50d9e91a37..55f9d2e808 100644
--- a/drivers/bus/fslmc/meson.build
+++ b/drivers/bus/fslmc/meson.build
@@ -30,4 +30,3 @@ sources = files(
)
includes += include_directories('mc', 'qbman/include', 'portal')
-includes += include_directories('../../../lib/eal/linux')
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 04/25] vfio: decouple EAL from VFIO internals
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (2 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 03/25] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 05/25] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
` (21 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson, Dmitry Kozlyuk
Currently, the multiprocess cleanup for VFIO is called via an internal-only
API, and this is the only thing `eal_vfio.h` is used for in EAL. In order
to make `eal_vfio.h` truly VFIO-internal, export a cleanup function from
`rte_vfio` API instead, and call that. This allows us to remove all EAL
references to `eal_vfio.h` in non-VFIO related files.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 6 ++++++
lib/eal/include/rte_vfio.h | 9 +++++++++
lib/eal/linux/eal.c | 3 +--
lib/eal/linux/eal_vfio.c | 7 +++++++
lib/eal/windows/eal.c | 6 ++++++
5 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 0fe54a9dd7..991ca45064 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -845,6 +845,12 @@ int rte_vfio_enable(__rte_unused const char *modname)
return -1;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
int rte_vfio_is_enabled(__rte_unused const char *modname)
{
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 0ddeb08f94..e76181c83c 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -109,6 +109,15 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
__rte_internal
int rte_vfio_enable(const char *modname);
+/**
+ * @internal
+ * Cleanup VFIO resources.
+ *
+ * This function is only relevant to Linux.
+ */
+__rte_internal
+void rte_vfio_cleanup(void);
+
/**
* @internal
* Check whether a VFIO-related kmod is enabled.
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index fc2e9b8c0e..5577bab24a 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -54,7 +54,6 @@
#include "eal_memcfg.h"
#include "eal_trace.h"
#include "eal_options.h"
-#include "eal_vfio.h"
#include "hotplug_mp.h"
#include "log_internal.h"
@@ -986,7 +985,7 @@ rte_eal_cleanup(void)
rte_service_finalize();
eal_bus_cleanup();
- vfio_mp_sync_cleanup();
+ rte_vfio_cleanup();
rte_mp_channel_cleanup();
rte_eal_alarm_cleanup();
rte_trace_save();
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 33fa04feaf..6c77076bc3 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2217,3 +2217,10 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_unmap(vfio_cfg, vaddr, iova, len);
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+ vfio_mp_sync_cleanup();
+}
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index de7a89a829..b8771d4d85 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -152,6 +152,12 @@ rte_eal_cleanup(void)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+void
+rte_vfio_cleanup(void)
+{
+}
+
/* Launch threads, called at application init(). */
RTE_EXPORT_SYMBOL(rte_eal_init)
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 05/25] vfio: do proper teardown on VFIO cleanup
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (3 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 04/25] vfio: decouple EAL from VFIO internals Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 06/25] vfio: remove modname parameter from init Anatoly Burakov
` (20 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO cleanup only unregisters multiprocess callback, but does
not destroy containers, groups, and user mem maps. Do all of that on VFIO
cleanup. In order to distinguish between config that is not initialized vs.
config that has been initialized but happens to have fd == 0, move the
global VFIO enabled flag out of the config, and add a separate per-config
"enabled" flag that can be checked to avoid attempting to clean up configs
that were never initialized in the first place.
While we're at it, also harden the API against repeated initialization and
attempts at using the API without having VFIO initialized.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 3 +-
lib/eal/linux/eal_vfio.c | 151 +++++++++++++++++++++++++++++++++++++--
2 files changed, 147 insertions(+), 7 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 991ca45064..aa6d4c2e16 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -846,8 +846,7 @@ int rte_vfio_enable(__rte_unused const char *modname)
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
-void
-rte_vfio_cleanup(void)
+void rte_vfio_cleanup(void)
{
}
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 6c77076bc3..d4173f7a01 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -46,7 +46,7 @@ struct user_mem_maps {
};
struct vfio_config {
- int vfio_enabled;
+ bool enabled;
int vfio_container_fd;
int vfio_active_groups;
const struct vfio_iommu_type *vfio_iommu_type;
@@ -58,6 +58,9 @@ struct vfio_config {
static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
+/* whether VFIO is enabled (usable) in this process */
+static bool vfio_enabled;
+
static int vfio_type1_dma_map(int);
static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_spapr_dma_map(int);
@@ -538,6 +541,11 @@ rte_vfio_get_group_fd(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
/* get the vfio_config it belongs to */
vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
@@ -686,7 +694,7 @@ vfio_sync_default_container(void)
return -1;
/* default container fd should have been opened in rte_vfio_enable() */
- if (!default_vfio_cfg->vfio_enabled ||
+ if (!vfio_enabled ||
default_vfio_cfg->vfio_container_fd < 0) {
EAL_LOG(ERR, "VFIO support is not initialized");
return -1;
@@ -738,6 +746,11 @@ rte_vfio_clear_group(int vfio_group_fd)
int i;
struct vfio_config *vfio_cfg;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
if (vfio_cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO group fd!");
@@ -773,6 +786,11 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
const struct internal_config *internal_conf =
eal_get_internal_configuration();
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
/* get group number */
ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
if (ret == 0) {
@@ -1019,6 +1037,11 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int iommu_group_num;
int ret;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
/* we don't want any DMA mapping messages to come while we're detaching
* VFIO device, because this might be the last device and we might need
* to unregister the callback.
@@ -1111,7 +1134,11 @@ rte_vfio_enable(const char *modname)
rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
+ if (vfio_enabled)
+ return 0;
+
for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
+ vfio_cfgs[i].enabled = false;
vfio_cfgs[i].vfio_container_fd = -1;
vfio_cfgs[i].vfio_active_groups = 0;
vfio_cfgs[i].vfio_iommu_type = NULL;
@@ -1167,7 +1194,8 @@ rte_vfio_enable(const char *modname)
/* check if we have VFIO driver enabled */
if (default_vfio_cfg->vfio_container_fd != -1) {
EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->vfio_enabled = 1;
+ default_vfio_cfg->enabled = true;
+ vfio_enabled = true;
} else {
EAL_LOG(NOTICE, "VFIO support could not be initialized");
}
@@ -1180,12 +1208,15 @@ int
rte_vfio_is_enabled(const char *modname)
{
const int mod_available = rte_eal_check_module(modname) > 0;
- return default_vfio_cfg->vfio_enabled && mod_available;
+ return vfio_enabled && mod_available;
}
int
vfio_get_iommu_type(void)
{
+ if (!vfio_enabled)
+ return -1;
+
if (default_vfio_cfg->vfio_iommu_type == NULL)
return -1;
@@ -1222,6 +1253,11 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
{
int ret;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
if (device_info == NULL || *vfio_dev_fd < 0)
return -1;
@@ -1357,7 +1393,7 @@ rte_vfio_get_container_fd(void)
* The default container is set up during rte_vfio_enable().
* This function does not create a new container.
*/
- if (!default_vfio_cfg->vfio_enabled)
+ if (!vfio_enabled)
return -1;
return default_vfio_cfg->vfio_container_fd;
@@ -1373,6 +1409,11 @@ rte_vfio_get_group_num(const char *sysfs_base,
char *tok[16], *group_tok, *end;
int ret;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
memset(linkname, 0, sizeof(linkname));
memset(filename, 0, sizeof(filename));
@@ -2073,6 +2114,11 @@ rte_vfio_container_create(void)
{
unsigned int i;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
/* Find an empty slot to store new vfio config */
for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
if (vfio_cfgs[i].vfio_container_fd == -1)
@@ -2090,6 +2136,7 @@ rte_vfio_container_create(void)
EAL_LOG(NOTICE, "Fail to create a new VFIO container");
return -1;
}
+ vfio_cfgs[i].enabled = true;
return vfio_cfgs[i].vfio_container_fd;
}
@@ -2101,6 +2148,16 @@ rte_vfio_container_destroy(int container_fd)
struct vfio_config *vfio_cfg;
unsigned int i;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
+ if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD) {
+ EAL_LOG(ERR, "Cannot destroy default VFIO container");
+ return -1;
+ }
+
vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
if (vfio_cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
@@ -2113,6 +2170,7 @@ rte_vfio_container_destroy(int container_fd)
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
+ vfio_cfg->enabled = false;
vfio_cfg->vfio_container_fd = -1;
vfio_cfg->vfio_active_groups = 0;
vfio_cfg->vfio_iommu_type = NULL;
@@ -2126,6 +2184,11 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
if (vfio_cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
@@ -2143,6 +2206,11 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
struct vfio_config *vfio_cfg;
unsigned int i;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
if (vfio_cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
@@ -2183,6 +2251,11 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
{
struct vfio_config *vfio_cfg;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
if (len == 0) {
rte_errno = EINVAL;
return -1;
@@ -2204,6 +2277,11 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
{
struct vfio_config *vfio_cfg;
+ if (!vfio_enabled) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+
if (len == 0) {
rte_errno = EINVAL;
return -1;
@@ -2218,9 +2296,72 @@ rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_unmap(vfio_cfg, vaddr, iova, len);
}
+static void
+vfio_cleanup_config(struct vfio_config *vfio_cfg)
+{
+ unsigned int i;
+
+ for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
+ struct vfio_group *group = &vfio_cfg->vfio_groups[i];
+
+ if (group->group_num == -1)
+ continue;
+ if (group->devices != 0) {
+ EAL_LOG(ERR, "Cannot cleanup VFIO group %d with %d devices",
+ group->group_num, group->devices);
+ return;
+ }
+ if (group->fd >= 0 && close(group->fd) < 0) {
+ EAL_LOG(ERR, "Cannot close VFIO group %d: %s",
+ group->group_num, strerror(errno));
+ return;
+ }
+
+ group->group_num = -1;
+ group->fd = -1;
+ group->devices = 0;
+ vfio_cfg->vfio_active_groups--;
+ }
+
+ /* if there are still active groups, we cannot cleanup the container */
+ if (vfio_cfg->vfio_active_groups != 0) {
+ EAL_LOG(ERR, "Cannot cleanup VFIO container with %d active groups",
+ vfio_cfg->vfio_active_groups);
+ return;
+ }
+
+ if (vfio_cfg->vfio_container_fd >= 0 &&
+ close(vfio_cfg->vfio_container_fd) < 0) {
+ EAL_LOG(ERR, "Cannot close VFIO container: %s", strerror(errno));
+ return;
+ }
+
+ vfio_cfg->vfio_container_fd = -1;
+ vfio_cfg->enabled = false;
+ vfio_cfg->vfio_iommu_type = NULL;
+
+ vfio_cfg->mem_maps.n_maps = 0;
+ memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
void
rte_vfio_cleanup(void)
{
+ unsigned int i;
+
+ if (!vfio_enabled)
+ return;
+
vfio_mp_sync_cleanup();
+
+ /* mem events can only be unregistered from the primary process */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+
+ /* cleanup all enabled configs, ignore failures */
+ for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
+ if (vfio_cfgs[i].enabled)
+ vfio_cleanup_config(&vfio_cfgs[i]);
+ }
}
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 06/25] vfio: remove modname parameter from init
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (4 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 05/25] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 07/25] vfio: rename API to reflect internal status Anatoly Burakov
` (19 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta, Bruce Richardson
All VFIO initialization always happens with "vfio" module, and now that the
API is not public, having a way to override the module name seems redundant
as there is not going to be a use case that calls into `rte_vfio_enable`
that isn't coming from EAL init. So, remove the parameter, and hardcode
the modname to "vfio" instead.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 2 +-
lib/eal/freebsd/eal.c | 2 +-
lib/eal/include/rte_vfio.h | 7 ++-----
lib/eal/linux/eal.c | 2 +-
lib/eal/linux/eal_vfio.c | 5 +++--
5 files changed, 8 insertions(+), 10 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c2499..f3791cce98 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -1303,7 +1303,7 @@ pci_vfio_is_enabled(void)
int status = rte_vfio_is_enabled("vfio_pci");
if (!status) {
- rte_vfio_enable("vfio");
+ rte_vfio_enable();
status = rte_vfio_is_enabled("vfio_pci");
}
return status;
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index aa6d4c2e16..52d265769e 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -839,7 +839,7 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
}
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
-int rte_vfio_enable(__rte_unused const char *modname)
+int rte_vfio_enable(void)
{
rte_errno = ENOTSUP;
return -1;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index e76181c83c..b53ce56a8d 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -94,20 +94,17 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
/**
* @internal
- * Enable a VFIO-related kmod.
+ * Initialize VFIO.
*
* This function is only relevant to linux and will return
* an error on BSD.
*
- * @param modname
- * kernel module name.
- *
* @return
* 0 on success.
* <0 on failure.
*/
__rte_internal
-int rte_vfio_enable(const char *modname);
+int rte_vfio_enable(void);
/**
* @internal
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 5577bab24a..17977baedc 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -771,7 +771,7 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (rte_vfio_enable("vfio")) {
+ if (rte_vfio_enable()) {
rte_eal_init_alert("Cannot init VFIO");
rte_errno = EAGAIN;
goto err_out;
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index d4173f7a01..da46ef2f17 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -25,6 +25,7 @@
#include "eal_internal_cfg.h"
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
+#define VFIO_MODNAME "vfio"
/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
* recreate the mappings for DPDK segments, but we cannot do so for memory that
@@ -1123,7 +1124,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
int
-rte_vfio_enable(const char *modname)
+rte_vfio_enable(void)
{
/* initialize group list */
unsigned int i, j;
@@ -1154,7 +1155,7 @@ rte_vfio_enable(const char *modname)
EAL_LOG(DEBUG, "Probing VFIO support...");
/* check if vfio module is loaded */
- vfio_available = rte_eal_check_module(modname);
+ vfio_available = rte_eal_check_module(VFIO_MODNAME);
/* return error directly */
if (vfio_available == -1) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 07/25] vfio: rename API to reflect internal status
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (5 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 06/25] vfio: remove modname parameter from init Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 08/25] bus/pci: rename mismatching error labels Anatoly Burakov
` (18 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Hemant Agrawal, Sachin Saxena,
Chenbo Xia, Tomasz Duszynski, Ajit Khaparde, Vikas Gupta,
Feifei Wang, Dimon Zhao, Leon Yu, Sam Chen, Christian Koue Muf,
Serhii Iliushyk, Chaoyong He, Vijay Kumar Srivastava,
Bruce Richardson, Dmitry Kozlyuk, Maxime Coquelin
Currently, even though VFIO API is internal-only, it is still named with an
`rte_` prefix, which implies public API. To reflect the fact that this API
is no longer public, rename all symbols and macros from `rte_`/`RTE_` to
`dev_`/`DEV_`, and adjust all callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/rel_notes/release_26_11.rst | 4 +-
drivers/bus/cdx/cdx.c | 6 +-
drivers/bus/cdx/cdx_vfio.c | 14 +--
drivers/bus/fslmc/fslmc_bus.c | 4 +-
drivers/bus/fslmc/fslmc_vfio.c | 12 +-
drivers/bus/pci/linux/pci.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 38 +++----
drivers/bus/pci/pci_common.c | 10 +-
drivers/bus/platform/platform.c | 10 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 +-
drivers/net/hinic3/base/hinic3_hwdev.c | 4 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 16 +--
drivers/net/ntnic/ntnic_ethdev.c | 10 +-
drivers/net/ntnic/ntnic_vfio.c | 22 ++--
drivers/vdpa/ifc/ifcvf_vdpa.c | 24 ++--
drivers/vdpa/nfp/nfp_vdpa.c | 24 ++--
drivers/vdpa/sfc/sfc_vdpa.c | 16 +--
drivers/vdpa/sfc/sfc_vdpa_hw.c | 12 +-
lib/eal/freebsd/eal.c | 73 ++++++------
lib/eal/include/{rte_vfio.h => dev_vfio.h} | 58 +++++-----
lib/eal/include/meson.build | 2 +-
lib/eal/linux/eal.c | 6 +-
lib/eal/linux/eal_vfio.c | 122 ++++++++++-----------
lib/eal/linux/eal_vfio_mp_sync.c | 6 +-
lib/eal/windows/eal.c | 14 +--
lib/vhost/vhost_user.c | 16 +--
26 files changed, 271 insertions(+), 264 deletions(-)
rename lib/eal/include/{rte_vfio.h => dev_vfio.h} (82%)
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index b5793b5c94..71d12a4398 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -94,8 +94,8 @@ API Changes
=======================================================
* vfio: The entire VFIO API (``rte_vfio_*``) was made internal.
- These functions are now available only to EAL and drivers,
- and are no longer part of the public API.
+ These functions were renamed to ``dev_vfio_*``, are now available
+ only to EAL and drivers, and are no longer part of public API.
ABI Changes
diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c
index c0b46a41ad..ee8f24e67e 100644
--- a/drivers/bus/cdx/cdx.c
+++ b/drivers/bus/cdx/cdx.c
@@ -73,7 +73,7 @@
#include <rte_devargs.h>
#include <rte_kvargs.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include <eal_filesystem.h>
@@ -403,7 +403,7 @@ cdx_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
RTE_SET_USED(dev);
- return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ return dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD,
(uintptr_t)addr, iova, len);
}
@@ -412,7 +412,7 @@ cdx_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
{
RTE_SET_USED(dev);
- return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ return dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD,
(uintptr_t)addr, iova, len);
}
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 11fe3265d2..02a34d6f18 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -23,7 +23,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "bus_cdx_driver.h"
#include "cdx_logs.h"
@@ -121,7 +121,7 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ ret = dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
vfio_dev_fd);
if (ret < 0) {
CDX_BUS_ERR("Cannot release VFIO device");
@@ -155,7 +155,7 @@ cdx_vfio_unmap_resource_secondary(struct rte_cdx_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
+ ret = dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev->device.name,
vfio_dev_fd);
if (ret < 0) {
CDX_BUS_ERR("Cannot release VFIO device");
@@ -400,7 +400,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
if (rte_intr_fd_set(dev->intr_handle, -1))
return -1;
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -477,7 +477,7 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
cdx_vfio_find_and_unmap_resource(vfio_res_list, dev);
rte_free(vfio_res);
err_vfio_dev_fd:
- rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
return -1;
}
@@ -509,7 +509,7 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -535,7 +535,7 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return 0;
err_vfio_dev_fd:
- rte_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
+ dev_vfio_release_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, vfio_dev_fd);
return -1;
}
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index 3626b12316..d55c414ead 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -17,7 +17,7 @@
#include <rte_memcpy.h>
#include <ethdev_driver.h>
#include <rte_mbuf_dyn.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
#include <fslmc_vfio.h>
@@ -500,7 +500,7 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && rte_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 && dev_vfio_noiommu_is_enabled() == 0)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 8d39895bac..153e6112db 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -34,7 +34,7 @@
#include <rte_kvargs.h>
#include <dev_driver.h>
#include <rte_eal_memconfig.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
#include "fslmc_vfio.h"
@@ -204,7 +204,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (rte_vfio_noiommu_is_enabled() > 0)
+ if (dev_vfio_noiommu_is_enabled() > 0)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
@@ -374,7 +374,7 @@ fslmc_get_group_id(const char *group_name,
int ret;
/* get group number */
- ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
+ ret = dev_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
if (ret <= 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
@@ -411,7 +411,7 @@ fslmc_vfio_open_group_fd(const char *group_name)
/* if primary, try to open the group */
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
/* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
goto add_vfio_group;
@@ -509,10 +509,10 @@ fslmc_vfio_open_container_fd(void)
/* if we're in a primary process, try to open the container */
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
if (vfio_container_fd < 0) {
DPAA2_BUS_ERR("Open VFIO container(%s), err(%d)",
- RTE_VFIO_CONTAINER_PATH, vfio_container_fd);
+ DEV_VFIO_CONTAINER_PATH, vfio_container_fd);
ret = vfio_container_fd;
goto err_exit;
}
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index 9aae0a5d14..af37a65c45 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -11,7 +11,7 @@
#include <rte_malloc.h>
#include <rte_devargs.h>
#include <rte_memcpy.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include "eal_filesystem.h"
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (rte_vfio_noiommu_is_enabled() == 1)
+ if (dev_vfio_noiommu_is_enabled() == 1)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index f3791cce98..71afc383e6 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -18,7 +18,7 @@
#include <rte_bus_pci.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
@@ -29,10 +29,10 @@
#include "pci_init.h"
#include "private.h"
-static struct rte_tailq_elem rte_vfio_tailq = {
+static struct rte_tailq_elem dev_vfio_tailq = {
.name = "VFIO_RESOURCE_LIST",
};
-EAL_REGISTER_TAILQ(rte_vfio_tailq)
+EAL_REGISTER_TAILQ(dev_vfio_tailq)
static int
pci_vfio_get_region(const struct rte_pci_device *dev, int index,
@@ -418,7 +418,7 @@ pci_vfio_is_ioport_bar(const struct rte_pci_device *dev, int vfio_dev_fd,
}
static int
-pci_rte_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
+pci_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
{
if (pci_vfio_setup_interrupts(dev, vfio_dev_fd) != 0) {
PCI_LOG(ERR, "Error setting up interrupts!");
@@ -738,7 +738,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
int i, j, ret;
struct mapped_pci_resource *vfio_res = NULL;
struct mapped_pci_res_list *vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
struct pci_map *maps;
@@ -752,7 +752,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -897,7 +897,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
free(reg);
}
- if (pci_rte_vfio_setup_device(dev, vfio_dev_fd) < 0) {
+ if (pci_vfio_setup_device(dev, vfio_dev_fd) < 0) {
PCI_LOG(ERR, "%s setup device failed", pci_addr);
goto err_map;
}
@@ -920,7 +920,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
err_vfio_res:
rte_free(vfio_res);
err_vfio_dev_fd:
- rte_vfio_release_device(rte_pci_get_sysfs_path(),
+ dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
}
@@ -935,7 +935,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
int j, ret, i = 0;
struct mapped_pci_resource *vfio_res = NULL;
struct mapped_pci_res_list *vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
struct pci_map *maps;
@@ -961,7 +961,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info);
if (ret)
return ret;
@@ -1006,7 +1006,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
err_vfio_dev:
- rte_vfio_release_device(rte_pci_get_sysfs_path(),
+ dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
}
@@ -1101,7 +1101,7 @@ pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
return -1;
}
- ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
vfio_dev_fd);
if (ret < 0) {
PCI_LOG(ERR, "Cannot release VFIO device");
@@ -1109,7 +1109,7 @@ pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
}
vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
/* if we haven't found our tailq entry, something's wrong */
@@ -1140,7 +1140,7 @@ pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
if (vfio_dev_fd < 0)
return -1;
- ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+ ret = dev_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
vfio_dev_fd);
if (ret < 0) {
PCI_LOG(ERR, "Cannot release VFIO device");
@@ -1148,7 +1148,7 @@ pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
}
vfio_res_list =
- RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+ RTE_TAILQ_CAST(dev_vfio_tailq.head, mapped_pci_res_list);
vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
/* if we haven't found our tailq entry, something's wrong */
@@ -1195,7 +1195,7 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (rte_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
+ if (dev_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
&vfio_dev_fd, &device_info) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
@@ -1300,11 +1300,11 @@ pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar,
int
pci_vfio_is_enabled(void)
{
- int status = rte_vfio_is_enabled("vfio_pci");
+ int status = dev_vfio_is_enabled("vfio_pci");
if (!status) {
- rte_vfio_enable();
- status = rte_vfio_is_enabled("vfio_pci");
+ dev_vfio_enable();
+ status = dev_vfio_is_enabled("vfio_pci");
}
return status;
}
diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index dc8db80d3b..fcc4495b92 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -25,7 +25,7 @@
#include <rte_string_fns.h>
#include <rte_common.h>
#include <rte_devargs.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_tailq.h>
#include "private.h"
@@ -487,8 +487,8 @@ pci_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
* try fallback to VFIO.
*/
if (pdev->kdrv == RTE_PCI_KDRV_VFIO)
- return rte_vfio_container_dma_map
- (RTE_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
+ return dev_vfio_container_dma_map
+ (DEV_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
iova, len);
rte_errno = ENOTSUP;
return -1;
@@ -507,8 +507,8 @@ pci_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
* try fallback to VFIO.
*/
if (pdev->kdrv == RTE_PCI_KDRV_VFIO)
- return rte_vfio_container_dma_unmap
- (RTE_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
+ return dev_vfio_container_dma_unmap
+ (DEV_VFIO_DEFAULT_CONTAINER_FD, (uintptr_t)addr,
iova, len);
rte_errno = ENOTSUP;
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 90d865a8df..9d7df9be80 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -23,7 +23,7 @@
#include <rte_log.h>
#include <rte_memory.h>
#include <rte_string_fns.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "private.h"
@@ -282,7 +282,7 @@ static void
device_cleanup(struct rte_platform_device *pdev)
{
device_unmap_resources(pdev);
- rte_vfio_release_device(PLATFORM_BUS_DEVICES_PATH, pdev->name, pdev->dev_fd);
+ dev_vfio_release_device(PLATFORM_BUS_DEVICES_PATH, pdev->name, pdev->dev_fd);
}
static int
@@ -292,7 +292,7 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = rte_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
@@ -458,7 +458,7 @@ platform_bus_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t l
if (pdrv->dma_map != NULL)
return pdrv->dma_map(pdev, addr, iova, len);
- return rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova, len);
+ return dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova, len);
}
static int
@@ -470,7 +470,7 @@ platform_bus_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, size_t
if (pdrv->dma_unmap != NULL)
return pdrv->dma_unmap(pdev, addr, iova, len);
- return rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova,
+ return dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD, (uint64_t)addr, iova,
len);
}
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index e7f7ed994c..053af0d227 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,7 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "bcmfs_device.h"
#include "bcmfs_logs.h"
@@ -25,7 +25,7 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = rte_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = dev_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
@@ -51,7 +51,7 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
return 0;
map_failed:
- rte_vfio_release_device(path, dev_obj, *dev_fd);
+ dev_vfio_release_device(path, dev_obj, *dev_fd);
return ret;
}
@@ -87,7 +87,7 @@ bcmfs_release_vfio(struct bcmfs_device *dev)
/* unmap the addr */
munmap(dev->mmap_addr, dev->mmap_size);
/* release the device */
- ret = rte_vfio_release_device(dev->dirname, dev->name,
+ ret = dev_vfio_release_device(dev->dirname, dev->name,
dev->vfio_dev_fd);
if (ret < 0) {
BCMFS_LOG(ERR, "cannot release device");
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index d09a8f7e7d..6a6cf44752 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -2,7 +2,7 @@
* Copyright(c) 2025 Huawei Technologies Co., Ltd
*/
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "hinic3_compat.h"
#include "hinic3_cmd.h"
@@ -78,7 +78,7 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && rte_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && dev_vfio_noiommu_is_enabled() != 1;
}
int
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index ec6840d60e..f1522772f5 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -4,7 +4,7 @@
#include "nbl_userdev.h"
#include <uapi/linux/vfio.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#define NBL_USERDEV_EVENT_CLB_NAME "nbl_userspace_mem_event_clb"
#define NBL_USERDEV_BAR0_SIZE 65536
@@ -405,7 +405,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
snprintf(pathname, sizeof(pathname),
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
- ret = rte_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
+ ret = dev_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
if (ret <= 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
@@ -414,7 +414,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
NBL_LOG(DEBUG, "nbl vfio group number %d", common->iommu_group_num);
/* vfio_container */
if (nbl_default_container < 0) {
- container = rte_vfio_container_create();
+ container = dev_vfio_container_create();
container_create = 1;
if (container < 0) {
@@ -424,7 +424,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = rte_vfio_container_group_bind(container, common->iommu_group_num);
+ vfio_group_fd = dev_vfio_container_group_bind(container, common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
@@ -535,10 +535,10 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
- rte_vfio_container_destroy(container);
+ dev_vfio_container_destroy(container);
return -1;
}
@@ -550,7 +550,7 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
rte_mcfg_mem_read_lock();
TAILQ_REMOVE(&nbl_adapter_list, adapter, next);
close(common->devfd);
- vfio_group_fd = rte_vfio_container_group_bind(nbl_default_container,
+ vfio_group_fd = dev_vfio_container_group_bind(nbl_default_container,
common->iommu_group_num);
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
@@ -558,7 +558,7 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = rte_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
+ ret = dev_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
if (ret)
NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
if (!nbl_group_count) {
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index 7cc90a7a5b..d9744a7d05 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -7,7 +7,7 @@
#include <rte_eal.h>
#include <rte_dev.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_ethdev.h>
#include <rte_bus_pci.h>
#include <ethdev_pci.h>
@@ -2690,12 +2690,12 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_rte_vfio_no_io_mmu_enabled = rte_vfio_noiommu_is_enabled();
- NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_rte_vfio_no_io_mmu_enabled);
+ const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_noiommu_is_enabled();
+ NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_dev_vfio_no_io_mmu_enabled);
- if (n_rte_vfio_no_io_mmu_enabled) {
+ if (n_dev_vfio_no_io_mmu_enabled) {
NT_LOG(ERR, NTNIC, "vfio_no_iommu_enabled=%d: this PMD needs VFIO IOMMU",
- n_rte_vfio_no_io_mmu_enabled);
+ n_dev_vfio_no_io_mmu_enabled);
return -1;
}
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 439468b3a2..8489a59859 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -6,7 +6,7 @@
#include <rte_malloc.h>
#include <rte_memory.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_dev.h>
#include <rte_bus_pci.h>
#include <rte_spinlock.h>
@@ -70,16 +70,16 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
if (ret <= 0)
return -1;
if (vf_num == 0) {
/* use default container for pf0 */
- vfio->container_fd = RTE_VFIO_DEFAULT_CONTAINER_FD;
+ vfio->container_fd = DEV_VFIO_DEFAULT_CONTAINER_FD;
} else {
- vfio->container_fd = rte_vfio_container_create();
+ vfio->container_fd = dev_vfio_container_create();
if (vfio->container_fd < 0) {
NT_LOG(ERR, NTNIC,
@@ -88,7 +88,7 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
}
- vfio->group_fd = rte_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
+ vfio->group_fd = dev_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
if (vfio->group_fd < 0) {
NT_LOG(ERR, NTNIC,
@@ -115,8 +115,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
err:
- if (vfio->container_fd != RTE_VFIO_DEFAULT_CONTAINER_FD)
- rte_vfio_container_destroy(vfio->container_fd);
+ if (vfio->container_fd != DEV_VFIO_DEFAULT_CONTAINER_FD)
+ dev_vfio_container_destroy(vfio->container_fd);
return -1;
}
@@ -135,7 +135,7 @@ nthw_vfio_remove(int vf_num)
return -1;
}
- rte_vfio_container_destroy(vfio->container_fd);
+ dev_vfio_container_destroy(vfio->container_fd);
return 0;
}
@@ -169,14 +169,14 @@ nthw_vfio_dma_map(int vf_num, void *virt_addr, uint64_t *iova_addr, uint64_t siz
vf_num, virt_addr, rte_malloc_virt2iova(virt_addr), gp_virt_base, vfio->iova_addr,
size);
- int res = rte_vfio_container_dma_map(vfio->container_fd, gp_virt_base, vfio->iova_addr,
+ int res = dev_vfio_container_dma_map(vfio->container_fd, gp_virt_base, vfio->iova_addr,
size);
NT_LOG(DBG, NTNIC, "VFIO MMAP res %i, container_fd %i, vf_num %i", res,
vfio->container_fd, vf_num);
if (res) {
- NT_LOG(ERR, NTNIC, "rte_vfio_container_dma_map failed: res %d", res);
+ NT_LOG(ERR, NTNIC, "dev_vfio_container_dma_map failed: res %d", res);
return -1;
}
@@ -210,7 +210,7 @@ nthw_vfio_dma_unmap(int vf_num, void *virt_addr, uint64_t iova_addr, uint64_t si
return -1;
}
- int res = rte_vfio_container_dma_unmap(vfio->container_fd, gp_virt_base, iova_addr, size);
+ int res = dev_vfio_container_dma_unmap(vfio->container_fd, gp_virt_base, iova_addr, size);
if (res != 0) {
NT_LOG(ERR, NTNIC,
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index f319d455ba..36d06f1e3f 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -20,7 +20,7 @@
#include <rte_vhost.h>
#include <rte_vdpa.h>
#include <vdpa_driver.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_spinlock.h>
#include <rte_log.h>
#include <rte_kvargs.h>
@@ -182,18 +182,18 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
&iommu_group_num);
if (ret <= 0) {
DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
return -1;
}
- internal->vfio_container_fd = rte_vfio_container_create();
+ internal->vfio_container_fd = dev_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = rte_vfio_container_group_bind(
+ internal->vfio_group_fd = dev_vfio_container_group_bind(
internal->vfio_container_fd, iommu_group_num);
if (internal->vfio_group_fd < 0)
goto err;
@@ -216,7 +216,7 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
return 0;
err:
- rte_vfio_container_destroy(internal->vfio_container_fd);
+ dev_vfio_container_destroy(internal->vfio_container_fd);
return -1;
}
@@ -246,7 +246,7 @@ ifcvf_dma_map(struct ifcvf_internal *internal, bool do_map)
reg->host_user_addr, reg->guest_phys_addr, reg->size);
if (do_map) {
- ret = rte_vfio_container_dma_map(vfio_container_fd,
+ ret = dev_vfio_container_dma_map(vfio_container_fd,
reg->host_user_addr, reg->guest_phys_addr,
reg->size);
if (ret < 0) {
@@ -254,7 +254,7 @@ ifcvf_dma_map(struct ifcvf_internal *internal, bool do_map)
goto exit;
}
} else {
- ret = rte_vfio_container_dma_unmap(vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(vfio_container_fd,
reg->host_user_addr, reg->guest_phys_addr,
reg->size);
if (ret < 0) {
@@ -388,7 +388,7 @@ vdpa_ifcvf_stop(struct ifcvf_internal *internal)
if (RTE_VHOST_NEED_LOG(features)) {
ifcvf_disable_logging(hw);
rte_vhost_get_log_base(internal->vid, &log_base, &log_size);
- rte_vfio_container_dma_unmap(internal->vfio_container_fd,
+ dev_vfio_container_dma_unmap(internal->vfio_container_fd,
log_base, IFCVF_LOG_BASE, log_size);
/*
* IFCVF marks dirty memory pages for only packet buffer,
@@ -791,7 +791,7 @@ m_ifcvf_start(struct ifcvf_internal *internal)
vring_init(&internal->m_vring[i], vq.size, vring_buf,
rte_mem_page_size());
- ret = rte_vfio_container_dma_map(internal->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(internal->vfio_container_fd,
(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
if (ret < 0) {
DRV_LOG(ERR, "mediated vring DMA map failed.");
@@ -900,7 +900,7 @@ m_ifcvf_stop(struct ifcvf_internal *internal)
size = RTE_ALIGN_CEIL(vring_size(vq.size, rte_mem_page_size()),
rte_mem_page_size());
- rte_vfio_container_dma_unmap(internal->vfio_container_fd,
+ dev_vfio_container_dma_unmap(internal->vfio_container_fd,
(uint64_t)(uintptr_t)internal->m_vring[i].desc,
m_vring_iova, size);
@@ -1205,7 +1205,7 @@ ifcvf_set_features(int vid)
ifcvf_sw_fallback_switchover(internal);
} else {
rte_vhost_get_log_base(vid, &log_base, &log_size);
- rte_vfio_container_dma_map(internal->vfio_container_fd,
+ dev_vfio_container_dma_map(internal->vfio_container_fd,
log_base, IFCVF_LOG_BASE, log_size);
ifcvf_enable_logging(&internal->hw, IFCVF_LOG_BASE, log_size);
}
@@ -1824,7 +1824,7 @@ ifcvf_pci_remove(struct rte_pci_device *pci_dev)
DRV_LOG(ERR, "failed to update datapath %s", pci_dev->name);
rte_pci_unmap_device(internal->pdev);
- rte_vfio_container_destroy(internal->vfio_container_fd);
+ dev_vfio_container_destroy(internal->vfio_container_fd);
rte_vdpa_unregister_device(internal->vdev);
pthread_mutex_lock(&internal_list_lock);
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 4c0ab3ff30..94740e768d 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -12,7 +12,7 @@
#include <nfp_common_pci.h>
#include <nfp_dev.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal_paging.h>
#include <rte_malloc.h>
#include <vdpa_driver.h>
@@ -120,16 +120,16 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
+ ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
&device->iommu_group);
if (ret <= 0)
return -1;
- device->vfio_container_fd = rte_vfio_container_create();
+ device->vfio_container_fd = dev_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = rte_vfio_container_group_bind(
+ device->vfio_group_fd = dev_vfio_container_group_bind(
device->vfio_container_fd, device->iommu_group);
if (device->vfio_group_fd < 0)
goto container_destroy;
@@ -146,9 +146,9 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
return 0;
group_unbind:
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
+ dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
- rte_vfio_container_destroy(device->vfio_container_fd);
+ dev_vfio_container_destroy(device->vfio_container_fd);
return -1;
}
@@ -157,8 +157,8 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
- rte_vfio_container_destroy(device->vfio_container_fd);
+ dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
+ dev_vfio_container_destroy(device->vfio_container_fd);
}
static int
@@ -173,7 +173,7 @@ nfp_vdpa_dma_do_unmap(struct rte_vhost_memory *mem,
for (i = 0; i < times; i++) {
region = &mem->regions[i];
- ret = rte_vfio_container_dma_unmap(vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(vfio_container_fd,
region->host_user_addr, region->guest_phys_addr,
region->size);
if (ret < 0) {
@@ -197,7 +197,7 @@ nfp_vdpa_dma_do_map(struct rte_vhost_memory *mem,
for (i = 0; i < times; i++) {
region = &mem->regions[i];
- ret = rte_vfio_container_dma_map(vfio_container_fd,
+ ret = dev_vfio_container_dma_map(vfio_container_fd,
region->host_user_addr, region->guest_phys_addr,
region->size);
if (ret < 0) {
@@ -282,7 +282,7 @@ nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
rte_mem_page_size());
- rte_vfio_container_dma_unmap(device->vfio_container_fd,
+ dev_vfio_container_dma_unmap(device->vfio_container_fd,
(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
m_vring_iova, size);
@@ -315,7 +315,7 @@ nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
- ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(device->vfio_container_fd,
(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
if (ret != 0) {
DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index eda111954f..c65f716a15 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -12,7 +12,7 @@
#include <rte_errno.h>
#include <rte_kvargs.h>
#include <rte_string_fns.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_vhost.h>
#include "efx.h"
@@ -74,13 +74,13 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
rte_pci_device_name(&dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- sva->vfio_container_fd = rte_vfio_container_create();
+ sva->vfio_container_fd = dev_vfio_container_create();
if (sva->vfio_container_fd < 0) {
sfc_vdpa_err(sva, "failed to create VFIO container");
goto fail_container_create;
}
- rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
+ rc = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
&sva->iommu_group_num);
if (rc <= 0) {
sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
@@ -89,7 +89,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
}
sva->vfio_group_fd =
- rte_vfio_container_group_bind(sva->vfio_container_fd,
+ dev_vfio_container_group_bind(sva->vfio_container_fd,
sva->iommu_group_num);
if (sva->vfio_group_fd < 0) {
sfc_vdpa_err(sva,
@@ -109,7 +109,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
+ if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
sva->iommu_group_num) != 0) {
sfc_vdpa_err(sva,
"failed to unbind IOMMU group %d from container %d",
@@ -118,7 +118,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
fail_group_bind:
fail_get_group_num:
- if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
+ if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
}
@@ -132,14 +132,14 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (rte_vfio_container_group_unbind(sva->vfio_container_fd,
+ if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
sva->iommu_group_num) != 0) {
sfc_vdpa_err(sva,
"failed to unbind IOMMU group %d from container %d",
sva->iommu_group_num, sva->vfio_container_fd);
}
- if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) {
+ if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
sva->vfio_container_fd);
diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c
index 7e43719f53..65050b9640 100644
--- a/drivers/vdpa/sfc/sfc_vdpa_hw.c
+++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c
@@ -6,7 +6,7 @@
#include <rte_common.h>
#include <rte_errno.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_vhost.h>
#include "efx.h"
@@ -64,7 +64,7 @@ sfc_vdpa_dma_alloc(struct sfc_vdpa_adapter *sva, const char *name,
mcdi_iova = SFC_VDPA_DEFAULT_MCDI_IOVA;
for (;;) {
- ret = rte_vfio_container_dma_map(sva->vfio_container_fd,
+ ret = dev_vfio_container_dma_map(sva->vfio_container_fd,
(uint64_t)mz->addr, mcdi_iova,
mcdi_buff_size);
if (ret == 0)
@@ -98,7 +98,7 @@ sfc_vdpa_dma_free(struct sfc_vdpa_adapter *sva, efsys_mem_t *esmp)
sfc_vdpa_log_init(sva, "name=%s", esmp->esm_mz->name);
- ret = rte_vfio_container_dma_unmap(sva->vfio_container_fd,
+ ret = dev_vfio_container_dma_unmap(sva->vfio_container_fd,
(uint64_t)esmp->esm_base,
esmp->esm_addr, sva->mcdi_buff_size);
if (ret < 0)
@@ -140,7 +140,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
mem_reg = &vhost_mem->regions[i];
if (do_map) {
- rc = rte_vfio_container_dma_map(vfio_container_fd,
+ rc = dev_vfio_container_dma_map(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
@@ -151,7 +151,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
goto failed_vfio_dma_map;
}
} else {
- rc = rte_vfio_container_dma_unmap(vfio_container_fd,
+ rc = dev_vfio_container_dma_unmap(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
@@ -171,7 +171,7 @@ sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map)
failed_vfio_dma_map:
for (j = 0; j < i; j++) {
mem_reg = &vhost_mem->regions[j];
- rte_vfio_container_dma_unmap(vfio_container_fd,
+ dev_vfio_container_dma_unmap(vfio_container_fd,
mem_reg->host_user_addr,
mem_reg->guest_phys_addr,
mem_reg->size);
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 52d265769e..e35aae2540 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -41,7 +41,7 @@
#include <rte_dev.h>
#include <rte_devargs.h>
#include <rte_version.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <malloc_heap.h>
#include <telemetry_internal.h>
@@ -819,8 +819,9 @@ rte_eal_vfio_get_vf_token(__rte_unused rte_uuid_t vf_token)
{
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
-int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
+int
+dev_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *vfio_dev_fd,
__rte_unused struct vfio_device_info *device_info)
@@ -829,8 +830,9 @@ int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
-int rte_vfio_release_device(__rte_unused const char *sysfs_base,
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
+int
+dev_vfio_release_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int fd)
{
@@ -838,40 +840,45 @@ int rte_vfio_release_device(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
-int rte_vfio_enable(void)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
+int
+dev_vfio_enable(void)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
-void rte_vfio_cleanup(void)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
+void
+dev_vfio_cleanup(void)
{
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
-int rte_vfio_is_enabled(__rte_unused const char *modname)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
+int
+dev_vfio_is_enabled(__rte_unused const char *modname)
{
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
-int rte_vfio_noiommu_is_enabled(void)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
+int
+dev_vfio_noiommu_is_enabled(void)
{
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
-int rte_vfio_clear_group(__rte_unused int vfio_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
+int
+dev_vfio_clear_group(__rte_unused int vfio_group_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
+dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
__rte_unused int *iommu_group_num)
{
@@ -879,59 +886,59 @@ rte_vfio_get_group_num(__rte_unused const char *sysfs_base,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
-rte_vfio_get_container_fd(void)
+dev_vfio_get_container_fd(void)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(__rte_unused int iommu_group_num)
+dev_vfio_get_group_fd(__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
-rte_vfio_container_create(void)
+dev_vfio_container_create(void)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
-rte_vfio_container_destroy(__rte_unused int container_fd)
+dev_vfio_container_destroy(__rte_unused int container_fd)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
-rte_vfio_container_group_bind(__rte_unused int container_fd,
+dev_vfio_container_group_bind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
int
-rte_vfio_container_group_unbind(__rte_unused int container_fd,
+dev_vfio_container_group_unbind(__rte_unused int container_fd,
__rte_unused int iommu_group_num)
{
rte_errno = ENOTSUP;
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(__rte_unused int container_fd,
+dev_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
@@ -940,9 +947,9 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(__rte_unused int container_fd,
+dev_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/dev_vfio.h
similarity index 82%
rename from lib/eal/include/rte_vfio.h
rename to lib/eal/include/dev_vfio.h
index b53ce56a8d..d0f741061e 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -2,14 +2,14 @@
* Copyright(c) 2017 6WIND S.A.
*/
-#ifndef _RTE_VFIO_H_
-#define _RTE_VFIO_H_
+#ifndef _DEV_VFIO_H_
+#define _DEV_VFIO_H_
/**
* @file
* @internal
*
- * RTE VFIO internal API.
+ * VFIO device API.
*
* This library provides VFIO related utility functions for use by drivers.
*/
@@ -25,11 +25,11 @@ extern "C" {
#ifdef RTE_EXEC_ENV_LINUX
-#define RTE_VFIO_DIR "/dev/vfio"
-#define RTE_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
-#define RTE_VFIO_GROUP_FMT "/dev/vfio/%u"
-#define RTE_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define RTE_VFIO_NOIOMMU_MODE \
+#define DEV_VFIO_DIR "/dev/vfio"
+#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
+#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
+#define DEV_VFIO_NOIOMMU_MODE \
"/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -37,7 +37,7 @@ extern "C" {
/* we don't need an actual definition, only pointer is used */
struct vfio_device_info;
-#define RTE_VFIO_DEFAULT_CONTAINER_FD (-1)
+#define DEV_VFIO_DEFAULT_CONTAINER_FD (-1)
/**
* @internal
@@ -66,7 +66,7 @@ struct vfio_device_info;
* >1 if the device cannot be managed this way.
*/
__rte_internal
-int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
+int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
@@ -90,7 +90,7 @@ int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
* <0 on failure.
*/
__rte_internal
-int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
+int dev_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
/**
* @internal
@@ -104,7 +104,7 @@ int rte_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* <0 on failure.
*/
__rte_internal
-int rte_vfio_enable(void);
+int dev_vfio_enable(void);
/**
* @internal
@@ -113,7 +113,7 @@ int rte_vfio_enable(void);
* This function is only relevant to Linux.
*/
__rte_internal
-void rte_vfio_cleanup(void);
+void dev_vfio_cleanup(void);
/**
* @internal
@@ -129,7 +129,7 @@ void rte_vfio_cleanup(void);
* 0 otherwise.
*/
__rte_internal
-int rte_vfio_is_enabled(const char *modname);
+int dev_vfio_is_enabled(const char *modname);
/**
* @internal
@@ -143,7 +143,7 @@ int rte_vfio_is_enabled(const char *modname);
* <0 for errors.
*/
__rte_internal
-int rte_vfio_noiommu_is_enabled(void);
+int dev_vfio_noiommu_is_enabled(void);
/**
* @internal
@@ -161,7 +161,7 @@ int rte_vfio_noiommu_is_enabled(void);
*/
__rte_internal
int
-rte_vfio_clear_group(int vfio_group_fd);
+dev_vfio_clear_group(int vfio_group_fd);
/**
* @internal
@@ -186,7 +186,7 @@ rte_vfio_clear_group(int vfio_group_fd);
*/
__rte_internal
int
-rte_vfio_get_group_num(const char *sysfs_base,
+dev_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num);
/**
@@ -213,7 +213,7 @@ rte_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
+dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info);
/**
@@ -229,7 +229,7 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
*/
__rte_internal
int
-rte_vfio_get_container_fd(void);
+dev_vfio_get_container_fd(void);
/**
* @internal
@@ -247,7 +247,7 @@ rte_vfio_get_container_fd(void);
*/
__rte_internal
int
-rte_vfio_get_group_fd(int iommu_group_num);
+dev_vfio_get_group_fd(int iommu_group_num);
/**
* @internal
@@ -267,7 +267,7 @@ rte_vfio_get_group_fd(int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_create(void);
+dev_vfio_container_create(void);
/**
* @internal
@@ -282,7 +282,7 @@ rte_vfio_container_create(void);
*/
__rte_internal
int
-rte_vfio_container_destroy(int container_fd);
+dev_vfio_container_destroy(int container_fd);
/**
* @internal
@@ -300,7 +300,7 @@ rte_vfio_container_destroy(int container_fd);
*/
__rte_internal
int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
+dev_vfio_container_group_bind(int container_fd, int iommu_group_num);
/**
* @internal
@@ -318,14 +318,14 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
+dev_vfio_container_group_unbind(int container_fd, int iommu_group_num);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
* @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
+ * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
* use the default container.
*
* @param vaddr
@@ -343,7 +343,7 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num);
*/
__rte_internal
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
/**
@@ -351,7 +351,7 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* Perform DMA unmapping for devices in a container.
*
* @param container_fd
- * the specified container fd. Use RTE_VFIO_DEFAULT_CONTAINER_FD to
+ * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
* use the default container.
*
* @param vaddr
@@ -369,11 +369,11 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr,
*/
__rte_internal
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr,
uint64_t iova, uint64_t len);
#ifdef __cplusplus
}
#endif
-#endif /* _RTE_VFIO_H_ */
+#endif /* _DEV_VFIO_H_ */
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index 9282143918..8d3a6d5e00 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -60,7 +60,7 @@ headers += files(
driver_sdk_headers = files(
'bus_driver.h',
'dev_driver.h',
- 'rte_vfio.h',
+ 'dev_vfio.h',
)
# special case install the generic headers, since they go in a subdir
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 17977baedc..d63edb4192 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -41,7 +41,7 @@
#include <rte_bus.h>
#include <rte_version.h>
#include <malloc_heap.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <telemetry_internal.h>
#include <eal_export.h>
@@ -771,7 +771,7 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (rte_vfio_enable()) {
+ if (dev_vfio_enable()) {
rte_eal_init_alert("Cannot init VFIO");
rte_errno = EAGAIN;
goto err_out;
@@ -985,7 +985,7 @@ rte_eal_cleanup(void)
rte_service_finalize();
eal_bus_cleanup();
- rte_vfio_cleanup();
+ dev_vfio_cleanup();
rte_mp_channel_cleanup();
rte_eal_alarm_cleanup();
rte_trace_save();
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index da46ef2f17..ea7e028cfa 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -15,7 +15,7 @@
#include <rte_log.h>
#include <rte_memory.h>
#include <rte_eal_memconfig.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <eal_export.h>
#include "eal_filesystem.h"
@@ -367,7 +367,7 @@ vfio_open_group_fd(int iommu_group_num, bool mp_request)
/* if not requesting via mp, open the group locally */
if (!mp_request) {
/* try regular group format */
- snprintf(filename, sizeof(filename), RTE_VFIO_GROUP_FMT, iommu_group_num);
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
if (vfio_group_fd < 0) {
/* if file not found, it's not an error */
@@ -378,7 +378,7 @@ vfio_open_group_fd(int iommu_group_num, bool mp_request)
}
/* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), RTE_VFIO_NOIOMMU_GROUP_FMT,
+ snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT,
iommu_group_num);
vfio_group_fd = open(filename, O_RDWR);
if (vfio_group_fd < 0) {
@@ -525,7 +525,7 @@ get_vfio_cfg_by_container_fd(int container_fd)
{
unsigned int i;
- if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD)
+ if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD)
return default_vfio_cfg;
for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
@@ -536,9 +536,9 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-rte_vfio_get_group_fd(int iommu_group_num)
+dev_vfio_get_group_fd(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -694,7 +694,7 @@ vfio_sync_default_container(void)
if (rte_eal_process_type() != RTE_PROC_SECONDARY)
return -1;
- /* default container fd should have been opened in rte_vfio_enable() */
+ /* default container fd should have been opened in dev_vfio_enable() */
if (!vfio_enabled ||
default_vfio_cfg->vfio_container_fd < 0) {
EAL_LOG(ERR, "VFIO support is not initialized");
@@ -740,9 +740,9 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_clear_group)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
int
-rte_vfio_clear_group(int vfio_group_fd)
+dev_vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -769,9 +769,9 @@ rte_vfio_clear_group(int vfio_group_fd)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_setup_device)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
-rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
+dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
{
struct vfio_group_status group_status = {
@@ -793,7 +793,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
}
/* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
if (ret == 0) {
EAL_LOG(NOTICE,
"%s not managed by VFIO driver, skipping",
@@ -806,7 +806,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -832,14 +832,14 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -860,7 +860,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -884,7 +884,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -901,7 +901,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -970,7 +970,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -1007,7 +1007,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1020,7 +1020,7 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
strerror(errno));
close(*vfio_dev_fd);
close(vfio_group_fd);
- rte_vfio_clear_group(vfio_group_fd);
+ dev_vfio_clear_group(vfio_group_fd);
return -1;
}
vfio_group_device_get(vfio_group_fd);
@@ -1028,9 +1028,9 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_release_device)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
int
-rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
+dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
int vfio_dev_fd)
{
struct vfio_config *vfio_cfg;
@@ -1050,7 +1050,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
rte_mcfg_mem_read_lock();
/* get group number */
- ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
if (ret <= 0) {
EAL_LOG(WARNING, "%s not managed by VFIO driver",
dev_addr);
@@ -1060,9 +1060,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = rte_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "rte_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "dev_vfio_get_group_fd failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1098,7 +1098,7 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (rte_vfio_clear_group(vfio_group_fd) < 0) {
+ if (dev_vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -1122,9 +1122,9 @@ rte_vfio_release_device(const char *sysfs_base, const char *dev_addr,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_enable)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
int
-rte_vfio_enable(void)
+dev_vfio_enable(void)
{
/* initialize group list */
unsigned int i, j;
@@ -1171,7 +1171,7 @@ rte_vfio_enable(void)
}
/* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(RTE_VFIO_DIR);
+ dir = opendir(DEV_VFIO_DIR);
if (dir == NULL) {
EAL_LOG(DEBUG,
"VFIO directory does not exist, skipping VFIO support...");
@@ -1204,9 +1204,9 @@ rte_vfio_enable(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
-rte_vfio_is_enabled(const char *modname)
+dev_vfio_is_enabled(const char *modname)
{
const int mod_available = rte_eal_check_module(modname) > 0;
return vfio_enabled && mod_available;
@@ -1247,9 +1247,9 @@ vfio_set_iommu_type(int vfio_container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_device_info)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
int
-rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
+dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
@@ -1263,7 +1263,7 @@ rte_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
return -1;
if (*vfio_dev_fd == 0) {
- ret = rte_vfio_setup_device(sysfs_base, dev_addr,
+ ret = dev_vfio_setup_device(sysfs_base, dev_addr,
vfio_dev_fd, device_info);
if (ret)
return -1;
@@ -1331,10 +1331,10 @@ vfio_open_container_fd(bool mp_request)
/* if not requesting via mp, open a new container locally */
if (!mp_request) {
- vfio_container_fd = open(RTE_VFIO_CONTAINER_PATH, O_RDWR);
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
if (vfio_container_fd < 0) {
EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- RTE_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
return -1;
}
@@ -1386,12 +1386,12 @@ vfio_open_container_fd(bool mp_request)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_container_fd)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
-rte_vfio_get_container_fd(void)
+dev_vfio_get_container_fd(void)
{
/* Return the default container fd if VFIO is enabled.
- * The default container is set up during rte_vfio_enable().
+ * The default container is set up during dev_vfio_enable().
* This function does not create a new container.
*/
if (!vfio_enabled)
@@ -1400,9 +1400,9 @@ rte_vfio_get_container_fd(void)
return default_vfio_cfg->vfio_container_fd;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_get_group_num)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-rte_vfio_get_group_num(const char *sysfs_base,
+dev_vfio_get_group_num(const char *sysfs_base,
const char *dev_addr, int *iommu_group_num)
{
char linkname[PATH_MAX];
@@ -2076,15 +2076,15 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_noiommu_is_enabled)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
int
-rte_vfio_noiommu_is_enabled(void)
+dev_vfio_noiommu_is_enabled(void)
{
int fd;
ssize_t cnt;
char c;
- fd = open(RTE_VFIO_NOIOMMU_MODE, O_RDONLY);
+ fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
if (fd < 0) {
if (errno != ENOENT) {
EAL_LOG(ERR, "Cannot open VFIO noiommu file "
@@ -2109,9 +2109,9 @@ rte_vfio_noiommu_is_enabled(void)
return c == 'Y';
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_create)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
-rte_vfio_container_create(void)
+dev_vfio_container_create(void)
{
unsigned int i;
@@ -2142,9 +2142,9 @@ rte_vfio_container_create(void)
return vfio_cfgs[i].vfio_container_fd;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_destroy)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
-rte_vfio_container_destroy(int container_fd)
+dev_vfio_container_destroy(int container_fd)
{
struct vfio_config *vfio_cfg;
unsigned int i;
@@ -2154,7 +2154,7 @@ rte_vfio_container_destroy(int container_fd)
return -1;
}
- if (container_fd == RTE_VFIO_DEFAULT_CONTAINER_FD) {
+ if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD) {
EAL_LOG(ERR, "Cannot destroy default VFIO container");
return -1;
}
@@ -2167,7 +2167,7 @@ rte_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- rte_vfio_container_group_unbind(container_fd,
+ dev_vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2179,9 +2179,9 @@ rte_vfio_container_destroy(int container_fd)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_bind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
-rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
+dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2199,9 +2199,9 @@ rte_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_group_unbind)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
int
-rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+dev_vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
@@ -2245,9 +2245,9 @@ rte_vfio_container_group_unbind(int container_fd, int iommu_group_num)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct vfio_config *vfio_cfg;
@@ -2271,9 +2271,9 @@ rte_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
return container_dma_map(vfio_cfg, vaddr, iova, len);
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
struct vfio_config *vfio_cfg;
@@ -2345,9 +2345,9 @@ vfio_cleanup_config(struct vfio_config *vfio_cfg)
memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
-rte_vfio_cleanup(void)
+dev_vfio_cleanup(void)
{
unsigned int i;
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 22136f2e8b..ae1c8b40b9 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -8,7 +8,7 @@
#include <rte_errno.h>
#include <rte_log.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_eal.h>
#include "eal_private.h"
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = rte_vfio_get_group_fd(m->group_num);
+ fd = dev_vfio_get_group_fd(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
@@ -49,7 +49,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
break;
case SOCKET_REQ_CONTAINER:
r->req = SOCKET_REQ_CONTAINER;
- fd = rte_vfio_get_container_fd();
+ fd = dev_vfio_get_container_fd();
if (fd < 0)
r->result = SOCKET_ERR;
else {
diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
index b8771d4d85..d59ce9331f 100644
--- a/lib/eal/windows/eal.c
+++ b/lib/eal/windows/eal.c
@@ -24,7 +24,7 @@
#include <eal_options.h>
#include <eal_private.h>
#include <rte_service_component.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include "eal_firmware.h"
#include "eal_hugepages.h"
@@ -152,9 +152,9 @@ rte_eal_cleanup(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_cleanup)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
-rte_vfio_cleanup(void)
+dev_vfio_cleanup(void)
{
}
@@ -459,9 +459,9 @@ eal_asprintf(char **buffer, const char *format, ...)
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_map)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-rte_vfio_container_dma_map(__rte_unused int container_fd,
+dev_vfio_container_dma_map(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
@@ -470,9 +470,9 @@ rte_vfio_container_dma_map(__rte_unused int container_fd,
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(rte_vfio_container_dma_unmap)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-rte_vfio_container_dma_unmap(__rte_unused int container_fd,
+dev_vfio_container_dma_unmap(__rte_unused int container_fd,
__rte_unused uint64_t vaddr,
__rte_unused uint64_t iova,
__rte_unused uint64_t len)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 020c993b29..35aa70dffc 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -42,7 +42,7 @@
#include <rte_common.h>
#include <rte_malloc.h>
#include <rte_log.h>
-#include <rte_vfio.h>
+#include <dev_vfio.h>
#include <rte_errno.h>
#include "iotlb.h"
@@ -191,7 +191,7 @@ async_dma_map_region(struct virtio_net *dev, struct rte_vhost_mem_region *reg, b
continue;
if (do_map) {
- ret = rte_vfio_container_dma_map(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ ret = dev_vfio_container_dma_map(DEV_VFIO_DEFAULT_CONTAINER_FD,
page->host_user_addr,
page->host_iova,
page->size);
@@ -216,7 +216,7 @@ async_dma_map_region(struct virtio_net *dev, struct rte_vhost_mem_region *reg, b
return -1;
}
} else {
- ret = rte_vfio_container_dma_unmap(RTE_VFIO_DEFAULT_CONTAINER_FD,
+ ret = dev_vfio_container_dma_unmap(DEV_VFIO_DEFAULT_CONTAINER_FD,
page->host_user_addr,
page->host_iova,
page->size);
@@ -267,7 +267,7 @@ free_all_mem_regions(struct virtio_net *dev)
if (!dev || !dev->mem)
return;
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map(dev, false);
for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
@@ -1564,7 +1564,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
dev->mem->nregions++;
}
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map(dev, true);
if (vhost_user_postcopy_register(dev, main_fd, ctx) < 0)
@@ -1750,7 +1750,7 @@ vhost_user_add_mem_reg(struct virtio_net **pdev,
dev->mem->nregions++;
- if (dev->async_copy && rte_vfio_is_enabled("vfio")) {
+ if (dev->async_copy && dev_vfio_is_enabled("vfio")) {
if (async_dma_map_region(dev, reg, true) < 0)
goto free_new_region_no_dma;
}
@@ -1791,7 +1791,7 @@ vhost_user_add_mem_reg(struct virtio_net **pdev,
return RTE_VHOST_MSG_RESULT_OK;
free_new_region:
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map_region(dev, reg, false);
free_new_region_no_dma:
remove_guest_pages(dev, reg);
@@ -1827,7 +1827,7 @@ vhost_user_rem_mem_reg(struct virtio_net **pdev,
if (region->userspace_addr == current_region->guest_user_addr
&& region->guest_phys_addr == current_region->guest_phys_addr
&& region->memory_size == current_region->size) {
- if (dev->async_copy && rte_vfio_is_enabled("vfio"))
+ if (dev->async_copy && dev_vfio_is_enabled("vfio"))
async_dma_map_region(dev, current_region, false);
if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
vhost_user_iotlb_cache_remove(dev,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 08/25] bus/pci: rename mismatching error labels
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (6 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 07/25] vfio: rename API to reflect internal status Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 09/25] vfio: split get device info from setup Anatoly Burakov
` (17 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Currently, there are two VFIO map functions - primary and secondary -
that use a very similar goto error pattern. However, the label names are
confusing.
Specifically, for primary, we have three error labels: vfio_res (release
VFIO resource), vfio_dev_fd (close the fd), and map (unmap all the mapped
regions). For secondary, there is vfio_dev (close the fd), and
vfio_dev_fd (unmap all mapped regions).
For consistency, align secondary process function's error labels to match
semantics of how they work in primary process function.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci_vfio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 71afc383e6..3e87e759f5 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -968,7 +968,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
- goto err_vfio_dev;
+ goto err_vfio_dev_fd;
/* map BARs */
maps = vfio_res->maps;
@@ -979,14 +979,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
if (ret < 0) {
PCI_LOG(ERR, "%s sparse mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
} else {
ret = pci_vfio_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
if (ret < 0) {
PCI_LOG(ERR, "%s mapping BAR%i failed: %s",
pci_addr, i, strerror(errno));
- goto err_vfio_dev_fd;
+ goto err_map;
}
}
@@ -995,17 +995,17 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
/* we need save vfio_dev_fd, so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
if (rte_intr_dev_fd_set(dev->vfio_req_intr_handle, vfio_dev_fd))
- goto err_vfio_dev_fd;
+ goto err_map;
return 0;
-err_vfio_dev_fd:
+err_map:
for (j = 0; j < i; j++) {
if (maps[j].addr)
pci_unmap_resource(maps[j].addr, maps[j].size);
}
-err_vfio_dev:
+err_vfio_dev_fd:
dev_vfio_release_device(rte_pci_get_sysfs_path(),
pci_addr, vfio_dev_fd);
return -1;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 09/25] vfio: split get device info from setup
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (7 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 08/25] bus/pci: rename mismatching error labels Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 10/25] vfio: add container device assignment API Anatoly Burakov
` (16 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Nipun Gupta, Nikhil Agarwal, Chenbo Xia, Tomasz Duszynski,
Ajit Khaparde, Vikas Gupta, Bruce Richardson
Currently, setup gets device info as part of setup, while the separate get
device info API also calls setup if the fd is zero. Untangle these two APIs
and make each do one thing, and adjust all existing callers.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/cdx/cdx_vfio.c | 12 ++++++++--
drivers/bus/pci/linux/pci_vfio.c | 18 ++++++++++----
drivers/bus/platform/platform.c | 9 ++++++-
drivers/crypto/bcmfs/bcmfs_vfio.c | 8 ++++++-
lib/eal/freebsd/eal.c | 12 ++++++++--
lib/eal/include/dev_vfio.h | 23 +++++++-----------
lib/eal/linux/eal_vfio.c | 40 +++++++++----------------------
7 files changed, 68 insertions(+), 54 deletions(-)
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 02a34d6f18..531e4ccc8d 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -401,10 +401,14 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
return -1;
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* allocate vfio_res and get region info */
vfio_res = rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0);
if (vfio_res == NULL) {
@@ -510,10 +514,14 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
}
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
/* map MMIO regions */
maps = vfio_res->maps;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 3e87e759f5..0363a4a6c5 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -753,10 +753,14 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
loc->domain, loc->bus, loc->devid, loc->function);
ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd))
goto err_vfio_dev_fd;
@@ -962,10 +966,14 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
}
ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info);
+ &vfio_dev_fd);
if (ret)
return ret;
+ ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
+ if (ret)
+ goto err_vfio_dev_fd;
+
ret = pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
@@ -1195,12 +1203,14 @@ pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
if (vfio_dev_fd < 0) {
return -1;
} else if (vfio_dev_fd == 0) {
- if (dev_vfio_get_device_info(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd, &device_info) != 0)
+ if (dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
+ &vfio_dev_fd) != 0)
return -1;
/* save vfio_dev_fd so it can be used during release */
if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd) != 0)
return -1;
+ if (dev_vfio_get_device_info(vfio_dev_fd, &device_info) != 0)
+ return -1;
if (pci_vfio_fill_regions(dev, vfio_dev_fd, &device_info) != 0)
return -1;
diff --git a/drivers/bus/platform/platform.c b/drivers/bus/platform/platform.c
index 9d7df9be80..950306e2d9 100644
--- a/drivers/bus/platform/platform.c
+++ b/drivers/bus/platform/platform.c
@@ -292,12 +292,19 @@ device_setup(struct rte_platform_device *pdev)
const char *name = pdev->name;
int ret;
- ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd, &dev_info);
+ ret = dev_vfio_setup_device(PLATFORM_BUS_DEVICES_PATH, name, &pdev->dev_fd);
if (ret) {
PLATFORM_LOG_LINE(ERR, "failed to setup %s", name);
return -ENODEV;
}
+ ret = dev_vfio_get_device_info(pdev->dev_fd, &dev_info);
+ if (ret) {
+ PLATFORM_LOG_LINE(ERR, "failed to get device info for %s", name);
+ ret = -ENODEV;
+ goto out;
+ }
+
/* This is an extra check to confirm that platform device was initialized
* by a kernel vfio-platform driver. On kernels that predate vfio-platform
* driver this flag obviously does not exist. In such scenarios this
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index 053af0d227..ad244d9756 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -25,12 +25,18 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_device_info d_info = { .argsz = sizeof(d_info) };
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
- ret = dev_vfio_setup_device(path, dev_obj, dev_fd, &d_info);
+ ret = dev_vfio_setup_device(path, dev_obj, dev_fd);
if (ret) {
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
+ ret = dev_vfio_get_device_info(*dev_fd, &d_info);
+ if (ret) {
+ BCMFS_LOG(ERR, "VFIO Getting device info failed");
+ goto map_failed;
+ }
+
/* getting device region info*/
ret = ioctl(*dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info);
if (ret < 0) {
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index e35aae2540..b2d902f060 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -823,8 +823,7 @@ RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
dev_vfio_setup_device(__rte_unused const char *sysfs_base,
__rte_unused const char *dev_addr,
- __rte_unused int *vfio_dev_fd,
- __rte_unused struct vfio_device_info *device_info)
+ __rte_unused int *vfio_dev_fd)
{
rte_errno = ENOTSUP;
return -1;
@@ -957,3 +956,12 @@ dev_vfio_container_dma_unmap(__rte_unused int container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
+int
+dev_vfio_get_device_info(__rte_unused int vfio_dev_fd,
+ __rte_unused struct vfio_device_info *device_info)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index d0f741061e..099a35aa0e 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -55,10 +55,7 @@ struct vfio_device_info;
* device location.
*
* @param vfio_dev_fd
- * VFIO fd.
- *
- * @param device_info
- * Device information.
+ * Pointer to VFIO fd, will be set to the opened device fd on success.
*
* @return
* 0 on success.
@@ -67,7 +64,7 @@ struct vfio_device_info;
*/
__rte_internal
int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+ int *vfio_dev_fd);
/**
* @internal
@@ -193,19 +190,16 @@ dev_vfio_get_group_num(const char *sysfs_base,
* @internal
* Get device information.
*
+ * This function retrieves VFIO device information from an already opened
+ * device. The device must be opened with `dev_vfio_setup_device()` first.
+ *
* This function is only relevant to Linux and will return an error on BSD.
*
- * @param sysfs_base
- * sysfs path prefix.
- *
- * @param dev_addr
- * device location.
- *
* @param vfio_dev_fd
- * VFIO fd.
+ * VFIO device fd (must be a valid, already opened fd).
*
* @param device_info
- * Device information.
+ * Pointer to device information structure to be filled.
*
* @return
* 0 on success.
@@ -213,8 +207,7 @@ dev_vfio_get_group_num(const char *sysfs_base,
*/
__rte_internal
int
-dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info);
+dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index ea7e028cfa..0a7e3e9c9b 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -772,7 +772,7 @@ dev_vfio_clear_group(int vfio_group_fd)
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
int
dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+ int *vfio_dev_fd)
{
struct vfio_group_status group_status = {
.argsz = sizeof(group_status)
@@ -994,7 +994,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
*vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
dev);
if (*vfio_dev_fd >= 0)
- goto dev_get_info;
+ goto out;
}
/* get a file descriptor for the device */
@@ -1011,18 +1011,8 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
}
- /* test and setup the device */
-dev_get_info:
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, "
- "error %i (%s)", dev_addr, errno,
- strerror(errno));
- close(*vfio_dev_fd);
- close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
- return -1;
- }
+ /* device is now set up */
+out:
vfio_group_device_get(vfio_group_fd);
return 0;
@@ -1249,8 +1239,7 @@ vfio_set_iommu_type(int vfio_container_fd)
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
int
-dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
- int *vfio_dev_fd, struct vfio_device_info *device_info)
+dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
@@ -1259,21 +1248,14 @@ dev_vfio_get_device_info(const char *sysfs_base, const char *dev_addr,
return -1;
}
- if (device_info == NULL || *vfio_dev_fd < 0)
+ if (device_info == NULL || vfio_dev_fd < 0)
return -1;
- if (*vfio_dev_fd == 0) {
- ret = dev_vfio_setup_device(sysfs_base, dev_addr,
- vfio_dev_fd, device_info);
- if (ret)
- return -1;
- } else {
- ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get device info, error %i (%s)",
- dev_addr, errno, strerror(errno));
- return -1;
- }
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
}
return 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 10/25] vfio: add container device assignment API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (8 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 09/25] vfio: split get device info from setup Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 11/25] net/nbl: do not use VFIO group bind API Anatoly Burakov
` (15 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson
Currently, VFIO has explicit group bind API's, but the way they're used is
such that no one actually cares about VFIO groups, and the real goal of
everyone using VFIO group bind API is to bind devices to particular VFIO
container, such that when `dev_vfio_setup_device` call eventually comes,
VFIO will pick up the correct container.
To remove dependency on group API's, add a new "container assign device"
API call that will do the same thing, but will not depend on managing VFIO
group fd's.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 10 ++++++++++
lib/eal/include/dev_vfio.h | 26 ++++++++++++++++++++++++++
lib/eal/linux/eal_vfio.c | 32 ++++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index b2d902f060..31340bb49a 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -965,3 +965,13 @@ dev_vfio_get_device_info(__rte_unused int vfio_dev_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(__rte_unused int vfio_container_fd,
+ __rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 099a35aa0e..eb8697e75c 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -277,6 +277,32 @@ __rte_internal
int
dev_vfio_container_destroy(int container_fd);
+/**
+ * @internal
+ *
+ * Assign a device to a VFIO container.
+ *
+ * Doing so will cause `dev_vfio_setup_device()` call to set up the device with
+ * the VFIO container specified in this assign operation.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @param vfio_container_fd
+ * VFIO container file descriptor.
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int
+dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr);
+
/**
* @internal
* Bind a IOMMU group to a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 0a7e3e9c9b..ae53d95de5 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -2161,6 +2161,38 @@ dev_vfio_container_destroy(int container_fd)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
+ const char *dev_addr)
+{
+ int iommu_group_num;
+ int ret;
+
+ ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
+ dev_addr);
+ return -1;
+ } else if (ret == 0) {
+ EAL_LOG(ERR,
+ "Device %s is not assigned to any IOMMU group",
+ dev_addr);
+ return -1;
+ }
+
+ ret = dev_vfio_container_group_bind(vfio_container_fd,
+ iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR,
+ "Cannot bind IOMMU group %d for device %s",
+ iommu_group_num, dev_addr);
+ return -1;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
int
dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 11/25] net/nbl: do not use VFIO group bind API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (9 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 10/25] vfio: add container device assignment API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 12/25] net/ntnic: use container device assignment API Anatoly Burakov
` (14 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Dimon Zhao, Leon Yu, Sam Chen
The NBL driver currently uses group bind API, but it is using it only to
get group fd and nothing else. In context of NBL driver, this is the only
usage of VFIO API's in the driver, and it is not necessary to use it for
what NBL driver is trying to accomplish.
Use a direct `open()` call instead, and store the group fd in common
structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/nbl/nbl_common/nbl_userdev.c | 24 ++++++++++++++++-------
drivers/net/nbl/nbl_include/nbl_include.h | 1 +
2 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index f1522772f5..b3b6548bb8 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -387,6 +387,15 @@ nbl_userdev_mem_event_callback(enum rte_mem_event type, const void *addr, size_t
}
}
+static int
+nbl_open_group_fd(int iommu_group_num)
+{
+ char path[PATH_MAX];
+
+ snprintf(path, sizeof(path), DEV_VFIO_GROUP_FMT, iommu_group_num);
+ return open(path, O_RDWR);
+}
+
static int nbl_mdev_map_device(struct nbl_adapter *adapter)
{
const struct rte_pci_device *pci_dev = adapter->pci_dev;
@@ -424,11 +433,16 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
NBL_LOG(DEBUG, "nbl vfio container %d", container);
- vfio_group_fd = dev_vfio_container_group_bind(container, common->iommu_group_num);
+ /*
+ * This assumes one device per group, as kernel will return -EBUSY on
+ * attempting to open the same group fd for a different device.
+ */
+ vfio_group_fd = nbl_open_group_fd(common->iommu_group_num);
if (vfio_group_fd < 0) {
NBL_LOG(ERR, "nbl vfio group bind failed, %d", vfio_group_fd);
goto free_container;
}
+ common->groupfd = vfio_group_fd;
/* check if the group is viable */
ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
@@ -535,7 +549,6 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
}
free_group:
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
free_container:
if (container_create)
dev_vfio_container_destroy(container);
@@ -550,17 +563,14 @@ static int nbl_mdev_unmap_device(struct nbl_adapter *adapter)
rte_mcfg_mem_read_lock();
TAILQ_REMOVE(&nbl_adapter_list, adapter, next);
close(common->devfd);
- vfio_group_fd = dev_vfio_container_group_bind(nbl_default_container,
- common->iommu_group_num);
+ vfio_group_fd = common->groupfd;
NBL_LOG(DEBUG, "close vfio_group_fd %d", vfio_group_fd);
ret = ioctl(vfio_group_fd, VFIO_GROUP_UNSET_CONTAINER, &nbl_default_container);
if (ret)
NBL_LOG(ERR, "unset container, error %i (%s) %d",
errno, strerror(errno), ret);
nbl_group_count--;
- ret = dev_vfio_container_group_unbind(nbl_default_container, common->iommu_group_num);
- if (ret)
- NBL_LOG(ERR, "vfio container group unbind failed %d", ret);
+ close(vfio_group_fd);
if (!nbl_group_count) {
rte_mem_event_callback_unregister(NBL_USERDEV_EVENT_CLB_NAME, NULL);
nbl_userdev_dma_free();
diff --git a/drivers/net/nbl/nbl_include/nbl_include.h b/drivers/net/nbl/nbl_include/nbl_include.h
index eeae6a3301..ba99b9f8e7 100644
--- a/drivers/net/nbl/nbl_include/nbl_include.h
+++ b/drivers/net/nbl/nbl_include/nbl_include.h
@@ -132,6 +132,7 @@ struct nbl_common_info {
u16 vsi_id;
u16 instance_id;
int devfd;
+ int groupfd;
int eventfd;
int ifindex;
int iommu_group_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 12/25] net/ntnic: use container device assignment API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (10 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 11/25] net/nbl: do not use VFIO group bind API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 13/25] vdpa/ifc: " Anatoly Burakov
` (13 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
The NTNIC driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_vfio.c | 30 +++++++++---------------------
1 file changed, 9 insertions(+), 21 deletions(-)
diff --git a/drivers/net/ntnic/ntnic_vfio.c b/drivers/net/ntnic/ntnic_vfio.c
index 8489a59859..cacd81643f 100644
--- a/drivers/net/ntnic/ntnic_vfio.c
+++ b/drivers/net/ntnic/ntnic_vfio.c
@@ -28,7 +28,6 @@ nt_vfio_vf_num(const struct rte_pci_device *pdev)
/* Internal API */
struct vfio_dev {
int container_fd;
- int group_fd;
int dev_fd;
uint64_t iova_addr;
};
@@ -50,7 +49,6 @@ nthw_vfio_setup(struct rte_pci_device *dev)
{
int ret;
char devname[RTE_DEV_NAME_MAX_LEN] = { 0 };
- int iommu_group_num;
int vf_num;
struct vfio_dev *vfio;
@@ -66,14 +64,9 @@ nthw_vfio_setup(struct rte_pci_device *dev)
}
vfio->dev_fd = -1;
- vfio->group_fd = -1;
vfio->iova_addr = START_VF_IOVA;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname, &iommu_group_num);
- if (ret <= 0)
- return -1;
-
if (vf_num == 0) {
/* use default container for pf0 */
vfio->container_fd = DEV_VFIO_DEFAULT_CONTAINER_FD;
@@ -86,17 +79,14 @@ nthw_vfio_setup(struct rte_pci_device *dev)
"VFIO device setup failed. VFIO container creation failed.");
return -1;
}
- }
+ ret = dev_vfio_container_assign_device(vfio->container_fd,
+ rte_pci_get_sysfs_path(), devname);
+ if (ret < 0) {
+ NT_LOG(ERR, NTNIC,
+ "VFIO device setup failed. Assign device to container failed.");
+ goto err;
+ }
- vfio->group_fd = dev_vfio_container_group_bind(vfio->container_fd, iommu_group_num);
-
- if (vfio->group_fd < 0) {
- NT_LOG(ERR, NTNIC,
- "VFIO device setup failed. VFIO container group bind failed.");
- goto err;
- }
-
- if (vf_num > 0) {
if (rte_pci_map_device(dev)) {
NT_LOG(ERR, NTNIC,
"Map VFIO device failed. is the vfio-pci driver loaded?");
@@ -106,10 +96,8 @@ nthw_vfio_setup(struct rte_pci_device *dev)
vfio->dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
- NT_LOG(DBG, NTNIC,
- "%s: VFIO id=%d, dev_fd=%d, container_fd=%d, group_fd=%d, iommu_group_num=%d",
- dev->name, vf_num, vfio->dev_fd, vfio->container_fd, vfio->group_fd,
- iommu_group_num);
+ NT_LOG(DBG, NTNIC, "%s: VFIO id=%d, dev_fd=%d, container_fd=%d",
+ dev->name, vf_num, vfio->dev_fd, vfio->container_fd);
return vf_num;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 13/25] vdpa/ifc: use container device assignment API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (11 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 12/25] net/ntnic: use container device assignment API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 14/25] vdpa/nfp: " Anatoly Burakov
` (12 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev
The IFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/ifc/ifcvf_vdpa.c | 34 +++-------------------------------
1 file changed, 3 insertions(+), 31 deletions(-)
diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c
index 36d06f1e3f..04ed2052d3 100644
--- a/drivers/vdpa/ifc/ifcvf_vdpa.c
+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c
@@ -58,7 +58,6 @@ struct ifcvf_internal {
struct ifcvf_hw hw;
int configured;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
rte_thread_t tid; /* thread for notify relay */
rte_thread_t intr_tid; /* thread for config space change interrupt relay */
@@ -174,28 +173,19 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)
{
struct rte_pci_device *dev = internal->pdev;
char devname[RTE_DEV_NAME_MAX_LEN] = {0};
- int iommu_group_num;
- int i, ret;
+ int i;
internal->vfio_dev_fd = -1;
- internal->vfio_group_fd = -1;
internal->vfio_container_fd = -1;
rte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,
- &iommu_group_num);
- if (ret <= 0) {
- DRV_LOG(ERR, "%s failed to get IOMMU group", devname);
- return -1;
- }
internal->vfio_container_fd = dev_vfio_container_create();
if (internal->vfio_container_fd < 0)
return -1;
- internal->vfio_group_fd = dev_vfio_container_group_bind(
- internal->vfio_container_fd, iommu_group_num);
- if (internal->vfio_group_fd < 0)
+ if (dev_vfio_container_assign_device(internal->vfio_container_fd,
+ rte_pci_get_sysfs_path(), devname) < 0)
goto err;
if (rte_pci_map_device(dev))
@@ -1213,22 +1203,6 @@ ifcvf_set_features(int vid)
return 0;
}
-static int
-ifcvf_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct internal_list *list;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- list = find_internal_resource_by_vdev(vdev);
- if (list == NULL) {
- DRV_LOG(ERR, "Invalid vDPA device: %p", vdev);
- return -1;
- }
-
- return list->internal->vfio_group_fd;
-}
-
static int
ifcvf_get_vfio_device_fd(int vid)
{
@@ -1474,7 +1448,6 @@ static struct rte_vdpa_dev_ops ifcvf_net_ops = {
.set_vring_state = ifcvf_set_vring_state,
.set_features = ifcvf_set_features,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_dev_type = ifcvf_get_device_type,
@@ -1605,7 +1578,6 @@ static struct rte_vdpa_dev_ops ifcvf_blk_ops = {
.dev_close = ifcvf_dev_close,
.set_vring_state = ifcvf_set_vring_state,
.migration_done = NULL,
- .get_vfio_group_fd = ifcvf_get_vfio_group_fd,
.get_vfio_device_fd = ifcvf_get_vfio_device_fd,
.get_notify_area = ifcvf_get_notify_area,
.get_config = ifcvf_blk_get_config,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 14/25] vdpa/nfp: use container device assignment API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (12 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 13/25] vdpa/ifc: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 15/25] vdpa/sfc: " Anatoly Burakov
` (11 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Chaoyong He
The NFP vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics. Since group API is no longer used
after these changes, remove its implementation.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/nfp/nfp_vdpa.c | 37 +++++--------------------------------
1 file changed, 5 insertions(+), 32 deletions(-)
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 94740e768d..76d7d616a6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -36,9 +36,7 @@ struct nfp_vdpa_dev {
struct nfp_vdpa_hw hw;
int vfio_container_fd;
- int vfio_group_fd;
int vfio_dev_fd;
- int iommu_group;
rte_thread_t tid; /**< Thread for notify relay */
int epoll_fd;
@@ -120,33 +118,26 @@ nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device)
struct rte_pci_device *pci_dev = device->pci_dev;
rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN);
- ret = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &device->iommu_group);
- if (ret <= 0)
- return -1;
device->vfio_container_fd = dev_vfio_container_create();
if (device->vfio_container_fd < 0)
return -1;
- device->vfio_group_fd = dev_vfio_container_group_bind(
- device->vfio_container_fd, device->iommu_group);
- if (device->vfio_group_fd < 0)
+ ret = dev_vfio_container_assign_device(device->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (ret < 0)
goto container_destroy;
- DRV_VDPA_LOG(DEBUG, "The container_fd=%d, group_fd=%d.",
- device->vfio_container_fd, device->vfio_group_fd);
+ DRV_VDPA_LOG(DEBUG, "container_fd=%d", device->vfio_container_fd);
ret = rte_pci_map_device(pci_dev);
if (ret != 0)
- goto group_unbind;
+ goto container_destroy;
device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle);
return 0;
-group_unbind:
- dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
container_destroy:
dev_vfio_container_destroy(device->vfio_container_fd);
@@ -157,7 +148,6 @@ static void
nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device)
{
rte_pci_unmap_device(device->pci_dev);
- dev_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group);
dev_vfio_container_destroy(device->vfio_container_fd);
}
@@ -1023,22 +1013,6 @@ nfp_vdpa_dev_close(int vid)
return 0;
}
-static int
-nfp_vdpa_get_vfio_group_fd(int vid)
-{
- struct rte_vdpa_device *vdev;
- struct nfp_vdpa_dev_node *node;
-
- vdev = rte_vhost_get_vdpa_device(vid);
- node = nfp_vdpa_find_node_by_vdev(vdev);
- if (node == NULL) {
- DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p.", vdev);
- return -ENODEV;
- }
-
- return node->device->vfio_group_fd;
-}
-
static int
nfp_vdpa_get_vfio_device_fd(int vid)
{
@@ -1190,7 +1164,6 @@ struct rte_vdpa_dev_ops nfp_vdpa_ops = {
.dev_close = nfp_vdpa_dev_close,
.set_vring_state = nfp_vdpa_set_vring_state,
.set_features = nfp_vdpa_set_features,
- .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd,
.get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd,
.get_notify_area = nfp_vdpa_get_notify_area,
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 15/25] vdpa/sfc: use container device assignment API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (13 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 14/25] vdpa/nfp: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 16/25] vdpa/mlx5: remove group-related API Anatoly Burakov
` (10 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Vijay Kumar Srivastava
The SFC vDPA driver uses VFIO group bind/unbind functionality for container
device assignment purposes. Use the new container device assignment API
instead to provide clearer semantics.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/sfc/sfc_vdpa.c | 39 +++++++------------------------------
drivers/vdpa/sfc/sfc_vdpa.h | 2 --
2 files changed, 7 insertions(+), 34 deletions(-)
diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c
index c65f716a15..bd5420cfa5 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.c
+++ b/drivers/vdpa/sfc/sfc_vdpa.c
@@ -80,22 +80,12 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
goto fail_container_create;
}
- rc = dev_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name,
- &sva->iommu_group_num);
- if (rc <= 0) {
- sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s",
- dev_name, rte_strerror(-rc));
- goto fail_get_group_num;
- }
-
- sva->vfio_group_fd =
- dev_vfio_container_group_bind(sva->vfio_container_fd,
- sva->iommu_group_num);
- if (sva->vfio_group_fd < 0) {
- sfc_vdpa_err(sva,
- "failed to bind IOMMU group %d to container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- goto fail_group_bind;
+ rc = dev_vfio_container_assign_device(sva->vfio_container_fd,
+ rte_pci_get_sysfs_path(), dev_name);
+ if (rc < 0) {
+ sfc_vdpa_err(sva, "failed to assign device %s to container %d",
+ dev_name, sva->vfio_container_fd);
+ goto fail_device_assign;
}
if (rte_pci_map_device(dev) != 0) {
@@ -109,15 +99,7 @@ sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva)
return 0;
fail_pci_map_device:
- if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
-fail_group_bind:
-fail_get_group_num:
+fail_device_assign:
if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva, "failed to destroy container %d",
sva->vfio_container_fd);
@@ -132,13 +114,6 @@ sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva)
{
rte_pci_unmap_device(sva->pdev);
- if (dev_vfio_container_group_unbind(sva->vfio_container_fd,
- sva->iommu_group_num) != 0) {
- sfc_vdpa_err(sva,
- "failed to unbind IOMMU group %d from container %d",
- sva->iommu_group_num, sva->vfio_container_fd);
- }
-
if (dev_vfio_container_destroy(sva->vfio_container_fd) != 0) {
sfc_vdpa_err(sva,
"failed to destroy container %d",
diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h
index 2b843e563d..99a81fd1b0 100644
--- a/drivers/vdpa/sfc/sfc_vdpa.h
+++ b/drivers/vdpa/sfc/sfc_vdpa.h
@@ -70,10 +70,8 @@ struct sfc_vdpa_adapter {
sfc_vdpa_filter_t filters;
- int vfio_group_fd;
int vfio_dev_fd;
int vfio_container_fd;
- int iommu_group_num;
struct sfc_vdpa_ops_data *ops_data;
};
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 16/25] vdpa/mlx5: remove group-related API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (14 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 15/25] vdpa/sfc: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 17/25] vhost: remove group-related API from driver Anatoly Burakov
` (9 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Matan Azrad, Viacheslav Ovsiienko
The mlx5 driver does not use the `get_group_fd` API provided by vDPA. This
API is about to be removed, so remove its mention from mlx5.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/vdpa/mlx5/mlx5_vdpa.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 11708e2005..1bb5de51b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -523,7 +523,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.set_vring_state = mlx5_vdpa_set_vring_state,
.set_features = mlx5_vdpa_features_set,
.migration_done = NULL,
- .get_vfio_group_fd = NULL,
.get_vfio_device_fd = mlx5_vdpa_get_device_fd,
.get_notify_area = mlx5_vdpa_get_notify_area,
.get_stats_names = mlx5_vdpa_get_stats_names,
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 17/25] vhost: remove group-related API from driver
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (15 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 16/25] vdpa/mlx5: remove group-related API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 18/25] vfio: remove group-based API Anatoly Burakov
` (8 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Maxime Coquelin, Chenbo Xia
The vDPA driver API has "get_vfio_group_fd" in its ops structure, but no
driver is using it any more, as better API's were added and implemented.
Remove this API from vDPA ops structure.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/prog_guide/vhost_lib.rst | 4 ----
doc/guides/rel_notes/deprecation.rst | 4 ----
doc/guides/rel_notes/release_26_11.rst | 3 +++
lib/vhost/vdpa_driver.h | 3 ---
4 files changed, 3 insertions(+), 11 deletions(-)
diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst
index 345a621716..99355b8afd 100644
--- a/doc/guides/prog_guide/vhost_lib.rst
+++ b/doc/guides/prog_guide/vhost_lib.rst
@@ -471,10 +471,6 @@ Finally, a set of device ops is defined for device specific operations:
Called to allow the device to respond to RARP sending.
-* ``get_vfio_group_fd``
-
- Called to get the VFIO group fd of the device.
-
* ``get_vfio_device_fd``
Called to get the VFIO device fd of the device.
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 8a82c815e3..e4a754d037 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -29,10 +29,6 @@ Deprecation Notices
Use the ``-S <service-corelist>`` parameter instead
to specify the cores to be used for background services in DPDK.
-* vdpa: The vDPA driver API will no longer offer ``get_vfio_group_fd``
- as part of its internal API. All drivers will be adjusted
- to use the new unified VFIO container device assignment API.
-
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported
in DPDK. DPDK has adopted the atomic operations from
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 71d12a4398..006b1d3e5a 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -117,6 +117,9 @@ ABI Changes
These functions are now available only to EAL and drivers,
and are no longer part of the public ABI.
+* vdpa: Removed ``get_vfio_group_fd`` from ``struct rte_vdpa_dev_ops``.
+ vDPA drivers should now use the VFIO container device assignment API.
+
Known Issues
------------
diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h
index 42392a0d14..c7b9be09fb 100644
--- a/lib/vhost/vdpa_driver.h
+++ b/lib/vhost/vdpa_driver.h
@@ -50,9 +50,6 @@ struct rte_vdpa_dev_ops {
/** Destination operations when migration done */
int (*migration_done)(int vid);
- /** Get the vfio group fd */
- int (*get_vfio_group_fd)(int vid);
-
/** Get the vfio device fd */
int (*get_vfio_device_fd)(int vid);
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 18/25] vfio: remove group-based API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (16 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 17/25] vhost: remove group-related API from driver Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 19/25] vfio: cleanup and refactor Anatoly Burakov
` (7 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson
All drivers have been adjusted to not use the VFIO group API directly and
instead rely on container device assignment model, so the group API is no
longer useful and can be removed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 34 ---------------
lib/eal/include/dev_vfio.h | 72 --------------------------------
lib/eal/linux/eal_vfio.c | 47 ++++++++++-----------
lib/eal/linux/eal_vfio.h | 2 +
lib/eal/linux/eal_vfio_mp_sync.c | 2 +-
5 files changed, 26 insertions(+), 131 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 31340bb49a..b607ea50b8 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -867,14 +867,6 @@ dev_vfio_noiommu_is_enabled(void)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
-int
-dev_vfio_clear_group(__rte_unused int vfio_group_fd)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
@@ -893,14 +885,6 @@ dev_vfio_get_container_fd(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
-int
-dev_vfio_get_group_fd(__rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
@@ -917,24 +901,6 @@ dev_vfio_container_destroy(__rte_unused int container_fd)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
-int
-dev_vfio_container_group_bind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
-int
-dev_vfio_container_group_unbind(__rte_unused int container_fd,
- __rte_unused int iommu_group_num)
-{
- rte_errno = ENOTSUP;
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
dev_vfio_container_dma_map(__rte_unused int container_fd,
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index eb8697e75c..f2dbf46481 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -142,24 +142,6 @@ int dev_vfio_is_enabled(const char *modname);
__rte_internal
int dev_vfio_noiommu_is_enabled(void);
-/**
- * @internal
- * Remove group fd from internal VFIO group fd array.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param vfio_group_fd
- * VFIO Group FD.
- *
- * @return
- * 0 on success.
- * <0 on failure.
- */
-__rte_internal
-int
-dev_vfio_clear_group(int vfio_group_fd);
-
/**
* @internal
* Parse IOMMU group number for a device.
@@ -224,24 +206,6 @@ __rte_internal
int
dev_vfio_get_container_fd(void);
-/**
- * @internal
- * Open VFIO group fd or get an existing one.
- *
- * This function is only relevant to linux and will return
- * an error on BSD.
- *
- * @param iommu_group_num
- * iommu group number
- *
- * @return
- * > 0 group fd
- * < 0 for errors
- */
-__rte_internal
-int
-dev_vfio_get_group_fd(int iommu_group_num);
-
/**
* @internal
* Create a new container for device binding.
@@ -303,42 +267,6 @@ int
dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
const char *dev_addr);
-/**
- * @internal
- * Bind a IOMMU group to a container.
- *
- * @param container_fd
- * the container's fd
- *
- * @param iommu_group_num
- * the iommu group number to bind to container
- *
- * @return
- * group fd if successful
- * <0 if failed
- */
-__rte_internal
-int
-dev_vfio_container_group_bind(int container_fd, int iommu_group_num);
-
-/**
- * @internal
- * Unbind a IOMMU group from a container.
- *
- * @param container_fd
- * the container fd of container
- *
- * @param iommu_group_num
- * the iommu group number to delete from container
- *
- * @return
- * 0 if successful
- * <0 if failed
- */
-__rte_internal
-int
-dev_vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
/**
* @internal
* Perform DMA mapping for devices in a container.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index ae53d95de5..985df228d1 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -71,6 +71,9 @@ static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
+static int vfio_container_group_bind(int container_fd, int iommu_group_num);
+static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
+
/* IOMMU types we support */
static const struct vfio_iommu_type iommu_types[] = {
/* x86 IOMMU, otherwise known as type 1 */
@@ -536,9 +539,8 @@ get_vfio_cfg_by_container_fd(int container_fd)
return NULL;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_fd)
int
-dev_vfio_get_group_fd(int iommu_group_num)
+vfio_get_group_fd_by_num(int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -740,9 +742,8 @@ vfio_sync_default_container(void)
return -1;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_clear_group)
-int
-dev_vfio_clear_group(int vfio_group_fd)
+static int
+vfio_clear_group(int vfio_group_fd)
{
int i;
struct vfio_config *vfio_cfg;
@@ -806,7 +807,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
return -1;
/* get the actual group fd */
- vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
return -1;
@@ -832,14 +833,14 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(ERR, "%s cannot get VFIO group status, "
"error %i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
} else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
EAL_LOG(ERR, "%s VFIO group is not viable! "
"Not all devices in IOMMU group bound to VFIO or unbound",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -860,7 +861,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s cannot add VFIO group to container, error "
"%i (%s)", dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -884,7 +885,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%s failed to select IOMMU type",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* lock memory hotplug before mapping and release it
@@ -901,7 +902,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
"%i (%s)",
dev_addr, errno, strerror(errno));
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
rte_mcfg_mem_read_unlock();
return -1;
}
@@ -970,7 +971,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
if (ret < 0) {
EAL_LOG(ERR, "Could not sync default VFIO container");
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
/* we have successfully initialized VFIO, notify user */
@@ -1007,7 +1008,7 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
dev_addr);
close(vfio_group_fd);
- dev_vfio_clear_group(vfio_group_fd);
+ vfio_clear_group(vfio_group_fd);
return -1;
}
@@ -1050,9 +1051,9 @@ dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
}
/* get the actual group fd */
- vfio_group_fd = dev_vfio_get_group_fd(iommu_group_num);
+ vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "dev_vfio_get_group_fd failed for %s",
+ EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
dev_addr);
ret = vfio_group_fd;
goto out;
@@ -1088,7 +1089,7 @@ dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
goto out;
}
- if (dev_vfio_clear_group(vfio_group_fd) < 0) {
+ if (vfio_clear_group(vfio_group_fd) < 0) {
EAL_LOG(INFO, "Error when clearing group for %s",
dev_addr);
ret = -1;
@@ -2149,7 +2150,7 @@ dev_vfio_container_destroy(int container_fd)
for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
if (vfio_cfg->vfio_groups[i].group_num != -1)
- dev_vfio_container_group_unbind(container_fd,
+ vfio_container_group_unbind(container_fd,
vfio_cfg->vfio_groups[i].group_num);
close(container_fd);
@@ -2181,7 +2182,7 @@ dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return -1;
}
- ret = dev_vfio_container_group_bind(vfio_container_fd,
+ ret = vfio_container_group_bind(vfio_container_fd,
iommu_group_num);
if (ret < 0) {
EAL_LOG(ERR,
@@ -2193,9 +2194,8 @@ dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_bind)
-int
-dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_bind(int container_fd, int iommu_group_num)
{
struct vfio_config *vfio_cfg;
@@ -2213,9 +2213,8 @@ dev_vfio_container_group_bind(int container_fd, int iommu_group_num)
return vfio_get_group_fd(vfio_cfg, iommu_group_num);
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_group_unbind)
-int
-dev_vfio_container_group_unbind(int container_fd, int iommu_group_num)
+static int
+vfio_container_group_unbind(int container_fd, int iommu_group_num)
{
struct vfio_group *cur_grp = NULL;
struct vfio_config *vfio_cfg;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 89c4b5ba45..30389fb274 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -51,6 +51,8 @@ vfio_set_iommu_type(int vfio_container_fd);
int
vfio_get_iommu_type(void);
+int vfio_get_group_fd_by_num(int iommu_group_num);
+
/* check if we have any supported extensions */
int
vfio_has_supported_extensions(int vfio_container_fd);
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index ae1c8b40b9..66341a701a 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -34,7 +34,7 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
case SOCKET_REQ_GROUP:
r->req = SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = dev_vfio_get_group_fd(m->group_num);
+ fd = vfio_get_group_fd_by_num(m->group_num);
if (fd < 0 && fd != -ENOENT)
r->result = SOCKET_ERR;
else if (fd == -ENOENT)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 19/25] vfio: cleanup and refactor
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (17 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 18/25] vfio: remove group-based API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 20/25] bus/pci: use the new VFIO mode API Anatoly Burakov
` (6 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Wathsala Vithanage, Bruce Richardson, Nipun Gupta,
Nikhil Agarwal, Hemant Agrawal, Sachin Saxena, Chenbo Xia,
Ajit Khaparde, Vikas Gupta, Dimon Zhao, Leon Yu, Sam Chen
Currently, VFIO code is a bit of an incoherent mess internally, with API's
bleeding into each other, inconsistent returns, and a certain amount of
spaghetti stemming from organic growth.
Refactor VFIO code to achieve the following goals:
- Make all error handling consistent, and provide/document rte_errno values
returned from API's to indicate various conditions.
- Introduce new "VFIO mode" concept. This new API will tell caller if
VFIO is enabled, and whether it is using group API, and whether it is
running in no-IOMMU mode.
- Decouple dev_vfio_setup_device semantics from PCI bus return convention.
Currently, when device is not managed by VFIO, dev_vfio_setup_device
will return 1, which is bus speak for "skip this device", however VFIO
has nothing to do with PCI bus and should not follow its API conventions.
- Perform device setup in device assign, and make device setup use shared
code path with device assign and explicitly assuming default container.
This is technically not necessary for group mode as device set up is a
two-step process in that mode, but coming cdev mode will have a
single-step device setup, and it would be easier if the worked the same
way under the hood.
- Add device deduplication based on sysfs path and device address.
- Make VFIO internals more readable. Introduce a lot of infrastructure and
more explicit validation, rather than over-reliance on sentinel values
and implicit assumptions. This will also make it easier to integrate cdev
mode down the line, as it will rely on most of this infrastructure.
- Rename internal structs and macros to use vfio_* namespace
This will change behavior of the following functions:
- `dev_vfio_setup_device` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `dev_vfio_get_group_num` - when the device is not managed by VFIO, the
function will now return -1 with `rte_errno` set to ENODEV
- `dev_vfio_container_destroy` - the function will now release and close
all group and device resources associated with the container being
destroyed by this call
- `dev_vfio_noiommu_is_enabled` is now using the node mode infrastructure
rather than reading a sysfs value
All users of `dev_vfio_setup_device` and `dev_vfio_get_group_num` have been
adjusted to account for API change.
Because the no-IOMMU check now relies on new mode infrastructure, the VFIO
initialization now needs to happen before bus scan, as buses depend on
no-IOMMU information.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
config/arm/meson.build | 1 +
config/meson.build | 1 +
drivers/bus/cdx/cdx_vfio.c | 17 +-
drivers/bus/fslmc/fslmc_vfio.c | 4 +-
drivers/bus/pci/linux/pci_vfio.c | 19 +-
drivers/crypto/bcmfs/bcmfs_vfio.c | 6 +-
drivers/net/nbl/nbl_common/nbl_userdev.c | 2 +-
lib/eal/freebsd/eal.c | 7 +
lib/eal/include/dev_vfio.h | 250 ++-
lib/eal/linux/eal.c | 15 +-
lib/eal/linux/eal_vfio.c | 2574 ++++++++--------------
lib/eal/linux/eal_vfio.h | 160 +-
lib/eal/linux/eal_vfio_group.c | 983 +++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 62 +-
lib/eal/linux/meson.build | 1 +
15 files changed, 2249 insertions(+), 1853 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_group.c
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 27b549a052..2b73cbef7e 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -147,6 +147,7 @@ implementer_cavium = {
'description': 'Cavium',
'flags': [
['RTE_MAX_VFIO_GROUPS', 128],
+ ['RTE_MAX_VFIO_DEVICES', 256],
['RTE_MAX_LCORE', 96],
['RTE_MAX_NUMA_NODES', 2]
],
diff --git a/config/meson.build b/config/meson.build
index d7f5e55c18..0e6e478fc8 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -387,6 +387,7 @@ dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
dpdk_conf.set('RTE_PKTMBUF_HEADROOM', get_option('pkt_mbuf_headroom'))
# values which have defaults which may be overridden
dpdk_conf.set('RTE_MAX_VFIO_GROUPS', 64)
+dpdk_conf.set('RTE_MAX_VFIO_DEVICES', 256)
dpdk_conf.set('RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB', 64)
dpdk_conf.set('RTE_LIBRTE_DPAA2_USE_PHYS_IOVA', true)
if get_option('mbuf_refcnt_atomic')
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 531e4ccc8d..7ba0937950 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -22,6 +22,7 @@
#include <eal_export.h>
#include <rte_eal_paging.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
#include <dev_vfio.h>
@@ -402,8 +403,12 @@ cdx_vfio_map_resource_primary(struct rte_cdx_device *dev)
ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
&vfio_dev_fd);
- if (ret)
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -513,11 +518,13 @@ cdx_vfio_map_resource_secondary(struct rte_cdx_device *dev)
return -1;
}
- ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(RTE_CDX_BUS_DEVICES_PATH, dev_name, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
-
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
goto err_vfio_dev_fd;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 153e6112db..c2b6ac1f3f 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -376,9 +376,9 @@ fslmc_get_group_id(const char *group_name,
/* get group number */
ret = dev_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
group_name, groupid);
- if (ret <= 0) {
+ if (ret < 0) {
DPAA2_BUS_ERR("Find %s IOMMU group", group_name);
- if (ret < 0)
+ if (rte_errno != ENODEV)
return ret;
return -EIO;
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index 0363a4a6c5..61489922da 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -20,6 +20,7 @@
#include <rte_malloc.h>
#include <dev_vfio.h>
#include <rte_eal.h>
+#include <rte_errno.h>
#include <bus_driver.h>
#include <rte_spinlock.h>
#include <rte_tailq.h>
@@ -752,10 +753,13 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)
snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
loc->domain, loc->bus, loc->devid, loc->function);
- ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
@@ -965,10 +969,13 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
return -1;
}
- ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr,
- &vfio_dev_fd);
- if (ret)
+ ret = dev_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, &vfio_dev_fd);
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
return ret;
+ }
ret = dev_vfio_get_device_info(vfio_dev_fd, &device_info);
if (ret)
diff --git a/drivers/crypto/bcmfs/bcmfs_vfio.c b/drivers/crypto/bcmfs/bcmfs_vfio.c
index ad244d9756..0e16af0365 100644
--- a/drivers/crypto/bcmfs/bcmfs_vfio.c
+++ b/drivers/crypto/bcmfs/bcmfs_vfio.c
@@ -9,6 +9,7 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
+#include <rte_errno.h>
#include <dev_vfio.h>
#include "bcmfs_device.h"
@@ -26,7 +27,10 @@ vfio_map_dev_obj(const char *path, const char *dev_obj,
struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) };
ret = dev_vfio_setup_device(path, dev_obj, dev_fd);
- if (ret) {
+ if (ret < 0) {
+ /* Device not managed by VFIO - skip */
+ if (rte_errno == ENODEV)
+ ret = 1;
BCMFS_LOG(ERR, "VFIO Setting for device failed");
return ret;
}
diff --git a/drivers/net/nbl/nbl_common/nbl_userdev.c b/drivers/net/nbl/nbl_common/nbl_userdev.c
index b3b6548bb8..10dd179ac6 100644
--- a/drivers/net/nbl/nbl_common/nbl_userdev.c
+++ b/drivers/net/nbl/nbl_common/nbl_userdev.c
@@ -415,7 +415,7 @@ static int nbl_mdev_map_device(struct nbl_adapter *adapter)
"%s/%s/", rte_pci_get_sysfs_path(), dev_name);
ret = dev_vfio_get_group_num(pathname, dev_name, &common->iommu_group_num);
- if (ret <= 0) {
+ if (ret < 0) {
NBL_LOG(ERR, "nbl vfio group number failed");
return -1;
}
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index b607ea50b8..753aaf8656 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -941,3 +941,10 @@ dev_vfio_container_assign_device(__rte_unused int vfio_container_fd,
rte_errno = ENOTSUP;
return -1;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_mode)
+enum dev_vfio_mode
+dev_vfio_get_mode(void)
+{
+ return DEV_VFIO_MODE_NONE;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index f2dbf46481..6404dd08c3 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -18,6 +18,7 @@
#include <stdint.h>
#include <rte_compat.h>
+#include <rte_common.h>
#ifdef __cplusplus
extern "C" {
@@ -29,8 +30,7 @@ extern "C" {
#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
-#define DEV_VFIO_NOIOMMU_MODE \
- "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
+#define DEV_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
#endif /* RTE_EXEC_ENV_LINUX */
@@ -39,28 +39,50 @@ struct vfio_device_info;
#define DEV_VFIO_DEFAULT_CONTAINER_FD (-1)
+/**
+ * @enum dev_vfio_mode
+ * Enumeration of VFIO operational modes.
+ *
+ * These modes define how VFIO devices are accessed and managed:
+ *
+ * - DEV_VFIO_MODE_NONE: VFIO is not enabled.
+ * - DEV_VFIO_MODE_GROUP: Legacy group mode.
+ * - DEV_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ */
+enum dev_vfio_mode {
+ DEV_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
+ DEV_VFIO_MODE_GROUP, /**< Group mode */
+ DEV_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+};
+
/**
* @internal
- * Setup vfio_cfg for the device identified by its address.
- * It discovers the configured I/O MMU groups or sets a new one for the device.
- * If a new groups is assigned, the DMA mapping is performed.
+ * Set up a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * If the device was not previously assigned to a container using
+ * `dev_vfio_container_assign_device()`, default container will be used.
+ *
+ * This function is only relevant on Linux.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param vfio_dev_fd
- * Pointer to VFIO fd, will be set to the opened device fd on success.
+ * Pointer to where VFIO device file descriptor will be stored.
*
* @return
* 0 on success.
- * <0 on failure.
- * >1 if the device cannot be managed this way.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - ENOSPC - No space in VFIO container to track the device.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ * - ENOMEM - Memory allocation failed for device tracking.
*/
__rte_internal
int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
@@ -68,23 +90,31 @@ int dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
/**
* @internal
- * Release a device mapped to a VFIO-managed I/O MMU group.
+ * Release a device managed by VFIO driver.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
+ *
+ * @note As a result of this function, all internal resources used by the device will be released,
+ * so if the device was using a non-default container, it will need to be reassigned to the
+ * container before it can be used again.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param fd
- * VFIO fd.
+ * A previously set up VFIO file descriptor.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOENT - Device not found in any container.
+ * - EINVAL - Invalid parameters.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int dev_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd);
@@ -93,12 +123,18 @@ int dev_vfio_release_device(const char *sysfs_base, const char *dev_addr, int fd
* @internal
* Initialize VFIO.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * In case of success, `dev_vfio_get_mode()` can be used to retrieve the VFIO mode in use.
+ *
+ * This function is only relevant on Linux.
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Operation not supported.
*/
__rte_internal
int dev_vfio_enable(void);
@@ -114,59 +150,73 @@ void dev_vfio_cleanup(void);
/**
* @internal
- * Check whether a VFIO-related kmod is enabled.
+ * Check if VFIO subsystem is initialized and a specified kernel module is loaded.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @param modname
- * kernel module name.
+ * Kernel module name.
*
* @return
- * 1 if true.
- * 0 otherwise.
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
*/
__rte_internal
int dev_vfio_is_enabled(const char *modname);
/**
* @internal
- * Whether VFIO NOIOMMU mode is enabled.
+ * Get current VFIO mode.
*
- * This function is only relevant to Linux.
+ * This function is only relevant on Linux.
*
* @return
- * 1 if true.
- * 0 if false.
- * <0 for errors.
+ * VFIO mode currently in use.
*/
__rte_internal
-int dev_vfio_noiommu_is_enabled(void);
+enum dev_vfio_mode
+dev_vfio_get_mode(void);
+
+/**
+ * @internal
+ * Check if VFIO NOIOMMU mode is enabled.
+ *
+ * This function is only relevant on Linux in group mode.
+ *
+ * @return
+ * 1 if enabled.
+ * 0 if not enabled or not supported.
+ */
+__rte_internal
+int
+dev_vfio_noiommu_is_enabled(void);
/**
* @internal
* Parse IOMMU group number for a device.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux in group mode.
*
* @param sysfs_base
- * sysfs path prefix.
- *
+ * Sysfs path prefix.
* @param dev_addr
- * device location.
- *
+ * Device identifier.
* @param iommu_group_num
- * iommu group number
+ * Pointer to where IOMMU group number will be stored.
*
* @return
- * >0 on success
- * 0 for non-existent group or VFIO
- * <0 for errors
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
-dev_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num);
+dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
/**
* @internal
@@ -185,7 +235,12 @@ dev_vfio_get_group_num(const char *sysfs_base,
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -193,14 +248,17 @@ dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info);
/**
* @internal
- * Get the default VFIO container fd
+ * Get the default VFIO container file descriptor.
*
- * This function is only relevant to linux and will return
- * an error on BSD.
+ * This function is only relevant on Linux.
*
* @return
- * > 0 default container fd
- * < 0 if VFIO is not enabled or not supported
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -208,7 +266,9 @@ dev_vfio_get_container_fd(void);
/**
* @internal
- * Create a new container for device binding.
+ * Create a new VFIO container for device assignment and DMA mapping.
+ *
+ * This function is only relevant on Linux.
*
* @note Any newly allocated DPDK memory will not be mapped into these
* containers by default, user needs to manage DMA mappings for
@@ -219,8 +279,14 @@ dev_vfio_get_container_fd(void);
* devices between multiple processes is not supported.
*
* @return
- * the container fd if successful
- * <0 if failed
+ * Non-negative container file descriptor on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENOSPC - Maximum number of containers reached.
+ * - EIO - Underlying VFIO operation failed.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -228,14 +294,22 @@ dev_vfio_container_create(void);
/**
* @internal
- * Destroy the container, unbind all vfio groups within it.
+ * Destroy a VFIO container and unmap all devices assigned to it.
+ *
+ * This function is only relevant on Linux.
*
* @param container_fd
- * the container fd to destroy
+ * File descriptor of container to destroy.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Container not managed by VFIO.
+ * - EINVAL - Invalid container file descriptor.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -260,33 +334,47 @@ dev_vfio_container_destroy(int container_fd);
*
* @return
* 0 on success.
- * <0 on failure.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EEXIST - Device already assigned to the container.
+ * - ENOSPC - No space in VFIO container to assign device.
+ * - EINVAL - Invalid container file descriptor.
+ * - EIO - Error during underlying VFIO operations.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ * - ENOMEM - Memory allocation failed for device tracking.
*/
__rte_internal
int
-dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr);
+dev_vfio_container_assign_device(int vfio_container_fd,
+ const char *sysfs_base, const char *dev_addr);
/**
* @internal
* Perform DMA mapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use DEV_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be mapped.
- *
* @param iova
* Starting IOVA address of memory to be mapped.
- *
* @param len
* Length of memory segment being mapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA mapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
@@ -297,22 +385,26 @@ dev_vfio_container_dma_map(int container_fd, uint64_t vaddr,
* @internal
* Perform DMA unmapping for devices in a container.
*
- * @param container_fd
- * the specified container fd. Use DEV_VFIO_DEFAULT_CONTAINER_FD to
- * use the default container.
+ * This function is only relevant on Linux.
*
+ * @param container_fd
+ * Container file descriptor. Use DEV_VFIO_DEFAULT_CONTAINER_FD to use the default container.
* @param vaddr
* Starting virtual address of memory to be unmapped.
- *
* @param iova
* Starting IOVA address of memory to be unmapped.
- *
* @param len
* Length of memory segment being unmapped.
*
* @return
- * 0 if successful
- * <0 if failed
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - EIO - DMA unmapping operation failed.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
*/
__rte_internal
int
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index d63edb4192..722cacdb45 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -676,6 +676,16 @@ rte_eal_init(int argc, char **argv)
}
}
+ /*
+ * VFIO must be initialized before bus scan because buses need to know
+ * about no-IOMMU mode status.
+ */
+ if (dev_vfio_enable()) {
+ rte_eal_init_alert("Cannot init VFIO");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
if (rte_bus_scan()) {
rte_eal_init_alert("Cannot scan the buses for devices");
rte_errno = ENODEV;
@@ -771,11 +781,6 @@ rte_eal_init(int argc, char **argv)
#endif
}
- if (dev_vfio_enable()) {
- rte_eal_init_alert("Cannot init VFIO");
- rte_errno = EAGAIN;
- goto err_out;
- }
/* in secondary processes, memory init may allocate additional fbarrays
* not present in primary processes, so to avoid any potential issues,
* initialize memzones first.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 985df228d1..1cd1a44eef 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -24,86 +24,43 @@
#include "eal_private.h"
#include "eal_internal_cfg.h"
-#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
-#define VFIO_MODNAME "vfio"
-
-/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
- * recreate the mappings for DPDK segments, but we cannot do so for memory that
- * was registered by the user themselves, so we need to store the user mappings
- * somewhere, to recreate them later.
+/*
+ * rte_errno convention:
+ *
+ * - EINVAL: invalid parameters
+ * - ENOTSUP: current mode does not support this operation
+ * - ENXIO: VFIO not initialized
+ * - ENODEV: device not managed by VFIO
+ * - ENOSPC: no space in config
+ * - EEXIST: device already assigned
+ * - ENOENT: group or device not found
+ * - ENOMEM: memory allocation failed
+ * - EIO: underlying VFIO operation failed
*/
-#define EAL_VFIO_MAX_USER_MEM_MAPS 256
-struct user_mem_map {
- uint64_t addr; /**< start VA */
- uint64_t iova; /**< start IOVA */
- uint64_t len; /**< total length of the mapping */
- uint64_t chunk; /**< this mapping can be split in chunks of this size */
-};
-struct user_mem_maps {
- rte_spinlock_recursive_t lock;
- int n_maps;
- struct user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+/* functions can fail for multiple reasons, and errno is tedious */
+enum vfio_result {
+ VFIO_SUCCESS,
+ VFIO_ERROR,
+ VFIO_EXISTS,
+ VFIO_NOT_SUPPORTED,
+ VFIO_NOT_MANAGED,
+ VFIO_NOT_FOUND,
+ VFIO_NO_SPACE,
+ VFIO_NO_MEM,
};
-struct vfio_config {
- bool enabled;
- int vfio_container_fd;
- int vfio_active_groups;
- const struct vfio_iommu_type *vfio_iommu_type;
- struct vfio_group vfio_groups[RTE_MAX_VFIO_GROUPS];
- struct user_mem_maps mem_maps;
+struct vfio_container vfio_containers[RTE_MAX_VFIO_CONTAINERS] = {0};
+struct vfio_config vfio_global_cfg = {
+ .mode = DEV_VFIO_MODE_NONE,
+ .default_cfg = &vfio_containers[0]
};
-/* per-process VFIO config */
-static struct vfio_config vfio_cfgs[RTE_MAX_VFIO_CONTAINERS];
-static struct vfio_config *default_vfio_cfg = &vfio_cfgs[0];
-
-/* whether VFIO is enabled (usable) in this process */
-static bool vfio_enabled;
-
-static int vfio_type1_dma_map(int);
-static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_spapr_dma_map(int);
-static int vfio_spapr_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_noiommu_dma_map(int);
-static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
-static int vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr,
+static int vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
uint64_t iova, uint64_t len, int do_map);
-static int vfio_container_group_bind(int container_fd, int iommu_group_num);
-static int vfio_container_group_unbind(int container_fd, int iommu_group_num);
-
-/* IOMMU types we support */
-static const struct vfio_iommu_type iommu_types[] = {
- /* x86 IOMMU, otherwise known as type 1 */
- {
- .type_id = VFIO_TYPE1_IOMMU,
- .name = "Type 1",
- .partial_unmap = false,
- .dma_map_func = &vfio_type1_dma_map,
- .dma_user_map_func = &vfio_type1_dma_mem_map
- },
- /* ppc64 IOMMU, otherwise known as spapr */
- {
- .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
- .name = "sPAPR",
- .partial_unmap = true,
- .dma_map_func = &vfio_spapr_dma_map,
- .dma_user_map_func = &vfio_spapr_dma_mem_map
- },
- /* IOMMU-less mode */
- {
- .type_id = VFIO_NOIOMMU_IOMMU,
- .name = "No-IOMMU",
- .partial_unmap = true,
- .dma_map_func = &vfio_noiommu_dma_map,
- .dma_user_map_func = &vfio_noiommu_dma_mem_map
- },
-};
-
static int
-is_null_map(const struct user_mem_map *map)
+is_null_map(const struct vfio_user_mem_map *map)
{
return map->addr == 0 && map->iova == 0 &&
map->len == 0 && map->chunk == 0;
@@ -115,8 +72,8 @@ is_null_map(const struct user_mem_map *map)
static int
user_mem_map_cmp(const void *a, const void *b)
{
- const struct user_mem_map *umm_a = a;
- const struct user_mem_map *umm_b = b;
+ const struct vfio_user_mem_map *umm_a = a;
+ const struct vfio_user_mem_map *umm_b = b;
/* move null entries to end */
if (is_null_map(umm_a))
@@ -153,14 +110,14 @@ user_mem_map_cmp(const void *a, const void *b)
* mappings that will be kept.
*/
static int
-process_maps(struct user_mem_map *src, size_t src_len,
- struct user_mem_map newmap[2], uint64_t vaddr, uint64_t len)
+process_maps(struct vfio_user_mem_map *src, size_t src_len,
+ struct vfio_user_mem_map newmap[2], uint64_t vaddr, uint64_t len)
{
- struct user_mem_map *src_first = &src[0];
- struct user_mem_map *src_last = &src[src_len - 1];
- struct user_mem_map *dst_first = &newmap[0];
+ struct vfio_user_mem_map *src_first = &src[0];
+ struct vfio_user_mem_map *src_last = &src[src_len - 1];
+ struct vfio_user_mem_map *dst_first = &newmap[0];
/* we can get at most two new segments */
- struct user_mem_map *dst_last = &newmap[1];
+ struct vfio_user_mem_map *dst_last = &newmap[1];
uint64_t first_off = vaddr - src_first->addr;
uint64_t last_off = (src_last->addr + src_last->len) - (vaddr + len);
int newmap_len = 0;
@@ -175,7 +132,7 @@ process_maps(struct user_mem_map *src, size_t src_len,
}
if (last_off != 0) {
/* if we had start offset, we have two segments */
- struct user_mem_map *last =
+ struct vfio_user_mem_map *last =
first_off == 0 ? dst_first : dst_last;
last->addr = (src_last->addr + src_last->len) - last_off;
last->iova = (src_last->iova + src_last->len) - last_off;
@@ -189,50 +146,50 @@ process_maps(struct user_mem_map *src, size_t src_len,
/* erase certain maps from the list */
static void
-delete_maps(struct user_mem_maps *user_mem_maps, struct user_mem_map *del_maps,
+delete_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, struct vfio_user_mem_map *del_maps,
size_t n_del)
{
unsigned int i;
size_t j;
- for (i = 0, j = 0; i < RTE_DIM(user_mem_maps->maps) && j < n_del; i++) {
- struct user_mem_map *left = &user_mem_maps->maps[i];
- struct user_mem_map *right = &del_maps[j];
+ for (i = 0, j = 0; i < RTE_DIM(vfio_user_mem_maps->maps) && j < n_del; i++) {
+ struct vfio_user_mem_map *left = &vfio_user_mem_maps->maps[i];
+ struct vfio_user_mem_map *right = &del_maps[j];
if (user_mem_map_cmp(left, right) == 0) {
memset(left, 0, sizeof(*left));
j++;
- user_mem_maps->n_maps--;
+ vfio_user_mem_maps->n_maps--;
}
}
}
static void
-copy_maps(struct user_mem_maps *user_mem_maps, struct user_mem_map *add_maps,
+copy_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, struct vfio_user_mem_map *add_maps,
size_t n_add)
{
unsigned int i;
size_t j;
- for (i = 0, j = 0; i < RTE_DIM(user_mem_maps->maps) && j < n_add; i++) {
- struct user_mem_map *left = &user_mem_maps->maps[i];
- struct user_mem_map *right = &add_maps[j];
+ for (i = 0, j = 0; i < RTE_DIM(vfio_user_mem_maps->maps) && j < n_add; i++) {
+ struct vfio_user_mem_map *left = &vfio_user_mem_maps->maps[i];
+ struct vfio_user_mem_map *right = &add_maps[j];
/* insert into empty space */
if (is_null_map(left)) {
memcpy(left, right, sizeof(*left));
j++;
- user_mem_maps->n_maps++;
+ vfio_user_mem_maps->n_maps++;
}
}
}
/* try merging two maps into one, return 1 if succeeded */
static int
-merge_map(struct user_mem_map *left, struct user_mem_map *right)
+merge_map(struct vfio_user_mem_map *left, struct vfio_user_mem_map *right)
{
/* merge the same maps into one */
- if (memcmp(left, right, sizeof(struct user_mem_map)) == 0)
+ if (memcmp(left, right, sizeof(struct vfio_user_mem_map)) == 0)
goto out;
if (left->addr + left->len != right->addr)
@@ -250,13 +207,13 @@ merge_map(struct user_mem_map *left, struct user_mem_map *right)
}
static bool
-addr_is_chunk_aligned(struct user_mem_map *maps, size_t n_maps,
+addr_is_chunk_aligned(struct vfio_user_mem_map *maps, size_t n_maps,
uint64_t vaddr, uint64_t iova)
{
unsigned int i;
for (i = 0; i < n_maps; i++) {
- struct user_mem_map *map = &maps[i];
+ struct vfio_user_mem_map *map = &maps[i];
uint64_t map_va_end = map->addr + map->len;
uint64_t map_iova_end = map->iova + map->len;
uint64_t map_va_off = vaddr - map->addr;
@@ -277,8 +234,8 @@ addr_is_chunk_aligned(struct user_mem_map *maps, size_t n_maps,
}
static int
-find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
- uint64_t iova, uint64_t len, struct user_mem_map *dst,
+find_user_mem_maps(struct vfio_user_mem_maps *vfio_user_mem_maps, uint64_t addr,
+ uint64_t iova, uint64_t len, struct vfio_user_mem_map *dst,
size_t dst_len)
{
uint64_t va_end = addr + len;
@@ -287,8 +244,8 @@ find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
size_t j;
int i, ret;
- for (i = 0, j = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map = &user_mem_maps->maps[i];
+ for (i = 0, j = 0; i < vfio_user_mem_maps->n_maps; i++) {
+ struct vfio_user_mem_map *map = &vfio_user_mem_maps->maps[i];
uint64_t map_va_end = map->addr + map->len;
uint64_t map_iova_end = map->iova + map->len;
@@ -328,313 +285,158 @@ find_user_mem_maps(struct user_mem_maps *user_mem_maps, uint64_t addr,
/* this will sort all user maps, and merge/compact any adjacent maps */
static void
-compact_user_maps(struct user_mem_maps *user_mem_maps)
+compact_user_maps(struct vfio_user_mem_maps *vfio_user_mem_maps)
{
unsigned int i;
- qsort(user_mem_maps->maps, RTE_DIM(user_mem_maps->maps),
- sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
+ qsort(vfio_user_mem_maps->maps, RTE_DIM(vfio_user_mem_maps->maps),
+ sizeof(vfio_user_mem_maps->maps[0]), user_mem_map_cmp);
/* we'll go over the list backwards when merging */
- for (i = RTE_DIM(user_mem_maps->maps) - 2; i != 0; i--) {
- struct user_mem_map *l, *r;
+ for (i = RTE_DIM(vfio_user_mem_maps->maps) - 2; i != 0; i--) {
+ struct vfio_user_mem_map *l, *r;
- l = &user_mem_maps->maps[i];
- r = &user_mem_maps->maps[i + 1];
+ l = &vfio_user_mem_maps->maps[i];
+ r = &vfio_user_mem_maps->maps[i + 1];
if (is_null_map(l) || is_null_map(r))
continue;
/* try and merge the maps */
if (merge_map(l, r))
- user_mem_maps->n_maps--;
+ vfio_user_mem_maps->n_maps--;
}
/* the entries are still sorted, but now they have holes in them, so
* sort the list again.
*/
- qsort(user_mem_maps->maps, RTE_DIM(user_mem_maps->maps),
- sizeof(user_mem_maps->maps[0]), user_mem_map_cmp);
+ qsort(vfio_user_mem_maps->maps, RTE_DIM(vfio_user_mem_maps->maps),
+ sizeof(vfio_user_mem_maps->maps[0]), user_mem_map_cmp);
}
-static int
-vfio_open_group_fd(int iommu_group_num, bool mp_request)
+bool
+vfio_container_is_default(struct vfio_container *cfg)
{
- int vfio_group_fd;
- char filename[PATH_MAX];
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open the group locally */
- if (!mp_request) {
- /* try regular group format */
- snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- /* if file not found, it's not an error */
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
-
- /* special case: try no-IOMMU path as well */
- snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT,
- iommu_group_num);
- vfio_group_fd = open(filename, O_RDWR);
- if (vfio_group_fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR,
- "Cannot open %s: %s",
- filename, strerror(errno));
- return -1;
- }
- return -ENOENT;
- }
- /* noiommu group found */
- }
-
- return vfio_group_fd;
- }
- /* if we're in a secondary process, request group fd from the primary
- * process via mp channel.
- */
- p->req = SOCKET_REQ_GROUP;
- p->group_num = iommu_group_num;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_group_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_group_fd = mp_rep->fds[0];
- } else if (p->result == SOCKET_NO_FD) {
- EAL_LOG(ERR, "Bad VFIO group fd");
- vfio_group_fd = -ENOENT;
- }
- }
-
- free(mp_reply.msgs);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- EAL_LOG(ERR, "Cannot request VFIO group fd");
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_num(int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++) {
- if (vfio_cfg->vfio_groups[j].group_num ==
- iommu_group_num)
- return vfio_cfg;
- }
- }
-
- return NULL;
-}
-
-static int
-vfio_get_group_fd(struct vfio_config *vfio_cfg,
- int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- int vfio_group_fd;
- unsigned int i;
-
- /* check if we already have the group descriptor open */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num)
- return vfio_cfg->vfio_groups[i].fd;
-
- /* Lets see first if there is room for a new group */
- if (vfio_cfg->vfio_active_groups == RTE_DIM(vfio_cfg->vfio_groups)) {
- EAL_LOG(ERR, "Maximum number of VFIO groups reached!");
- return -1;
- }
-
- /* Now lets get an index for the new group */
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num == -1) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
- }
-
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "No VFIO group free slot found");
- return -1;
- }
-
- /*
- * When opening a group fd, we need to decide whether to open it locally
- * or request it from the primary process via mp_sync.
- *
- * For the default container, secondary processes use mp_sync so that
- * the primary process tracks the group fd and maintains VFIO state
- * across all processes.
- *
- * For custom containers, we open the group fd locally in each process
- * since custom containers are process-local and the primary has no
- * knowledge of them. Requesting a group fd from the primary for a
- * container it doesn't know about would be incorrect.
- */
- const struct internal_config *internal_conf = eal_get_internal_configuration();
- bool mp_request = (internal_conf->process_type == RTE_PROC_SECONDARY) &&
- (vfio_cfg == default_vfio_cfg);
-
- vfio_group_fd = vfio_open_group_fd(iommu_group_num, mp_request);
- if (vfio_group_fd < 0) {
- EAL_LOG(ERR, "Failed to open VFIO group %d",
- iommu_group_num);
- return vfio_group_fd;
- }
-
- cur_grp->group_num = iommu_group_num;
- cur_grp->fd = vfio_group_fd;
- vfio_cfg->vfio_active_groups++;
-
- return vfio_group_fd;
-}
-
-static struct vfio_config *
-get_vfio_cfg_by_group_fd(int vfio_group_fd)
-{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return vfio_cfg;
- }
-
- return NULL;
+ return cfg == vfio_global_cfg.default_cfg;
}
-static struct vfio_config *
-get_vfio_cfg_by_container_fd(int container_fd)
+static struct vfio_container *
+vfio_container_get_by_fd(int container_fd)
{
- unsigned int i;
+ struct vfio_container *cfg;
if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD)
- return default_vfio_cfg;
+ return vfio_global_cfg.default_cfg;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == container_fd)
- return &vfio_cfgs[i];
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ if (cfg->container_fd == container_fd)
+ return cfg;
}
-
return NULL;
}
-int
-vfio_get_group_fd_by_num(int iommu_group_num)
+static struct vfio_container *
+vfio_container_get_by_group_num(int group_num)
{
- struct vfio_config *vfio_cfg;
+ struct vfio_container *cfg;
+ struct vfio_group *grp;
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp)
+ if (grp->group_num == group_num)
+ return cfg;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
+ return NULL;
}
-static int
-get_vfio_group_idx(int vfio_group_fd)
+static struct vfio_container *
+vfio_container_create(void)
{
- struct vfio_config *vfio_cfg;
- unsigned int i, j;
-
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfg = &vfio_cfgs[i];
- for (j = 0; j < RTE_DIM(vfio_cfg->vfio_groups); j++)
- if (vfio_cfg->vfio_groups[j].fd == vfio_group_fd)
- return j;
+ struct vfio_container *cfg;
+
+ /* find an unused container config */
+ VFIO_CONTAINER_FOREACH(cfg) {
+ if (!cfg->active) {
+ *cfg = VFIO_CONTAINER_INITIALIZER;
+ cfg->active = true;
+ return cfg;
+ }
}
-
- return -1;
+ /* no space */
+ return NULL;
}
static void
-vfio_group_device_get(int vfio_group_fd)
+vfio_container_erase(struct vfio_container *cfg)
{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
- }
+ if (cfg->container_fd >= 0 && close(cfg->container_fd))
+ EAL_LOG(ERR, "Error when closing container, %d (%s)", errno, strerror(errno));
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices++;
+ *cfg = VFIO_CONTAINER_INITIALIZER;
}
-static void
-vfio_group_device_put(int vfio_group_fd)
+static struct vfio_device *
+vfio_device_create(struct vfio_container *cfg, enum vfio_device_type type)
{
- struct vfio_config *vfio_cfg;
- int i;
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return;
+ struct vfio_device *dev;
+
+ /* is there space? */
+ if (cfg->n_devices == RTE_DIM(cfg->devices))
+ return NULL;
+
+ VFIO_DEVICE_FOREACH(cfg, dev) {
+ if (dev->active)
+ continue;
+ dev->active = true;
+ dev->type = type;
+ /* set to invalid fd */
+ dev->fd = -1;
+
+ cfg->n_devices++;
+ return dev;
}
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- else
- vfio_cfg->vfio_groups[i].devices--;
+ /* should not happen */
+ EAL_LOG(WARNING, "Could not find space in device list for container");
+ return NULL;
}
-static int
-vfio_group_device_count(int vfio_group_fd)
+static void
+vfio_device_erase(struct vfio_container *cfg, struct vfio_device *dev)
{
- struct vfio_config *vfio_cfg;
- int i;
+ if (dev->fd >= 0 && close(dev->fd))
+ EAL_LOG(ERR, "Error when closing device, %d (%s)", errno, strerror(errno));
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
- return -1;
+ if (dev->type == VFIO_DEVICE_TYPE_GROUP) {
+ free(dev->sysfs_base);
+ free(dev->dev_addr);
}
+ *dev = (struct vfio_device){0};
+ cfg->n_devices--;
+}
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0) {
- EAL_LOG(ERR, "Wrong VFIO group index (%d)", i);
- return -1;
+static struct vfio_device *
+vfio_device_get_by_addr(struct vfio_container *cfg,
+ const char *sysfs_base,
+ const char *dev_addr)
+{
+ struct vfio_device *dev;
+
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ /* only group mode have sysfs_base and dev_addr */
+ if (dev->type != VFIO_DEVICE_TYPE_GROUP)
+ continue;
+ if (strcmp(dev->sysfs_base, sysfs_base) == 0 &&
+ strcmp(dev->dev_addr, dev_addr) == 0)
+ return dev;
}
-
- return vfio_cfg->vfio_groups[i].devices;
+ return NULL;
}
static void
vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
void *arg __rte_unused)
{
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
struct rte_memseg_list *msl;
struct rte_memseg *ms;
size_t cur_len = 0;
@@ -649,11 +451,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
/* Maintain granularity of DMA map/unmap to memseg size */
for (; cur_len < len; cur_len += page_sz) {
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 1);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, vfio_va,
- vfio_va, page_sz, 0);
+ vfio_dma_mem_map(cfg, vfio_va, vfio_va, page_sz, 0);
vfio_va += page_sz;
}
@@ -671,11 +471,9 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
goto next;
}
if (type == RTE_MEM_EVENT_ALLOC)
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 1);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
else
- vfio_dma_mem_map(default_vfio_cfg, ms->addr_64,
- ms->iova, ms->len, 0);
+ vfio_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 0);
next:
cur_len += ms->len;
++ms;
@@ -683,91 +481,279 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,
}
static int
-vfio_sync_default_container(void)
+vfio_register_mem_event_callback(void)
{
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
- int iommu_type_id;
- unsigned int i;
+ int ret;
- /* cannot be called from primary */
- if (rte_eal_process_type() != RTE_PROC_SECONDARY)
- return -1;
+ ret = rte_mem_event_callback_register(VFIO_MEM_EVENT_CLB_NAME,
+ vfio_mem_event_callback, NULL);
- /* default container fd should have been opened in dev_vfio_enable() */
- if (!vfio_enabled ||
- default_vfio_cfg->vfio_container_fd < 0) {
- EAL_LOG(ERR, "VFIO support is not initialized");
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Could not install memory event callback for VFIO");
return -1;
}
+ if (ret)
+ EAL_LOG(DEBUG, "Memory event callbacks not supported");
+ else
+ EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- /* find default container's IOMMU type */
- p->req = SOCKET_REQ_IOMMU_TYPE;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- iommu_type_id = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK)
- iommu_type_id = p->iommu_type_id;
- }
- free(mp_reply.msgs);
- if (iommu_type_id < 0) {
- EAL_LOG(ERR,
- "Could not get IOMMU type for default container");
- return -1;
- }
-
- /* we now have an fd for default container, as well as its IOMMU type.
- * now, set up default VFIO container config to match.
- */
- for (i = 0; i < RTE_DIM(iommu_types); i++) {
- const struct vfio_iommu_type *t = &iommu_types[i];
- if (t->type_id != iommu_type_id)
- continue;
-
- /* we found our IOMMU type */
- default_vfio_cfg->vfio_iommu_type = t;
-
- return 0;
- }
- EAL_LOG(ERR, "Could not find IOMMU type id (%i)",
- iommu_type_id);
- return -1;
+ return 0;
}
static int
-vfio_clear_group(int vfio_group_fd)
+vfio_setup_dma_mem(struct vfio_container *cfg)
{
- int i;
- struct vfio_config *vfio_cfg;
+ struct vfio_user_mem_maps *vfio_user_mem_maps = &cfg->mem_maps;
+ int i, ret;
- if (!vfio_enabled) {
+ /* do we need to map DPDK-managed memory? */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() == RTE_PROC_PRIMARY)
+ ret = vfio_global_cfg.ops->dma_map_func(cfg);
+ else
+ ret = 0;
+ if (ret) {
+ EAL_LOG(ERR, "DMA remapping failed, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * not all IOMMU types support DMA mapping, but if we have mappings in the list - that
+ * means we have previously mapped something successfully, so we can be sure that DMA
+ * mapping is supported.
+ */
+ for (i = 0; i < vfio_user_mem_maps->n_maps; i++) {
+ struct vfio_user_mem_map *map;
+ map = &vfio_user_mem_maps->maps[i];
+
+ ret = vfio_global_cfg.ops->dma_user_map_func(cfg, map->addr, map->iova, map->len, 1);
+ if (ret) {
+ EAL_LOG(ERR, "Couldn't map user memory for DMA: "
+ "va: 0x%" PRIx64 " "
+ "iova: 0x%" PRIx64 " "
+ "len: 0x%" PRIu64,
+ map->addr, map->iova,
+ map->len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static enum vfio_result
+vfio_group_assign_device(struct vfio_container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+ struct vfio_device *dev;
+ char *sysfs_dup, *dev_dup;
+ int iommu_group_num;
+ enum vfio_result res;
+ int ret;
+
+ /* if this device is already assigned, reuse it instead of opening a second fd */
+ dev = vfio_device_get_by_addr(cfg, sysfs_base, dev_addr);
+ if (dev != NULL) {
+ *out_dev = dev;
+ return VFIO_EXISTS;
+ }
+
+ sysfs_dup = strdup(sysfs_base);
+ dev_dup = strdup(dev_addr);
+ if (sysfs_dup == NULL || dev_dup == NULL) {
+ EAL_LOG(ERR, "Cannot allocate memory for device %s", dev_addr);
+ free(sysfs_dup);
+ free(dev_dup);
+ return VFIO_NO_MEM;
+ }
+
+ /* allocate new device in config */
+ dev = vfio_device_create(cfg, VFIO_DEVICE_TYPE_GROUP);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO device");
+ return VFIO_NO_SPACE;
+ }
+
+ /* store sysfs path and device address */
+ dev->sysfs_base = sysfs_dup;
+ dev->dev_addr = dev_dup;
+
+ /* remember to register mem event callback for default container in primary */
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* get group number for this device */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot get IOMMU group for %s", dev_addr);
+ res = VFIO_ERROR;
+ goto device_erase;
+ } else if (ret == 0) {
+ res = VFIO_NOT_MANAGED;
+ goto device_erase;
+ }
+
+ /* group may already exist as multiple devices may share group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* no device currently uses this group, create it */
+ grp = vfio_group_create(cfg, iommu_group_num);
+ if (grp == NULL) {
+ EAL_LOG(ERR, "Cannot allocate group for device %s", dev_addr);
+ res = VFIO_NO_SPACE;
+ goto device_erase;
+ }
+
+ /* open group fd */
+ ret = vfio_group_open_fd(cfg, grp);
+ if (ret == -ENOENT) {
+ EAL_LOG(DEBUG, "Device %s (IOMMU group %d) not managed by VFIO",
+ dev_addr, iommu_group_num);
+ res = VFIO_NOT_MANAGED;
+ goto group_erase;
+ } else if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO group %d for device %s",
+ iommu_group_num, dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* prepare group (viability + container attach) */
+ ret = vfio_group_prepare(cfg, grp);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /* set up IOMMU type once per container */
+ if (!group_cfg->iommu_type_set) {
+ ret = vfio_group_setup_iommu(cfg);
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->iommu_type_set = true;
+ }
+
+ /* set up DMA memory once per container */
+ if (!cfg->dma_setup_done) {
+ rte_spinlock_recursive_lock(&cfg->mem_maps.lock);
+ ret = vfio_setup_dma_mem(cfg);
+ rte_spinlock_recursive_unlock(&cfg->mem_maps.lock);
+ if (ret < 0) {
+ EAL_LOG(ERR, "DMA remapping for %s failed", dev_addr);
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ cfg->dma_setup_done = true;
+ }
+
+ /* set up mem event callback if needed */
+ if (need_clb && !group_cfg->mem_event_clb_set) {
+ ret = vfio_register_mem_event_callback();
+ if (ret < 0) {
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+ group_cfg->mem_event_clb_set = true;
+ }
+ }
+
+ /* open dev fd */
+ ret = vfio_group_setup_device_fd(dev_addr, grp, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot open VFIO device %s, error %i (%s)",
+ dev_addr, errno, strerror(errno));
+ res = VFIO_ERROR;
+ goto group_erase;
+ }
+
+ /*
+ * we do not need to look in other configs: if we were to attempt to use
+ * a different container, the kernel wouldn't have allowed us to bind the
+ * group to the container in the first place.
+ */
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+group_erase:
+ /* this may be a pre-existing group so only erase it if it has no devices */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+ /* if we registered callback, unregister it */
+ if (group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+device_erase:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
+int
+dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const char *dev_addr)
+{
+ struct vfio_container *cfg;
+ enum vfio_result res;
+ struct vfio_device *dev;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
+ return -1;
+ }
+ /* protect memory configuration while setting up IOMMU/DMA */
+ rte_mcfg_mem_read_lock();
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ res = VFIO_NOT_SUPPORTED;
+ break;
+ }
+ rte_mcfg_mem_read_unlock();
+
+ switch (res) {
+ case VFIO_SUCCESS:
+ return 0;
+ case VFIO_EXISTS:
+ rte_errno = EEXIST;
+ return -1;
+ case VFIO_NOT_MANAGED:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
rte_errno = ENODEV;
return -1;
- }
-
- vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO group fd!");
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
return -1;
- }
-
- i = get_vfio_group_idx(vfio_group_fd);
- if (i < 0)
+ case VFIO_NO_MEM:
+ EAL_LOG(ERR, "Memory allocation failed for device %s", dev_addr);
+ rte_errno = ENOMEM;
return -1;
- vfio_cfg->vfio_groups[i].group_num = -1;
- vfio_cfg->vfio_groups[i].fd = -1;
- vfio_cfg->vfio_groups[i].devices = 0;
- vfio_cfg->vfio_active_groups--;
-
- return 0;
+ default:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ return -1;
+ }
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_setup_device)
@@ -775,374 +761,280 @@ int
dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
int *vfio_dev_fd)
{
- struct vfio_group_status group_status = {
- .argsz = sizeof(group_status)
- };
- struct vfio_config *vfio_cfg;
- struct user_mem_maps *user_mem_maps;
- int vfio_container_fd;
- int vfio_group_fd;
- int iommu_group_num;
- rte_uuid_t vf_token;
- int i, ret;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
-
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
- }
-
- /* get group number */
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret == 0) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /* if negative, something failed */
- if (ret < 0)
- return -1;
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0 && vfio_group_fd != -ENOENT)
- return -1;
-
- /*
- * if vfio_group_fd == -ENOENT, that means the device
- * isn't managed by VFIO
- */
- if (vfio_group_fd == -ENOENT) {
- EAL_LOG(NOTICE,
- "%s not managed by VFIO driver, skipping",
- dev_addr);
- return 1;
- }
-
- /*
- * at this point, we know that this group is viable (meaning, all devices
- * are either bound to VFIO or not bound to anything)
- */
-
- /* check if the group is viable */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status);
- if (ret) {
- EAL_LOG(ERR, "%s cannot get VFIO group status, "
- "error %i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) {
- EAL_LOG(ERR, "%s VFIO group is not viable! "
- "Not all devices in IOMMU group bound to VFIO or unbound",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
- vfio_container_fd = vfio_cfg->vfio_container_fd;
- user_mem_maps = &vfio_cfg->mem_maps;
-
- /* check if group does not have a container yet */
- if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) {
-
- /* add group to a container */
- ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER,
- &vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "%s cannot add VFIO group to container, error "
- "%i (%s)", dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /*
- * pick an IOMMU type and set up DMA mappings for container
- *
- * needs to be done only once, only when first group is
- * assigned to a container and only in primary process.
- * Note this can happen several times with the hotplug
- * functionality.
- */
- if (internal_conf->process_type == RTE_PROC_PRIMARY &&
- vfio_cfg->vfio_active_groups == 1 &&
- vfio_group_device_count(vfio_group_fd) == 0) {
- const struct vfio_iommu_type *t;
-
- /* select an IOMMU type which we will be using */
- t = vfio_set_iommu_type(vfio_container_fd);
- if (!t) {
- EAL_LOG(ERR,
- "%s failed to select IOMMU type",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* lock memory hotplug before mapping and release it
- * after registering callback, to prevent races
- */
- rte_mcfg_mem_read_lock();
- if (vfio_cfg == default_vfio_cfg)
- ret = t->dma_map_func(vfio_container_fd);
- else
- ret = 0;
- if (ret) {
- EAL_LOG(ERR,
- "%s DMA remapping failed, error "
- "%i (%s)",
- dev_addr, errno, strerror(errno));
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
-
- vfio_cfg->vfio_iommu_type = t;
-
- /* re-map all user-mapped segments */
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
-
- /* this IOMMU type may not support DMA mapping, but
- * if we have mappings in the list - that means we have
- * previously mapped something successfully, so we can
- * be sure that DMA mapping is supported.
- */
- for (i = 0; i < user_mem_maps->n_maps; i++) {
- struct user_mem_map *map;
- map = &user_mem_maps->maps[i];
-
- ret = t->dma_user_map_func(
- vfio_container_fd,
- map->addr, map->iova, map->len,
- 1);
- if (ret) {
- EAL_LOG(ERR, "Couldn't map user memory for DMA: "
- "va: 0x%" PRIx64 " "
- "iova: 0x%" PRIx64 " "
- "len: 0x%" PRIu64,
- map->addr, map->iova,
- map->len);
- rte_spinlock_recursive_unlock(
- &user_mem_maps->lock);
- rte_mcfg_mem_read_unlock();
- return -1;
- }
- }
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
-
- /* register callback for mem events */
- if (vfio_cfg == default_vfio_cfg)
- ret = rte_mem_event_callback_register(
- VFIO_MEM_EVENT_CLB_NAME,
- vfio_mem_event_callback, NULL);
- else
- ret = 0;
- /* unlock memory hotplug */
- rte_mcfg_mem_read_unlock();
-
- if (ret && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Could not install memory event callback for VFIO");
- return -1;
- }
- if (ret)
- EAL_LOG(DEBUG, "Memory event callbacks not supported");
- else
- EAL_LOG(DEBUG, "Installed memory event callback for VFIO");
- }
- } else if (rte_eal_process_type() != RTE_PROC_PRIMARY &&
- vfio_cfg == default_vfio_cfg &&
- vfio_cfg->vfio_iommu_type == NULL) {
- /* if we're not a primary process, we do not set up the VFIO
- * container because it's already been set up by the primary
- * process. instead, we simply ask the primary about VFIO type
- * we are using, and set the VFIO config up appropriately.
- */
- ret = vfio_sync_default_container();
- if (ret < 0) {
- EAL_LOG(ERR, "Could not sync default VFIO container");
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
- /* we have successfully initialized VFIO, notify user */
- const struct vfio_iommu_type *t =
- default_vfio_cfg->vfio_iommu_type;
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- }
-
- rte_eal_vfio_get_vf_token(vf_token);
-
- /* get a file descriptor for the device with VF token firstly */
- if (!rte_uuid_is_null(vf_token)) {
- char vf_token_str[RTE_UUID_STRLEN];
- char dev[PATH_MAX];
-
- rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
- snprintf(dev, sizeof(dev),
- "%s vf_token=%s", dev_addr, vf_token_str);
-
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD,
- dev);
- if (*vfio_dev_fd >= 0)
- goto out;
- }
-
- /* get a file descriptor for the device */
- *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
- if (*vfio_dev_fd < 0) {
- /* if we cannot get a device fd, this implies a problem with
- * the VFIO group or the container not having IOMMU configured.
- */
-
- EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed",
- dev_addr);
- close(vfio_group_fd);
- vfio_clear_group(vfio_group_fd);
- return -1;
- }
-
- /* device is now set up */
-out:
- vfio_group_device_get(vfio_group_fd);
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
-int
-dev_vfio_release_device(const char *sysfs_base, const char *dev_addr,
- int vfio_dev_fd)
-{
- struct vfio_config *vfio_cfg;
- int vfio_group_fd;
- int iommu_group_num;
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+ enum vfio_result res;
int ret;
- if (!vfio_enabled) {
- rte_errno = ENODEV;
+ if (sysfs_base == NULL || dev_addr == NULL || vfio_dev_fd == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
}
- /* we don't want any DMA mapping messages to come while we're detaching
- * VFIO device, because this might be the last device and we might need
- * to unregister the callback.
- */
rte_mcfg_mem_read_lock();
- /* get group number */
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret <= 0) {
- EAL_LOG(WARNING, "%s not managed by VFIO driver",
- dev_addr);
- /* This is an error at this point. */
- ret = -1;
- goto out;
- }
-
- /* get the actual group fd */
- vfio_group_fd = vfio_get_group_fd_by_num(iommu_group_num);
- if (vfio_group_fd < 0) {
- EAL_LOG(INFO, "vfio_get_group_fd_by_num failed for %s",
- dev_addr);
- ret = vfio_group_fd;
- goto out;
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num;
+
+ /* find group number */
+ ret = vfio_group_get_num(sysfs_base, dev_addr, &iommu_group_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ /* find config by group */
+ cfg = vfio_container_get_by_group_num(iommu_group_num);
+ if (cfg == NULL)
+ cfg = vfio_global_cfg.default_cfg;
+
+ res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
}
-
- /* get the vfio_config it belongs to */
- vfio_cfg = get_vfio_cfg_by_group_num(iommu_group_num);
- vfio_cfg = vfio_cfg ? vfio_cfg : default_vfio_cfg;
-
- /* At this point we got an active group. Closing it will make the
- * container detachment. If this is the last active group, VFIO kernel
- * code will unset the container and the IOMMU mappings.
- */
-
- /* Closing a device */
- if (close(vfio_dev_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_dev_fd for %s",
- dev_addr);
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
ret = -1;
- goto out;
+ goto unlock;
}
- /* An VFIO group can have several devices attached. Just when there is
- * no devices remaining should the group be closed.
- */
- vfio_group_device_put(vfio_group_fd);
- if (!vfio_group_device_count(vfio_group_fd)) {
-
- if (close(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when closing vfio_group_fd for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
-
- if (vfio_clear_group(vfio_group_fd) < 0) {
- EAL_LOG(INFO, "Error when clearing group for %s",
- dev_addr);
- ret = -1;
- goto out;
- }
+ switch (res) {
+ case VFIO_NOT_MANAGED:
+not_managed:
+ EAL_LOG(DEBUG, "Device %s not managed by VFIO", dev_addr);
+ rte_errno = ENODEV;
+ ret = -1;
+ goto unlock;
+ case VFIO_SUCCESS:
+ case VFIO_EXISTS:
+ break;
+ case VFIO_NO_SPACE:
+ EAL_LOG(ERR, "No space in VFIO container to assign device %s", dev_addr);
+ rte_errno = ENOSPC;
+ ret = -1;
+ goto unlock;
+ case VFIO_NO_MEM:
+ EAL_LOG(ERR, "Memory allocation failed for device %s", dev_addr);
+ rte_errno = ENOMEM;
+ ret = -1;
+ goto unlock;
+ default:
+assign_fail:
+ EAL_LOG(ERR, "Error assigning device %s to container", dev_addr);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
}
-
- /* if there are no active device groups, unregister the callback to
- * avoid spurious attempts to map/unmap memory from VFIO.
- */
- if (vfio_cfg == default_vfio_cfg && vfio_cfg->vfio_active_groups == 0 &&
- rte_eal_process_type() != RTE_PROC_SECONDARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME,
- NULL);
+ *vfio_dev_fd = dev->fd;
/* success */
ret = 0;
-out:
+unlock:
rte_mcfg_mem_read_unlock();
+
return ret;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_release_device)
+int
+dev_vfio_release_device(const char *sysfs_base __rte_unused,
+ const char *dev_addr, int vfio_dev_fd)
+{
+ struct vfio_container *cfg = NULL, *icfg;
+ struct vfio_device *dev = NULL, *idev;
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ rte_mcfg_mem_read_lock();
+
+ /* we need to find both config and device */
+ VFIO_CONTAINER_FOREACH_ACTIVE(icfg) {
+ VFIO_DEVICE_FOREACH_ACTIVE(icfg, idev) {
+ if (idev->fd != vfio_dev_fd)
+ continue;
+ cfg = icfg;
+ dev = idev;
+ goto found;
+ }
+ }
+found:
+ if (dev == NULL) {
+ EAL_LOG(ERR, "Device %s not managed by any container", dev_addr);
+ rte_errno = ENOENT;
+ ret = -1;
+ goto unlock;
+ }
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ int iommu_group_num = dev->group;
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_group *grp;
+
+ bool need_clb = vfio_container_is_default(cfg) &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY;
+
+ /* find the group */
+ grp = vfio_group_get_by_num(cfg, iommu_group_num);
+ if (grp == NULL) {
+ /* shouldn't happen because we already know the device is valid */
+ EAL_LOG(ERR, "IOMMU group %d not found in container",
+ iommu_group_num);
+ rte_errno = EIO;
+ ret = -1;
+ goto unlock;
+ }
+
+ /* close device handle */
+ vfio_device_erase(cfg, dev);
+
+ /* remove device from group */
+ grp->n_devices--;
+
+ /* was this the last device? */
+ if (grp->n_devices == 0)
+ vfio_group_erase(cfg, grp);
+
+ /* if no more groups left, remove callback */
+ if (need_clb && group_cfg->n_groups == 0 && group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+ break;
+ }
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ ret = -1;
+ goto unlock;
+ }
+ ret = 0;
+unlock:
+ rte_mcfg_mem_read_unlock();
+
+ return ret;
+}
+
+static int
+vfio_sync_mode(struct vfio_container *cfg, enum dev_vfio_mode *mode)
+{
+ struct vfio_mp_param *p;
+ struct rte_mp_msg mp_req = {0};
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {5, 0};
+
+ /* request container from primary via mp_sync */
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+ p = (struct vfio_mp_param *)mp_req.param;
+ p->req = VFIO_SOCKET_REQ_CONTAINER;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ struct rte_mp_msg *mp_rep;
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
+ cfg->container_fd = mp_rep->fds[0];
+ *mode = p->mode;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request container_fd");
+ return -1;
+}
+
+static enum dev_vfio_mode
+vfio_select_mode(void)
+{
+ struct vfio_container *cfg;
+ enum dev_vfio_mode mode = DEV_VFIO_MODE_NONE;
+
+ cfg = vfio_container_create();
+ /* cannot happen */
+ if (cfg == NULL || cfg != vfio_global_cfg.default_cfg) {
+ EAL_LOG(ERR, "Unexpected VFIO config structure");
+ return DEV_VFIO_MODE_NONE;
+ }
+
+ /* for secondary, just ask the primary for the container and mode */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ if (vfio_sync_mode(cfg, &mode) < 0)
+ goto err;
+
+ /* primary handles DMA setup for default containers */
+ cfg->dma_setup_done = true;
+ return (enum dev_vfio_mode)mode;
+ }
+ /* if we failed mp sync setup, we cannot initialize VFIO */
+ if (vfio_mp_sync_setup() < 0)
+ return DEV_VFIO_MODE_NONE;
+
+ /* try group mode first */
+ if (vfio_group_enable(cfg) == 0) {
+ /* check for noiommu */
+ int ret = vfio_group_noiommu_is_enabled();
+ if (ret < 0)
+ goto err_mpsync;
+ else if (ret == 1)
+ return DEV_VFIO_MODE_NOIOMMU;
+ return DEV_VFIO_MODE_GROUP;
+ }
+err_mpsync:
+ vfio_mp_sync_cleanup();
+err:
+ vfio_container_erase(cfg);
+
+ return DEV_VFIO_MODE_NONE;
+}
+
+static const char *
+vfio_mode_to_str(enum dev_vfio_mode mode)
+{
+ switch (mode) {
+ case DEV_VFIO_MODE_GROUP: return "group";
+ case DEV_VFIO_MODE_NOIOMMU: return "noiommu";
+ default: return "not initialized";
+ }
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_enable)
int
dev_vfio_enable(void)
{
- /* initialize group list */
- unsigned int i, j;
int vfio_available;
- DIR *dir;
- const struct internal_config *internal_conf =
- eal_get_internal_configuration();
+ enum dev_vfio_mode mode = DEV_VFIO_MODE_NONE;
- rte_spinlock_recursive_t lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER;
-
- if (vfio_enabled)
+ /* if already enabled, nothing to do */
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_NONE)
return 0;
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- vfio_cfgs[i].enabled = false;
- vfio_cfgs[i].vfio_container_fd = -1;
- vfio_cfgs[i].vfio_active_groups = 0;
- vfio_cfgs[i].vfio_iommu_type = NULL;
- vfio_cfgs[i].mem_maps.lock = lock;
-
- for (j = 0; j < RTE_DIM(vfio_cfgs[i].vfio_groups); j++) {
- vfio_cfgs[i].vfio_groups[j].fd = -1;
- vfio_cfgs[i].vfio_groups[j].group_num = -1;
- vfio_cfgs[i].vfio_groups[j].devices = 0;
- }
- }
-
EAL_LOG(DEBUG, "Probing VFIO support...");
/* check if vfio module is loaded */
@@ -1160,37 +1052,16 @@ dev_vfio_enable(void)
"VFIO modules not loaded, skipping VFIO support...");
return 0;
}
+ EAL_LOG(DEBUG, "VFIO module '%s' loaded, attempting to initialize VFIO...", VFIO_MODNAME);
+ mode = vfio_select_mode();
- /* VFIO directory might not exist (e.g., unprivileged containers) */
- dir = opendir(DEV_VFIO_DIR);
- if (dir == NULL) {
- EAL_LOG(DEBUG,
- "VFIO directory does not exist, skipping VFIO support...");
- return 0;
- }
- closedir(dir);
-
- if (internal_conf->process_type == RTE_PROC_PRIMARY) {
- if (vfio_mp_sync_setup() == -1) {
- default_vfio_cfg->vfio_container_fd = -1;
- } else {
- /* open a default container */
- default_vfio_cfg->vfio_container_fd = vfio_open_container_fd(false);
- }
- } else {
- /* get the default container from the primary process */
- default_vfio_cfg->vfio_container_fd =
- vfio_open_container_fd(true);
- }
-
- /* check if we have VFIO driver enabled */
- if (default_vfio_cfg->vfio_container_fd != -1) {
- EAL_LOG(INFO, "VFIO support initialized");
- default_vfio_cfg->enabled = true;
- vfio_enabled = true;
- } else {
+ /* have we initialized anything? */
+ if (mode == DEV_VFIO_MODE_NONE)
EAL_LOG(NOTICE, "VFIO support could not be initialized");
- }
+ else
+ EAL_LOG(NOTICE, "VFIO support initialized: %s mode", vfio_mode_to_str(mode));
+
+ vfio_global_cfg.mode = mode;
return 0;
}
@@ -1199,43 +1070,17 @@ RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
dev_vfio_is_enabled(const char *modname)
{
- const int mod_available = rte_eal_check_module(modname) > 0;
- return vfio_enabled && mod_available;
+ const int mod_available = modname ? rte_eal_check_module(modname) > 0 : 0;
+ return vfio_global_cfg.default_cfg->active && mod_available;
}
int
vfio_get_iommu_type(void)
{
- if (!vfio_enabled)
+ if (vfio_global_cfg.ops == NULL)
return -1;
- if (default_vfio_cfg->vfio_iommu_type == NULL)
- return -1;
-
- return default_vfio_cfg->vfio_iommu_type->type_id;
-}
-
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd)
-{
- unsigned idx;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- t->type_id);
- if (!ret) {
- EAL_LOG(INFO, "Using IOMMU type %d (%s)",
- t->type_id, t->name);
- return t;
- }
- /* not an error, there may be more supported IOMMU types */
- EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
- "%i (%s)", t->type_id, t->name, errno,
- strerror(errno));
- }
- /* if we didn't find a suitable IOMMU type, fail */
- return NULL;
+ return vfio_global_cfg.ops->type_id;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_info)
@@ -1244,131 +1089,27 @@ dev_vfio_get_device_info(int vfio_dev_fd, struct vfio_device_info *device_info)
{
int ret;
- if (!vfio_enabled) {
- rte_errno = ENODEV;
+ if (device_info == NULL) {
+ rte_errno = EINVAL;
return -1;
}
- if (device_info == NULL || vfio_dev_fd < 0)
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
return -1;
+ }
ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info);
if (ret) {
- EAL_LOG(ERR, "Cannot get device info, error %i (%s)",
- errno, strerror(errno));
+ EAL_LOG(ERR, "Cannot get device info, error %d (%s)", errno, strerror(errno));
+ rte_errno = errno;
return -1;
}
return 0;
}
-int
-vfio_has_supported_extensions(int vfio_container_fd)
-{
- int ret;
- unsigned idx, n_extensions = 0;
- for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
- const struct vfio_iommu_type *t = &iommu_types[idx];
-
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
- t->type_id);
- if (ret < 0) {
- EAL_LOG(ERR, "Could not get IOMMU type, error "
- "%i (%s)", errno, strerror(errno));
- close(vfio_container_fd);
- return -1;
- } else if (ret == 1) {
- /* we found a supported extension */
- n_extensions++;
- }
- EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
- t->type_id, t->name,
- ret ? "supported" : "not supported");
- }
-
- /* if we didn't find any supported IOMMU types, fail */
- if (!n_extensions) {
- close(vfio_container_fd);
- return -1;
- }
-
- return 0;
-}
-
-/*
- * Open a new VFIO container fd.
- *
- * If mp_request is true, requests a new container fd from the primary process
- * via mp channel (for secondary processes that need to open the default container).
- *
- * Otherwise, opens a new container fd locally by opening /dev/vfio/vfio.
- */
-int
-vfio_open_container_fd(bool mp_request)
-{
- int ret, vfio_container_fd;
- struct rte_mp_msg mp_req, *mp_rep;
- struct rte_mp_reply mp_reply = {0};
- struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
- struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
-
- /* if not requesting via mp, open a new container locally */
- if (!mp_request) {
- vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
- if (vfio_container_fd < 0) {
- EAL_LOG(ERR, "Cannot open VFIO container %s, error %i (%s)",
- DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
- return -1;
- }
-
- /* check VFIO API version */
- ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
- if (ret != VFIO_API_VERSION) {
- if (ret < 0)
- EAL_LOG(ERR,
- "Could not get VFIO API version, error "
- "%i (%s)", errno, strerror(errno));
- else
- EAL_LOG(ERR, "Unsupported VFIO API version!");
- close(vfio_container_fd);
- return -1;
- }
-
- ret = vfio_has_supported_extensions(vfio_container_fd);
- if (ret) {
- EAL_LOG(ERR,
- "No supported IOMMU extensions found!");
- return -1;
- }
-
- return vfio_container_fd;
- }
- /*
- * if we're in a secondary process, request container fd from the
- * primary process via mp channel
- */
- p->req = SOCKET_REQ_CONTAINER;
- strcpy(mp_req.name, EAL_VFIO_MP);
- mp_req.len_param = sizeof(*p);
- mp_req.num_fds = 0;
-
- vfio_container_fd = -1;
- if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
- mp_reply.nb_received == 1) {
- mp_rep = &mp_reply.msgs[0];
- p = (struct vfio_mp_param *)mp_rep->param;
- if (p->result == SOCKET_OK && mp_rep->num_fds == 1) {
- vfio_container_fd = mp_rep->fds[0];
- free(mp_reply.msgs);
- return vfio_container_fd;
- }
- }
-
- free(mp_reply.msgs);
- EAL_LOG(ERR, "Cannot request VFIO container fd");
- return -1;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_container_fd)
int
dev_vfio_get_container_fd(void)
@@ -1377,516 +1118,54 @@ dev_vfio_get_container_fd(void)
* The default container is set up during dev_vfio_enable().
* This function does not create a new container.
*/
- if (!vfio_enabled)
- return -1;
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_NONE)
+ return vfio_global_cfg.default_cfg->container_fd;
- return default_vfio_cfg->vfio_container_fd;
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
-dev_vfio_get_group_num(const char *sysfs_base,
- const char *dev_addr, int *iommu_group_num)
+dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
{
- char linkname[PATH_MAX];
- char filename[PATH_MAX];
- char *tok[16], *group_tok, *end;
int ret;
- if (!vfio_enabled) {
+ if (sysfs_base == NULL || dev_addr == NULL || iommu_group_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP && vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+ ret = vfio_group_get_num(sysfs_base, dev_addr, iommu_group_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
rte_errno = ENODEV;
return -1;
}
-
- memset(linkname, 0, sizeof(linkname));
- memset(filename, 0, sizeof(filename));
-
- /* try to find out IOMMU group for this device */
- snprintf(linkname, sizeof(linkname),
- "%s/%s/iommu_group", sysfs_base, dev_addr);
-
- ret = readlink(linkname, filename, sizeof(filename));
-
- /* if the link doesn't exist, no VFIO for us */
- if (ret < 0)
- return 0;
-
- ret = rte_strsplit(filename, sizeof(filename),
- tok, RTE_DIM(tok), '/');
-
- if (ret <= 0) {
- EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
- return -1;
- }
-
- /* IOMMU group is always the last token */
- errno = 0;
- group_tok = tok[ret - 1];
- end = group_tok;
- *iommu_group_num = strtol(group_tok, &end, 10);
- if ((end != group_tok && *end != '\0') || errno != 0) {
- EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
- return -1;
- }
-
- return 1;
-}
-
-static int
-type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
- void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova,
- ms->len, 1);
-}
-
-static int
-vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_type1_dma_map dma_map;
- struct vfio_iommu_type1_dma_unmap dma_unmap;
- int ret;
-
- if (do_map != 0) {
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- /**
- * In case the mapping was already done EEXIST will be
- * returned from kernel.
- */
- if (errno == EEXIST) {
- EAL_LOG(DEBUG,
- "Memory segment is already mapped, skipping");
- } else {
- EAL_LOG(ERR,
- "Cannot set up DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
- } else {
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot clear DMA remapping, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- } else if (dma_unmap.size != len) {
- EAL_LOG(ERR, "Unexpected size %"PRIu64
- " of DMA remapping cleared instead of %"PRIu64,
- (uint64_t)dma_unmap.size, len);
- rte_errno = EIO;
- return -1;
- }
- }
-
- return 0;
-}
-
-static int
-vfio_type1_dma_map(int vfio_container_fd)
-{
- return rte_memseg_walk(type1_map, &vfio_container_fd);
-}
-
-/* Track the size of the statically allocated DMA window for SPAPR */
-uint64_t spapr_dma_win_len;
-uint64_t spapr_dma_win_page_sz;
-
-static int
-vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map)
-{
- struct vfio_iommu_spapr_register_memory reg = {
- .argsz = sizeof(reg),
- .vaddr = (uintptr_t) vaddr,
- .size = len,
- .flags = 0
- };
- int ret;
-
- if (do_map != 0) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (iova + len > spapr_dma_win_len) {
- EAL_LOG(ERR, "DMA map attempt outside DMA window");
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot register vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = vaddr;
- dma_map.size = len;
- dma_map.iova = iova;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
- VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
- if (ret) {
- EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- } else {
- struct vfio_iommu_type1_dma_map dma_unmap;
-
- memset(&dma_unmap, 0, sizeof(dma_unmap));
- dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
- dma_unmap.size = len;
- dma_unmap.iova = iova;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA,
- &dma_unmap);
- if (ret) {
- EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
- if (ret) {
- EAL_LOG(ERR,
- "Cannot unregister vaddr for IOMMU, error "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_map_walk(const struct rte_memseg_list *msl,
- const struct rte_memseg *ms, void *arg)
-{
- int *vfio_container_fd = arg;
-
- /* skip external memory that isn't a heap */
- if (msl->external && !msl->heap)
- return 0;
-
- /* skip any segments with invalid IOVA addresses */
- if (ms->iova == RTE_BAD_IOVA)
- return 0;
-
- return vfio_spapr_dma_do_map(*vfio_container_fd,
- ms->addr_64, ms->iova, ms->len, 1);
-}
-
-struct spapr_size_walk_param {
- uint64_t max_va;
- uint64_t page_sz;
- bool is_user_managed;
-};
-
-/*
- * In order to set the DMA window size required for the SPAPR IOMMU
- * we need to walk the existing virtual memory allocations as well as
- * find the hugepage size used.
- */
-static int
-vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
-{
- struct spapr_size_walk_param *param = arg;
- uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
-
- if (msl->external && !msl->heap) {
- /* ignore user managed external memory */
- param->is_user_managed = true;
- return 0;
- }
-
- if (max > param->max_va) {
- param->page_sz = msl->page_sz;
- param->max_va = max;
- }
-
- return 0;
-}
-
-/*
- * Find the highest memory address used in physical or virtual address
- * space and use that as the top of the DMA window.
- */
-static int
-find_highest_mem_addr(struct spapr_size_walk_param *param)
-{
- /* find the maximum IOVA address for setting the DMA window size */
- if (rte_eal_iova_mode() == RTE_IOVA_PA) {
- static const char proc_iomem[] = "/proc/iomem";
- static const char str_sysram[] = "System RAM";
- uint64_t start, end, max = 0;
- char *line = NULL;
- char *dash, *space;
- size_t line_len;
-
- /*
- * Example "System RAM" in /proc/iomem:
- * 00000000-1fffffffff : System RAM
- * 200000000000-201fffffffff : System RAM
- */
- FILE *fd = fopen(proc_iomem, "r");
- if (fd == NULL) {
- EAL_LOG(ERR, "Cannot open %s", proc_iomem);
- return -1;
- }
- /* Scan /proc/iomem for the highest PA in the system */
- while (getline(&line, &line_len, fd) != -1) {
- if (strstr(line, str_sysram) == NULL)
- continue;
-
- space = strstr(line, " ");
- dash = strstr(line, "-");
-
- /* Validate the format of the memory string */
- if (space == NULL || dash == NULL || space < dash) {
- EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
- line, proc_iomem);
- continue;
- }
-
- start = strtoull(line, NULL, 16);
- end = strtoull(dash + 1, NULL, 16);
- EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
- " to 0x%" PRIx64, start, end);
- if (end > max)
- max = end;
- }
- free(line);
- fclose(fd);
-
- if (max == 0) {
- EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
- "entry in file %s", proc_iomem);
- return -1;
- }
-
- spapr_dma_win_len = rte_align64pow2(max + 1);
- return 0;
- } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
- EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
- PRIx64, param->max_va);
- spapr_dma_win_len = rte_align64pow2(param->max_va);
- return 0;
- }
-
- spapr_dma_win_len = 0;
- EAL_LOG(ERR, "Unsupported IOVA mode");
- return -1;
-}
-
-
-/*
- * The SPAPRv2 IOMMU supports 2 DMA windows with starting
- * address at 0 or 1<<59. By default, a DMA window is set
- * at address 0, 2GB long, with a 4KB page. For DPDK we
- * must remove the default window and setup a new DMA window
- * based on the hugepage size and memory requirements of
- * the application before we can map memory for DMA.
- */
-static int
-spapr_dma_win_size(void)
-{
- struct spapr_size_walk_param param;
-
- /* only create DMA window once */
- if (spapr_dma_win_len > 0)
- return 0;
-
- /* walk the memseg list to find the page size/max VA address */
- memset(¶m, 0, sizeof(param));
- if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
- EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
- return -1;
- }
-
- /* we can't be sure if DMA window covers external memory */
- if (param.is_user_managed)
- EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
-
- /* check physical/virtual memory size */
- if (find_highest_mem_addr(¶m) < 0)
- return -1;
- EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
- spapr_dma_win_len);
- spapr_dma_win_page_sz = param.page_sz;
- rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
- return 0;
-}
-
-static int
-vfio_spapr_create_dma_window(int vfio_container_fd)
-{
- struct vfio_iommu_spapr_tce_create create = {
- .argsz = sizeof(create), };
- struct vfio_iommu_spapr_tce_remove remove = {
- .argsz = sizeof(remove), };
- struct vfio_iommu_spapr_tce_info info = {
- .argsz = sizeof(info), };
- int ret;
-
- ret = spapr_dma_win_size();
- if (ret < 0)
- return ret;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
- if (ret) {
- EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
- errno, strerror(errno));
- return -1;
- }
-
- /*
- * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
- * can't be changed for v1 but it can be changed for v2. Since DPDK only
- * supports v2, remove the default DMA window so it can be resized.
- */
- remove.start_addr = info.dma32_window_start;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
- if (ret)
- return -1;
-
- /* create a new DMA window (start address is not selectable) */
- create.window_size = spapr_dma_win_len;
- create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
- create.levels = 1;
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
-#ifdef VFIO_IOMMU_SPAPR_INFO_DDW
- /*
- * The vfio_iommu_spapr_tce_info structure was modified in
- * Linux kernel 4.2.0 to add support for the
- * vfio_iommu_spapr_tce_ddw_info structure needed to try
- * multiple table levels. Skip the attempt if running with
- * an older kernel.
- */
- if (ret) {
- /* if at first we don't succeed, try more levels */
- uint32_t levels;
-
- for (levels = create.levels + 1;
- ret && levels <= info.ddw.levels; levels++) {
- create.levels = levels;
- ret = ioctl(vfio_container_fd,
- VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
- }
- }
-#endif /* VFIO_IOMMU_SPAPR_INFO_DDW */
- if (ret) {
- EAL_LOG(ERR, "Cannot create new DMA window, error "
- "%i (%s)", errno, strerror(errno));
- EAL_LOG(ERR,
- "Consider using a larger hugepage size if supported by the system");
- return -1;
- }
-
- /* verify the start address */
- if (create.start_addr != 0) {
- EAL_LOG(ERR, "Received unsupported start address 0x%"
- PRIx64, (uint64_t)create.start_addr);
- return -1;
- }
- return ret;
-}
-
-static int
-vfio_spapr_dma_mem_map(int vfio_container_fd, uint64_t vaddr,
- uint64_t iova, uint64_t len, int do_map)
-{
- int ret = 0;
-
- if (do_map) {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 1)) {
- EAL_LOG(ERR, "Failed to map DMA");
- ret = -1;
- }
- } else {
- if (vfio_spapr_dma_do_map(vfio_container_fd,
- vaddr, iova, len, 0)) {
- EAL_LOG(ERR, "Failed to unmap DMA");
- ret = -1;
- }
- }
-
- return ret;
-}
-
-static int
-vfio_spapr_dma_map(int vfio_container_fd)
-{
- if (vfio_spapr_create_dma_window(vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Could not create new DMA window!");
- return -1;
- }
-
- /* map all existing DPDK segments for DMA */
- if (rte_memseg_walk(vfio_spapr_map_walk, &vfio_container_fd) < 0)
- return -1;
-
- return 0;
-}
-
-static int
-vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
-{
- /* No-IOMMU mode does not need DMA mapping */
- return 0;
-}
-
-static int
-vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
- uint64_t __rte_unused vaddr,
- uint64_t __rte_unused iova, uint64_t __rte_unused len,
- int __rte_unused do_map)
-{
- /* No-IOMMU mode does not need DMA mapping */
return 0;
}
static int
-vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
{
- const struct vfio_iommu_type *t = vfio_cfg->vfio_iommu_type;
+ const struct vfio_iommu_ops *t = vfio_global_cfg.ops;
if (!t) {
EAL_LOG(ERR, "VFIO support not initialized");
- rte_errno = ENODEV;
return -1;
}
@@ -1894,33 +1173,30 @@ vfio_dma_mem_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
EAL_LOG(ERR,
"VFIO custom DMA region mapping not supported by IOMMU %s",
t->name);
- rte_errno = ENOTSUP;
return -1;
}
- return t->dma_user_map_func(vfio_cfg->vfio_container_fd, vaddr, iova,
- len, do_map);
+ return t->dma_user_map_func(cfg, vaddr, iova, len, do_map);
}
static int
-container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map *new_map;
- struct user_mem_maps *user_mem_maps;
+ struct vfio_user_mem_map *new_map;
+ struct vfio_user_mem_maps *vfio_user_mem_maps;
bool has_partial_unmap;
int ret = 0;
- user_mem_maps = &vfio_cfg->mem_maps;
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
- if (user_mem_maps->n_maps == RTE_DIM(user_mem_maps->maps)) {
+ vfio_user_mem_maps = &cfg->mem_maps;
+ rte_spinlock_recursive_lock(&vfio_user_mem_maps->lock);
+ if (vfio_user_mem_maps->n_maps == RTE_DIM(vfio_user_mem_maps->maps)) {
EAL_LOG(ERR, "No more space for user mem maps");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* map the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 1)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 1)) {
/* technically, this will fail if there are currently no devices
* plugged in, even if a device were added later, this mapping
* might have succeeded. however, since we cannot verify if this
@@ -1933,35 +1209,35 @@ container_dma_map(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
goto out;
}
/* do we have partial unmap support? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_global_cfg.ops->partial_unmap;
/* create new user mem map entry */
- new_map = &user_mem_maps->maps[user_mem_maps->n_maps++];
+ new_map = &vfio_user_mem_maps->maps[vfio_user_mem_maps->n_maps++];
new_map->addr = vaddr;
new_map->iova = iova;
new_map->len = len;
/* for IOMMU types supporting partial unmap, we don't need chunking */
new_map->chunk = has_partial_unmap ? 0 : len;
- compact_user_maps(user_mem_maps);
+ compact_user_maps(vfio_user_mem_maps);
out:
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
+ rte_spinlock_recursive_unlock(&vfio_user_mem_maps->lock);
return ret;
}
static int
-container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
+container_dma_unmap(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len)
{
- struct user_mem_map orig_maps[RTE_DIM(vfio_cfg->mem_maps.maps)];
- struct user_mem_map new_maps[2]; /* can be at most 2 */
- struct user_mem_maps *user_mem_maps;
+ struct vfio_user_mem_map orig_maps[RTE_DIM(cfg->mem_maps.maps)];
+ struct vfio_user_mem_map new_maps[2]; /* can be at most 2 */
+ struct vfio_user_mem_maps *vfio_user_mem_maps;
int n_orig, n_new, ret = 0;
bool has_partial_unmap;
unsigned int newlen;
- user_mem_maps = &vfio_cfg->mem_maps;
- rte_spinlock_recursive_lock(&user_mem_maps->lock);
+ vfio_user_mem_maps = &cfg->mem_maps;
+ rte_spinlock_recursive_lock(&vfio_user_mem_maps->lock);
/*
* Previously, we had adjacent mappings entirely contained within one
@@ -1981,18 +1257,17 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
* the start and the end of our requested unmap. We need to collect all
* maps that include our unmapped region.
*/
- n_orig = find_user_mem_maps(user_mem_maps, vaddr, iova, len,
+ n_orig = find_user_mem_maps(vfio_user_mem_maps, vaddr, iova, len,
orig_maps, RTE_DIM(orig_maps));
/* did we find anything? */
if (n_orig < 0) {
EAL_LOG(ERR, "Couldn't find previously mapped region");
- rte_errno = EINVAL;
ret = -1;
goto out;
}
/* do we have partial unmap capability? */
- has_partial_unmap = vfio_cfg->vfio_iommu_type->partial_unmap;
+ has_partial_unmap = vfio_global_cfg.ops->partial_unmap;
/*
* if we don't support partial unmap, we must check if start and end of
@@ -2008,7 +1283,6 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
if (!start_aligned || !end_aligned) {
EAL_LOG(DEBUG, "DMA partial unmap unsupported");
- rte_errno = ENOTSUP;
ret = -1;
goto out;
}
@@ -2023,39 +1297,31 @@ container_dma_unmap(struct vfio_config *vfio_cfg, uint64_t vaddr, uint64_t iova,
n_new = process_maps(orig_maps, n_orig, new_maps, vaddr, len);
/* can we store the new maps in our list? */
- newlen = (user_mem_maps->n_maps - n_orig) + n_new;
- if (newlen >= RTE_DIM(user_mem_maps->maps)) {
+ newlen = (vfio_user_mem_maps->n_maps - n_orig) + n_new;
+ if (newlen >= RTE_DIM(vfio_user_mem_maps->maps)) {
EAL_LOG(ERR, "Not enough space to store partial mapping");
- rte_errno = ENOMEM;
ret = -1;
goto out;
}
/* unmap the entry */
- if (vfio_dma_mem_map(vfio_cfg, vaddr, iova, len, 0)) {
+ if (vfio_dma_mem_map(cfg, vaddr, iova, len, 0)) {
/* there may not be any devices plugged in, so unmapping will
- * fail with ENODEV/ENOTSUP rte_errno values, but that doesn't
- * stop us from removing the mapping, as the assumption is we
- * won't be needing this memory any more and thus will want to
- * prevent it from being remapped again on hotplug. so, only
- * fail if we indeed failed to unmap (e.g. if the mapping was
- * within our mapped range but had invalid alignment).
+ * fail, but that doesn't stop us from removing the mapping,
+ * as the assumption is we won't be needing this memory any
+ * more and thus will want to prevent it from being remapped
+ * again on hotplug. Ignore the error and proceed with
+ * removing the mapping from our records.
*/
- if (rte_errno != ENODEV && rte_errno != ENOTSUP) {
- EAL_LOG(ERR, "Couldn't unmap region for DMA");
- ret = -1;
- goto out;
- } else {
- EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
- }
+ EAL_LOG(DEBUG, "DMA unmapping failed, but removing mappings anyway");
}
/* we have unmapped the region, so now update the maps */
- delete_maps(user_mem_maps, orig_maps, n_orig);
- copy_maps(user_mem_maps, new_maps, n_new);
- compact_user_maps(user_mem_maps);
+ delete_maps(vfio_user_mem_maps, orig_maps, n_orig);
+ copy_maps(vfio_user_mem_maps, new_maps, n_new);
+ compact_user_maps(vfio_user_mem_maps);
out:
- rte_spinlock_recursive_unlock(&user_mem_maps->lock);
+ rte_spinlock_recursive_unlock(&vfio_user_mem_maps->lock);
return ret;
}
@@ -2063,319 +1329,207 @@ RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
int
dev_vfio_noiommu_is_enabled(void)
{
- int fd;
- ssize_t cnt;
- char c;
-
- fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
- if (fd < 0) {
- if (errno != ENOENT) {
- EAL_LOG(ERR, "Cannot open VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
- /*
- * else the file does not exists
- * i.e. noiommu is not enabled
- */
- return 0;
- }
-
- cnt = read(fd, &c, 1);
- close(fd);
- if (cnt != 1) {
- EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
- "%i (%s)", errno, strerror(errno));
- return -1;
- }
-
- return c == 'Y';
+ return vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
{
- unsigned int i;
+ struct vfio_container *cfg;
+ int container_fd;
- if (!vfio_enabled) {
- rte_errno = ENODEV;
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
return -1;
}
-
- /* Find an empty slot to store new vfio config */
- for (i = 1; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].vfio_container_fd == -1)
- break;
- }
-
- if (i == RTE_DIM(vfio_cfgs)) {
- EAL_LOG(ERR, "Exceed max VFIO container limit");
+ cfg = vfio_container_create();
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "Reached VFIO container limit");
+ rte_errno = ENOSPC;
return -1;
}
- /* Create a new container fd */
- vfio_cfgs[i].vfio_container_fd = vfio_open_container_fd(false);
- if (vfio_cfgs[i].vfio_container_fd < 0) {
- EAL_LOG(NOTICE, "Fail to create a new VFIO container");
- return -1;
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ {
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Fail to create a new VFIO container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+ break;
}
- vfio_cfgs[i].enabled = true;
-
- return vfio_cfgs[i].vfio_container_fd;
+ default:
+ EAL_LOG(NOTICE, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
+ goto err;
+ }
+ return container_fd;
+err:
+ vfio_container_erase(cfg);
+ return -1;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_destroy)
int
dev_vfio_container_destroy(int container_fd)
{
- struct vfio_config *vfio_cfg;
- unsigned int i;
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
- if (!vfio_enabled) {
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
+ EAL_LOG(ERR, "VFIO container fd not managed by VFIO");
rte_errno = ENODEV;
return -1;
}
- if (container_fd == DEV_VFIO_DEFAULT_CONTAINER_FD) {
+ /* forbid destroying default container */
+ if (vfio_container_is_default(cfg)) {
EAL_LOG(ERR, "Cannot destroy default VFIO container");
- return -1;
- }
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
- return -1;
- }
-
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++)
- if (vfio_cfg->vfio_groups[i].group_num != -1)
- vfio_container_group_unbind(container_fd,
- vfio_cfg->vfio_groups[i].group_num);
-
- close(container_fd);
- vfio_cfg->enabled = false;
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->vfio_active_groups = 0;
- vfio_cfg->vfio_iommu_type = NULL;
-
- return 0;
-}
-
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_assign_device)
-int
-dev_vfio_container_assign_device(int vfio_container_fd, const char *sysfs_base,
- const char *dev_addr)
-{
- int iommu_group_num;
- int ret;
-
- ret = dev_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR, "Cannot get IOMMU group number for device %s",
- dev_addr);
- return -1;
- } else if (ret == 0) {
- EAL_LOG(ERR,
- "Device %s is not assigned to any IOMMU group",
- dev_addr);
- return -1;
- }
-
- ret = vfio_container_group_bind(vfio_container_fd,
- iommu_group_num);
- if (ret < 0) {
- EAL_LOG(ERR,
- "Cannot bind IOMMU group %d for device %s",
- iommu_group_num, dev_addr);
- return -1;
- }
-
- return 0;
-}
-
-static int
-vfio_container_group_bind(int container_fd, int iommu_group_num)
-{
- struct vfio_config *vfio_cfg;
-
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
- }
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
- return -1;
- }
-
- return vfio_get_group_fd(vfio_cfg, iommu_group_num);
-}
-
-static int
-vfio_container_group_unbind(int container_fd, int iommu_group_num)
-{
- struct vfio_group *cur_grp = NULL;
- struct vfio_config *vfio_cfg;
- unsigned int i;
-
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
- }
-
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
- EAL_LOG(ERR, "Invalid VFIO container fd");
- return -1;
- }
-
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- if (vfio_cfg->vfio_groups[i].group_num == iommu_group_num) {
- cur_grp = &vfio_cfg->vfio_groups[i];
- break;
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_GROUP:
+ case DEV_VFIO_MODE_NOIOMMU:
+ /* erase all devices */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device in IOMMU group %d still open, closing", dev->group);
+ /*
+ * technically we could've done back-reference lookup and closed our groups
+ * following a device close, but since we're closing and erasing all groups
+ * anyway, we can afford to not bother.
+ */
+ vfio_device_erase(cfg, dev);
}
- }
- /* This should not happen */
- if (cur_grp == NULL) {
- EAL_LOG(ERR, "Specified VFIO group number not found");
+ /* erase all groups */
+ struct vfio_group *grp;
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) {
+ EAL_LOG(DEBUG, "IOMMU group %d still open, closing", grp->group_num);
+ vfio_group_erase(cfg, grp);
+ }
+ break;
+ default:
+ EAL_LOG(ERR, "Unsupported VFIO mode");
+ rte_errno = ENOTSUP;
return -1;
}
- if (cur_grp->fd >= 0 && close(cur_grp->fd) < 0) {
- EAL_LOG(ERR,
- "Error when closing vfio_group_fd for iommu_group_num "
- "%d", iommu_group_num);
- return -1;
- }
- cur_grp->group_num = -1;
- cur_grp->fd = -1;
- cur_grp->devices = 0;
- vfio_cfg->vfio_active_groups--;
+ /* erase entire config */
+ vfio_container_erase(cfg);
return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_map)
int
-dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+dev_vfio_container_dma_map(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
-
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
- }
+ struct vfio_container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_map(vfio_cfg, vaddr, iova, len);
+ if (container_dma_map(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_dma_unmap)
int
-dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova,
- uint64_t len)
+dev_vfio_container_dma_unmap(int container_fd, uint64_t vaddr, uint64_t iova, uint64_t len)
{
- struct vfio_config *vfio_cfg;
-
- if (!vfio_enabled) {
- rte_errno = ENODEV;
- return -1;
- }
+ struct vfio_container *cfg;
if (len == 0) {
rte_errno = EINVAL;
return -1;
}
- vfio_cfg = get_vfio_cfg_by_container_fd(container_fd);
- if (vfio_cfg == NULL) {
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ cfg = vfio_container_get_by_fd(container_fd);
+ if (cfg == NULL) {
EAL_LOG(ERR, "Invalid VFIO container fd");
+ rte_errno = EINVAL;
return -1;
}
- return container_dma_unmap(vfio_cfg, vaddr, iova, len);
+ if (container_dma_unmap(cfg, vaddr, iova, len) < 0) {
+ rte_errno = EIO;
+ return -1;
+ }
+
+ return 0;
}
-static void
-vfio_cleanup_config(struct vfio_config *vfio_cfg)
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_mode)
+enum dev_vfio_mode
+dev_vfio_get_mode(void)
{
- unsigned int i;
-
- for (i = 0; i < RTE_DIM(vfio_cfg->vfio_groups); i++) {
- struct vfio_group *group = &vfio_cfg->vfio_groups[i];
-
- if (group->group_num == -1)
- continue;
- if (group->devices != 0) {
- EAL_LOG(ERR, "Cannot cleanup VFIO group %d with %d devices",
- group->group_num, group->devices);
- return;
- }
- if (group->fd >= 0 && close(group->fd) < 0) {
- EAL_LOG(ERR, "Cannot close VFIO group %d: %s",
- group->group_num, strerror(errno));
- return;
- }
-
- group->group_num = -1;
- group->fd = -1;
- group->devices = 0;
- vfio_cfg->vfio_active_groups--;
- }
-
- /* if there are still active groups, we cannot cleanup the container */
- if (vfio_cfg->vfio_active_groups != 0) {
- EAL_LOG(ERR, "Cannot cleanup VFIO container with %d active groups",
- vfio_cfg->vfio_active_groups);
- return;
- }
-
- if (vfio_cfg->vfio_container_fd >= 0 &&
- close(vfio_cfg->vfio_container_fd) < 0) {
- EAL_LOG(ERR, "Cannot close VFIO container: %s", strerror(errno));
- return;
- }
-
- vfio_cfg->vfio_container_fd = -1;
- vfio_cfg->enabled = false;
- vfio_cfg->vfio_iommu_type = NULL;
-
- vfio_cfg->mem_maps.n_maps = 0;
- memset(vfio_cfg->mem_maps.maps, 0, sizeof(vfio_cfg->mem_maps.maps));
+ return vfio_global_cfg.mode;
}
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_cleanup)
void
dev_vfio_cleanup(void)
{
- unsigned int i;
-
- if (!vfio_enabled)
- return;
+ struct vfio_container *cfg;
vfio_mp_sync_cleanup();
- /* mem events can only be unregistered from the primary process */
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
- rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+ struct vfio_device *dev;
+ struct vfio_group *grp;
- /* cleanup all enabled configs, ignore failures */
- for (i = 0; i < RTE_DIM(vfio_cfgs); i++) {
- if (vfio_cfgs[i].enabled)
- vfio_cleanup_config(&vfio_cfgs[i]);
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev)
+ vfio_device_erase(cfg, dev);
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp)
+ vfio_group_erase(cfg, grp);
+
+ /* callback is only registered for the default container in the primary */
+ if (group_cfg->mem_event_clb_set) {
+ rte_mem_event_callback_unregister(VFIO_MEM_EVENT_CLB_NAME, NULL);
+ group_cfg->mem_event_clb_set = false;
+ }
+
+ vfio_container_erase(cfg);
}
}
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 30389fb274..3cff4d4f91 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -6,34 +6,95 @@
#define EAL_VFIO_H_
#include <rte_common.h>
+#include <rte_spinlock.h>
#include <stdint.h>
+#include <dev_vfio.h>
+
+/* hot plug/unplug of VFIO groups may cause all DMA maps to be dropped. we can
+ * recreate the mappings for DPDK segments, but we cannot do so for memory that
+ * was registered by the user themselves, so we need to store the user mappings
+ * somewhere, to recreate them later.
+ */
+#define EAL_VFIO_MAX_USER_MEM_MAPS 256
+
+/* user memory map entry */
+struct vfio_user_mem_map {
+ uint64_t addr; /**< start VA */
+ uint64_t iova; /**< start IOVA */
+ uint64_t len; /**< total length of the mapping */
+ uint64_t chunk; /**< this mapping can be split in chunks of this size */
+};
+
+/* user memory maps container (common for all API modes) */
+struct vfio_user_mem_maps {
+ rte_spinlock_recursive_t lock;
+ int n_maps;
+ struct vfio_user_mem_map maps[EAL_VFIO_MAX_USER_MEM_MAPS];
+};
+
/*
* we don't need to store device fd's anywhere since they can be obtained from
* the group fd via an ioctl() call.
*/
struct vfio_group {
+ bool active;
int group_num;
int fd;
- int devices;
+ int n_devices;
+};
+
+enum vfio_device_type {
+ VFIO_DEVICE_TYPE_NONE = 0,
+ VFIO_DEVICE_TYPE_GROUP,
+};
+
+/* device tracking (common for group and cdev modes) */
+struct vfio_device {
+ bool active;
+ enum vfio_device_type type;
+ int fd;
+ int group; /**< back-reference to group list */
+ char *sysfs_base; /**< sysfs path prefix */
+ char *dev_addr; /**< device address */
+};
+
+/* group mode specific configuration */
+struct vfio_group_config {
+ bool iommu_type_set;
+ bool mem_event_clb_set;
+ size_t n_groups;
+ struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
+};
+
+/* per-container configuration */
+struct vfio_container {
+ bool active;
+ bool dma_setup_done;
+ int container_fd;
+ struct vfio_user_mem_maps mem_maps;
+ struct vfio_group_config group_cfg;
+ int n_devices;
+ struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
/* DMA mapping function prototype.
- * Takes VFIO container fd as a parameter.
+ * Takes VFIO container config as a parameter.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_func_t)(int);
+typedef int (*vfio_dma_func_t)(struct vfio_container *cfg);
/* Custom memory region DMA mapping function prototype.
- * Takes VFIO container fd, virtual address, physical address, length and
+ * Takes VFIO container config, virtual address, physical address, length and
* operation type (0 to unmap 1 for map) as a parameters.
* Returns 0 on success, -1 on error.
*/
-typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
- uint64_t len, int do_map);
+typedef int (*vfio_dma_user_func_t)(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
-struct vfio_iommu_type {
+/* mode-independent ops */
+struct vfio_iommu_ops {
int type_id;
const char *name;
bool partial_unmap;
@@ -41,33 +102,83 @@ struct vfio_iommu_type {
vfio_dma_func_t dma_map_func;
};
-/* get the vfio container that devices are bound to by default */
-int vfio_open_container_fd(bool mp_request);
+/* global configuration */
+struct vfio_config {
+ struct vfio_container *default_cfg;
+ enum dev_vfio_mode mode;
+ const struct vfio_iommu_ops *ops;
+};
-/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
-const struct vfio_iommu_type *
-vfio_set_iommu_type(int vfio_container_fd);
+/* per-process, per-container data */
+extern struct vfio_container vfio_containers[RTE_MAX_VFIO_CONTAINERS];
-int
-vfio_get_iommu_type(void);
+/* current configuration */
+extern struct vfio_config vfio_global_cfg;
-int vfio_get_group_fd_by_num(int iommu_group_num);
+#define VFIO_CONTAINER_FOREACH(cfg) \
+ for ((cfg) = &vfio_containers[0]; \
+ (cfg) < &vfio_containers[RTE_DIM(vfio_containers)]; \
+ (cfg)++)
-/* check if we have any supported extensions */
-int
-vfio_has_supported_extensions(int vfio_container_fd);
+#define VFIO_CONTAINER_FOREACH_ACTIVE(cfg) \
+ VFIO_CONTAINER_FOREACH((cfg)) \
+ if (((cfg)->active))
+#define VFIO_GROUP_FOREACH(cfg, grp) \
+ for ((grp) = &((cfg)->group_cfg.groups[0]); \
+ (grp) < &((cfg)->group_cfg.groups[RTE_DIM((cfg)->group_cfg.groups)]); \
+ (grp)++)
+
+#define VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) \
+ VFIO_GROUP_FOREACH((cfg), (grp)) \
+ if ((grp)->active)
+
+#define VFIO_DEVICE_FOREACH(cfg, dev) \
+ for ((dev) = &((cfg)->devices[0]); \
+ (dev) < &((cfg)->devices[RTE_DIM((cfg)->devices)]); \
+ (dev)++)
+
+#define VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) \
+ VFIO_DEVICE_FOREACH((cfg), (dev)) \
+ if ((dev)->active)
+
+/* for containers, we need to initialize the fd, and mem maps lock */
+#define VFIO_CONTAINER_INITIALIZER \
+ ((struct vfio_container){ \
+ .mem_maps = {.lock = RTE_SPINLOCK_RECURSIVE_INITIALIZER,}, \
+ .container_fd = -1, \
+ })
+
+int vfio_get_iommu_type(void);
int vfio_mp_sync_setup(void);
void vfio_mp_sync_cleanup(void);
+bool vfio_container_is_default(struct vfio_container *cfg);
+/* group mode functions */
+int vfio_group_enable(struct vfio_container *cfg);
+int vfio_group_open_container_fd(void);
+int vfio_group_noiommu_is_enabled(void);
+int vfio_group_get_num(const char *sysfs_base, const char *dev_addr,
+ int *iommu_group_num);
+struct vfio_group *vfio_group_get_by_num(struct vfio_container *cfg, int iommu_group);
+struct vfio_group *vfio_group_create(struct vfio_container *cfg, int iommu_group);
+void vfio_group_erase(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_open_fd(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_prepare(struct vfio_container *cfg, struct vfio_group *grp);
+int vfio_group_setup_iommu(struct vfio_container *cfg);
+int vfio_group_setup_device_fd(const char *dev_addr,
+ struct vfio_group *grp, struct vfio_device *dev);
+
+#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
+#define VFIO_MODNAME "vfio"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
-#define SOCKET_REQ_CONTAINER 0x100
-#define SOCKET_REQ_GROUP 0x200
-#define SOCKET_REQ_IOMMU_TYPE 0x400
-#define SOCKET_OK 0x0
-#define SOCKET_NO_FD 0x1
-#define SOCKET_ERR 0xFF
+#define VFIO_SOCKET_REQ_CONTAINER 0x100
+#define VFIO_SOCKET_REQ_GROUP 0x200
+#define VFIO_SOCKET_REQ_IOMMU_TYPE 0x400
+#define VFIO_SOCKET_OK 0x0
+#define VFIO_SOCKET_NO_FD 0x1
+#define VFIO_SOCKET_ERR 0xFF
struct vfio_mp_param {
int req;
@@ -75,6 +186,7 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ enum dev_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_group.c b/lib/eal/linux/eal_vfio_group.c
new file mode 100644
index 0000000000..c06cd50731
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_group.c
@@ -0,0 +1,983 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+#include <dev_vfio.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_type1_dma_map(struct vfio_container *);
+static int vfio_type1_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_spapr_dma_map(struct vfio_container *);
+static int vfio_spapr_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+static int vfio_noiommu_dma_map(struct vfio_container *);
+static int vfio_noiommu_dma_mem_map(struct vfio_container *, uint64_t, uint64_t, uint64_t, int);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_ops iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ {
+ .type_id = VFIO_TYPE1_IOMMU,
+ .name = "Type 1",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_type1_dma_map,
+ .dma_user_map_func = &vfio_type1_dma_mem_map
+ },
+ /* ppc64 IOMMU, otherwise known as spapr */
+ {
+ .type_id = VFIO_SPAPR_TCE_v2_IOMMU,
+ .name = "sPAPR",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_spapr_dma_map,
+ .dma_user_map_func = &vfio_spapr_dma_mem_map
+ },
+ /* IOMMU-less mode */
+ {
+ .type_id = VFIO_NOIOMMU_IOMMU,
+ .name = "No-IOMMU",
+ .partial_unmap = true,
+ .dma_map_func = &vfio_noiommu_dma_map,
+ .dma_user_map_func = &vfio_noiommu_dma_mem_map
+ },
+};
+
+static const struct vfio_iommu_ops *
+vfio_group_set_iommu_type(int vfio_container_fd)
+{
+ unsigned int idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, t->type_id);
+ if (ret == 0)
+ return t;
+ /* not an error, there may be more supported IOMMU types */
+ EAL_LOG(DEBUG, "Set IOMMU type %d (%s) failed, error "
+ "%i (%s)", t->type_id, t->name, errno,
+ strerror(errno));
+ }
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
+
+static int
+type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_type1_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_type1_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_type1_dma_map dma_map;
+ struct vfio_iommu_type1_dma_unmap dma_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ |
+ VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (dma_unmap.size != len) {
+ EAL_LOG(ERR, "Unexpected size %"PRIu64
+ " of DMA remapping cleared instead of %"PRIu64,
+ (uint64_t)dma_unmap.size, len);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+vfio_type1_dma_map(struct vfio_container *cfg)
+{
+ return rte_memseg_walk(type1_map, cfg);
+}
+
+/* Track the size of the statically allocated DMA window for SPAPR */
+static uint64_t spapr_dma_win_len;
+static uint64_t spapr_dma_win_page_sz;
+
+static int
+vfio_spapr_dma_do_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct vfio_iommu_spapr_register_memory reg = {
+ .argsz = sizeof(reg),
+ .vaddr = (uintptr_t) vaddr,
+ .size = len,
+ .flags = 0
+ };
+ int ret;
+
+ if (do_map != 0) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (iova + len > spapr_dma_win_len) {
+ EAL_LOG(ERR, "DMA map attempt outside DMA window");
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_REGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot register vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = vaddr;
+ dma_map.size = len;
+ dma_map.iova = iova;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot map vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ } else {
+ struct vfio_iommu_type1_dma_map dma_unmap;
+
+ memset(&dma_unmap, 0, sizeof(dma_unmap));
+ dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+ dma_unmap.size = len;
+ dma_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_UNMAP_DMA,
+ &dma_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot unmap vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY, ®);
+ if (ret) {
+ EAL_LOG(ERR,
+ "Cannot unregister vaddr for IOMMU, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_map_walk(const struct rte_memseg_list *msl,
+ const struct rte_memseg *ms, void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_spapr_dma_do_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+struct spapr_size_walk_param {
+ uint64_t max_va;
+ uint64_t page_sz;
+ bool is_user_managed;
+};
+
+/*
+ * In order to set the DMA window size required for the SPAPR IOMMU
+ * we need to walk the existing virtual memory allocations as well as
+ * find the hugepage size used.
+ */
+static int
+vfio_spapr_size_walk(const struct rte_memseg_list *msl, void *arg)
+{
+ struct spapr_size_walk_param *param = arg;
+ uint64_t max = (uint64_t) msl->base_va + (uint64_t) msl->len;
+
+ if (msl->external && !msl->heap) {
+ /* ignore user managed external memory */
+ param->is_user_managed = true;
+ return 0;
+ }
+
+ if (max > param->max_va) {
+ param->page_sz = msl->page_sz;
+ param->max_va = max;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the highest memory address used in physical or virtual address
+ * space and use that as the top of the DMA window.
+ */
+static int
+find_highest_mem_addr(struct spapr_size_walk_param *param)
+{
+ /* find the maximum IOVA address for setting the DMA window size */
+ if (rte_eal_iova_mode() == RTE_IOVA_PA) {
+ static const char proc_iomem[] = "/proc/iomem";
+ static const char str_sysram[] = "System RAM";
+ uint64_t start, end, max = 0;
+ char *line = NULL;
+ char *dash, *space;
+ size_t line_len;
+
+ /*
+ * Example "System RAM" in /proc/iomem:
+ * 00000000-1fffffffff : System RAM
+ * 200000000000-201fffffffff : System RAM
+ */
+ FILE *fd = fopen(proc_iomem, "r");
+ if (fd == NULL) {
+ EAL_LOG(ERR, "Cannot open %s", proc_iomem);
+ return -1;
+ }
+ /* Scan /proc/iomem for the highest PA in the system */
+ while (getline(&line, &line_len, fd) != -1) {
+ if (strstr(line, str_sysram) == NULL)
+ continue;
+
+ space = strstr(line, " ");
+ dash = strstr(line, "-");
+
+ /* Validate the format of the memory string */
+ if (space == NULL || dash == NULL || space < dash) {
+ EAL_LOG(ERR, "Can't parse line \"%s\" in file %s",
+ line, proc_iomem);
+ continue;
+ }
+
+ start = strtoull(line, NULL, 16);
+ end = strtoull(dash + 1, NULL, 16);
+ EAL_LOG(DEBUG, "Found system RAM from 0x%" PRIx64
+ " to 0x%" PRIx64, start, end);
+ if (end > max)
+ max = end;
+ }
+ free(line);
+ fclose(fd);
+
+ if (max == 0) {
+ EAL_LOG(ERR, "Failed to find valid \"System RAM\" "
+ "entry in file %s", proc_iomem);
+ return -1;
+ }
+
+ spapr_dma_win_len = rte_align64pow2(max + 1);
+ return 0;
+ } else if (rte_eal_iova_mode() == RTE_IOVA_VA) {
+ EAL_LOG(DEBUG, "Highest VA address in memseg list is 0x%"
+ PRIx64, param->max_va);
+ spapr_dma_win_len = rte_align64pow2(param->max_va);
+ return 0;
+ }
+
+ spapr_dma_win_len = 0;
+ EAL_LOG(ERR, "Unsupported IOVA mode");
+ return -1;
+}
+
+
+/*
+ * The SPAPRv2 IOMMU supports 2 DMA windows with starting
+ * address at 0 or 1<<59. By default, a DMA window is set
+ * at address 0, 2GB long, with a 4KB page. For DPDK we
+ * must remove the default window and setup a new DMA window
+ * based on the hugepage size and memory requirements of
+ * the application before we can map memory for DMA.
+ */
+static int
+spapr_dma_win_size(void)
+{
+ struct spapr_size_walk_param param;
+
+ /* only create DMA window once */
+ if (spapr_dma_win_len > 0)
+ return 0;
+
+ /* walk the memseg list to find the page size/max VA address */
+ memset(¶m, 0, sizeof(param));
+ if (rte_memseg_list_walk(vfio_spapr_size_walk, ¶m) < 0) {
+ EAL_LOG(ERR, "Failed to walk memseg list for DMA window size");
+ return -1;
+ }
+
+ /* we can't be sure if DMA window covers external memory */
+ if (param.is_user_managed)
+ EAL_LOG(WARNING, "Detected user managed external memory which may not be managed by the IOMMU");
+
+ /* check physical/virtual memory size */
+ if (find_highest_mem_addr(¶m) < 0)
+ return -1;
+ EAL_LOG(DEBUG, "Setting DMA window size to 0x%" PRIx64,
+ spapr_dma_win_len);
+ spapr_dma_win_page_sz = param.page_sz;
+ rte_mem_set_dma_mask(rte_ctz64(spapr_dma_win_len));
+ return 0;
+}
+
+static int
+vfio_spapr_create_dma_window(struct vfio_container *cfg)
+{
+ struct vfio_iommu_spapr_tce_create create = {
+ .argsz = sizeof(create), };
+ struct vfio_iommu_spapr_tce_remove remove = {
+ .argsz = sizeof(remove), };
+ struct vfio_iommu_spapr_tce_info info = {
+ .argsz = sizeof(info), };
+ int ret;
+
+ ret = spapr_dma_win_size();
+ if (ret < 0)
+ return ret;
+
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get IOMMU info, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ /*
+ * sPAPR v1/v2 IOMMU always has a default 1G DMA window set. The window
+ * can't be changed for v1 but it can be changed for v2. Since DPDK only
+ * supports v2, remove the default DMA window so it can be resized.
+ */
+ remove.start_addr = info.dma32_window_start;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_REMOVE, &remove);
+ if (ret)
+ return -1;
+
+ /* create a new DMA window (start address is not selectable) */
+ create.window_size = spapr_dma_win_len;
+ create.page_shift = rte_ctz64(spapr_dma_win_page_sz);
+ create.levels = 1;
+ ret = ioctl(cfg->container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ /*
+ * The vfio_iommu_spapr_tce_info structure was modified in
+ * Linux kernel 4.2.0 to add support for the
+ * vfio_iommu_spapr_tce_ddw_info structure needed to try
+ * multiple table levels. Skip the attempt if running with
+ * an older kernel.
+ */
+ if (ret) {
+ /* if at first we don't succeed, try more levels */
+ uint32_t levels;
+
+ for (levels = create.levels + 1;
+ ret && levels <= info.ddw.levels; levels++) {
+ create.levels = levels;
+ ret = ioctl(cfg->container_fd,
+ VFIO_IOMMU_SPAPR_TCE_CREATE, &create);
+ }
+ }
+ if (ret) {
+ EAL_LOG(ERR, "Cannot create new DMA window, error "
+ "%i (%s)", errno, strerror(errno));
+ EAL_LOG(ERR,
+ "Consider using a larger hugepage size if supported by the system");
+ return -1;
+ }
+
+ /* verify the start address */
+ if (create.start_addr != 0) {
+ EAL_LOG(ERR, "Received unsupported start address 0x%"
+ PRIx64, (uint64_t)create.start_addr);
+ return -1;
+ }
+ return ret;
+}
+
+static int
+vfio_spapr_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map)
+{
+ int ret = 0;
+
+ if (do_map) {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 1)) {
+ EAL_LOG(ERR, "Failed to map DMA");
+ ret = -1;
+ }
+ } else {
+ if (vfio_spapr_dma_do_map(cfg, vaddr, iova, len, 0)) {
+ EAL_LOG(ERR, "Failed to unmap DMA");
+ ret = -1;
+ }
+ }
+
+ return ret;
+}
+
+static int
+vfio_spapr_dma_map(struct vfio_container *cfg)
+{
+ if (vfio_spapr_create_dma_window(cfg) < 0) {
+ EAL_LOG(ERR, "Could not create new DMA window!");
+ return -1;
+ }
+
+ /* map all existing DPDK segments for DMA */
+ if (rte_memseg_walk(vfio_spapr_map_walk, cfg) < 0)
+ return -1;
+
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_map(struct vfio_container *cfg __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+static int
+vfio_noiommu_dma_mem_map(struct vfio_container *cfg __rte_unused,
+ uint64_t vaddr __rte_unused,
+ uint64_t iova __rte_unused, uint64_t len __rte_unused,
+ int do_map __rte_unused)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+struct vfio_group *
+vfio_group_create(struct vfio_container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ if (cfg->group_cfg.n_groups >= RTE_DIM(cfg->group_cfg.groups)) {
+ EAL_LOG(ERR, "Cannot add more VFIO groups to container");
+ return NULL;
+ }
+ VFIO_GROUP_FOREACH(cfg, grp) {
+ if (grp->active)
+ continue;
+ cfg->group_cfg.n_groups++;
+ grp->active = true;
+ grp->group_num = iommu_group;
+ grp->fd = -1;
+ return grp;
+ }
+ /* should not happen */
+ return NULL;
+}
+
+void
+vfio_group_erase(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_config *group_cfg = &cfg->group_cfg;
+
+ if (grp->fd >= 0 && close(grp->fd) < 0)
+ EAL_LOG(ERR, "Error when closing group fd %d", grp->fd);
+
+ *grp = (struct vfio_group){0};
+ group_cfg->n_groups--;
+
+ /* if this was the last group in config, erase IOMMU setup and unregister callback */
+ if (group_cfg->n_groups == 0) {
+ cfg->dma_setup_done = false;
+ group_cfg->iommu_type_set = false;
+ }
+}
+
+struct vfio_group *
+vfio_group_get_by_num(struct vfio_container *cfg, int iommu_group)
+{
+ struct vfio_group *grp;
+
+ VFIO_GROUP_FOREACH_ACTIVE(cfg, grp) {
+ if (grp->group_num == iommu_group)
+ return grp;
+ }
+ return NULL;
+}
+
+static int
+vfio_open_group_sysfs(int iommu_group_num)
+{
+ char filename[PATH_MAX];
+ int fd;
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_GROUP)
+ snprintf(filename, sizeof(filename), DEV_VFIO_GROUP_FMT, iommu_group_num);
+ else if (vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU)
+ snprintf(filename, sizeof(filename), DEV_VFIO_NOIOMMU_GROUP_FMT, iommu_group_num);
+
+ /* reset errno before open to differentiate errors */
+ errno = 0;
+ fd = open(filename, O_RDWR);
+
+ /* we have to differentiate between failed open and non-existence */
+ if (errno == ENOENT)
+ return -ENOENT;
+ return fd;
+}
+
+static int
+vfio_group_request_fd(int iommu_group_num)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int vfio_group_fd = -1;
+
+ p->req = VFIO_SOCKET_REQ_GROUP;
+ p->group_num = iommu_group_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1) {
+ vfio_group_fd = mp_rep->fds[0];
+ } else if (p->result == VFIO_SOCKET_NO_FD) {
+ EAL_LOG(ERR, "Bad VFIO group fd");
+ vfio_group_fd = -ENOENT;
+ }
+ }
+
+ free(mp_reply.msgs);
+ return vfio_group_fd;
+}
+
+int
+vfio_group_open_fd(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ int vfio_group_fd;
+
+ /* we make multiprocess request only in secondary processes for default config */
+ if ((rte_eal_process_type() != RTE_PROC_PRIMARY) && (vfio_container_is_default(cfg)))
+ vfio_group_fd = vfio_group_request_fd(grp->group_num);
+ else
+ vfio_group_fd = vfio_open_group_sysfs(grp->group_num);
+
+ /* pass the non-existence up the chain */
+ if (vfio_group_fd == -ENOENT)
+ return vfio_group_fd;
+ else if (vfio_group_fd < 0) {
+ EAL_LOG(ERR, "Failed to open VFIO group %d", grp->group_num);
+ return vfio_group_fd;
+ }
+ grp->fd = vfio_group_fd;
+ return 0;
+}
+
+static const struct vfio_iommu_ops *
+vfio_group_sync_iommu_ops(void)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int iommu_type_id;
+ unsigned int i;
+
+ /* find default container's IOMMU type */
+ p->req = VFIO_SOCKET_REQ_IOMMU_TYPE;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ iommu_type_id = -1;
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK)
+ iommu_type_id = p->iommu_type_id;
+ }
+ free(mp_reply.msgs);
+ if (iommu_type_id < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type from primary process");
+ return NULL;
+ }
+
+ /* we now have an fd for default container, as well as its IOMMU type.
+ * now, set up default VFIO container config to match.
+ */
+ for (i = 0; i < RTE_DIM(iommu_types); i++) {
+ const struct vfio_iommu_ops *t = &iommu_types[i];
+ if (t->type_id != iommu_type_id)
+ continue;
+
+ return t;
+ }
+ EAL_LOG(ERR, "Could not find IOMMU type id (%i)", iommu_type_id);
+ return NULL;
+}
+
+int
+vfio_group_noiommu_is_enabled(void)
+{
+ int fd;
+ ssize_t cnt;
+ char c;
+
+ fd = open(DEV_VFIO_NOIOMMU_MODE, O_RDONLY);
+ if (fd < 0) {
+ if (errno != ENOENT) {
+ EAL_LOG(ERR, "Cannot open VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ /*
+ * else the file does not exists
+ * i.e. noiommu is not enabled
+ */
+ return 0;
+ }
+
+ cnt = read(fd, &c, 1);
+ close(fd);
+ if (cnt != 1) {
+ EAL_LOG(ERR, "Unable to read from VFIO noiommu file "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+
+ return c == 'Y';
+}
+
+static int
+vfio_has_supported_extensions(int vfio_container_fd)
+{
+ int ret;
+ unsigned int idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_ops *t = &iommu_types[idx];
+
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Could not get IOMMU type, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
+ }
+ EAL_LOG(DEBUG, "IOMMU type %d (%s) is %s",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (n_extensions == 0)
+ return -1;
+
+ return 0;
+}
+
+int
+vfio_group_open_container_fd(void)
+{
+ int ret, vfio_container_fd;
+
+ vfio_container_fd = open(DEV_VFIO_CONTAINER_PATH, O_RDWR);
+ if (vfio_container_fd < 0) {
+ EAL_LOG(DEBUG, "Cannot open VFIO container %s, error %i (%s)",
+ DEV_VFIO_CONTAINER_PATH, errno, strerror(errno));
+ return -1;
+ }
+
+ /* check VFIO API version */
+ ret = ioctl(vfio_container_fd, VFIO_GET_API_VERSION);
+ if (ret != VFIO_API_VERSION) {
+ if (ret < 0)
+ EAL_LOG(DEBUG,
+ "Could not get VFIO API version, error "
+ "%i (%s)", errno, strerror(errno));
+ else
+ EAL_LOG(DEBUG, "Unsupported VFIO API version!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ ret = vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ EAL_LOG(DEBUG,
+ "No supported IOMMU extensions found!");
+ close(vfio_container_fd);
+ return -1;
+ }
+
+ return vfio_container_fd;
+}
+
+int
+vfio_group_enable(struct vfio_container *cfg)
+{
+ int container_fd;
+ DIR *dir;
+
+ /* VFIO directory might not exist (e.g., unprivileged containers) */
+ dir = opendir(DEV_VFIO_DIR);
+ if (dir == NULL) {
+ EAL_LOG(DEBUG,
+ "VFIO directory does not exist, skipping VFIO group support...");
+ return 1;
+ }
+ closedir(dir);
+
+ /* open a default container */
+ container_fd = vfio_group_open_container_fd();
+ if (container_fd < 0)
+ return -1;
+
+ cfg->container_fd = container_fd;
+ return 0;
+}
+
+int
+vfio_group_prepare(struct vfio_container *cfg, struct vfio_group *grp)
+{
+ struct vfio_group_status group_status = {
+ .argsz = sizeof(group_status)};
+ int ret;
+
+ /*
+ * We need to assign group to a container and check if it is viable, but there are cases
+ * where we don't need to do that.
+ *
+ * For default container, we need to set up the group only in primary process, as secondary
+ * process would have requested group fd over IPC, which implies it would have already been
+ * set up by the primary.
+ *
+ * For custom containers, every process sets up its own groups.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ EAL_LOG(DEBUG, "Skipping setup for VFIO group %d", grp->group_num);
+ return 0;
+ }
+
+ /* check if the group is viable */
+ ret = ioctl(grp->fd, VFIO_GROUP_GET_STATUS, &group_status);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot get VFIO group status for group %d, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+
+ if ((group_status.flags & VFIO_GROUP_FLAGS_VIABLE) == 0) {
+ EAL_LOG(ERR, "VFIO group %d is not viable! "
+ "Not all devices in IOMMU group bound to VFIO or unbound",
+ grp->group_num);
+ return -1;
+ }
+
+ /* set container for group if necessary */
+ if ((group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET) == 0) {
+ /* add group to a container */
+ ret = ioctl(grp->fd, VFIO_GROUP_SET_CONTAINER, &cfg->container_fd);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot add VFIO group %d to container, error %i (%s)",
+ grp->group_num, errno, strerror(errno));
+ return -1;
+ }
+ } else {
+ /* group is already added to a container - this should not happen */
+ EAL_LOG(ERR, "VFIO group %d is already assigned to a container", grp->group_num);
+ return -1;
+ }
+ return 0;
+}
+
+int
+vfio_group_setup_iommu(struct vfio_container *cfg)
+{
+ const struct vfio_iommu_ops *ops;
+
+ /*
+ * Setting IOMMU type is a per-container operation (via ioctl on container fd), but the ops
+ * structure is global and shared across all containers.
+ *
+ * For secondary processes with default container, we sync ops from primary. For all other
+ * cases (primary, or secondary with custom containers), we set IOMMU type on the container
+ * which also discovers the ops.
+ */
+ if (vfio_container_is_default(cfg) && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ /* Secondary process: sync ops from primary for default container */
+ ops = vfio_group_sync_iommu_ops();
+ if (ops == NULL)
+ return -1;
+ } else {
+ /* Primary process OR custom container: set IOMMU type on container */
+ ops = vfio_group_set_iommu_type(cfg->container_fd);
+ if (ops == NULL)
+ return -1;
+ }
+
+ /* Set or verify global ops */
+ if (vfio_global_cfg.ops == NULL) {
+ vfio_global_cfg.ops = ops;
+ EAL_LOG(INFO, "IOMMU type set to %d (%s)", ops->type_id, ops->name);
+ } else if (vfio_global_cfg.ops != ops) {
+ /* This shouldn't happen on the same machine, but log it */
+ EAL_LOG(WARNING,
+ "Container has different IOMMU type (%d - %s) than previously set (%d - %s)",
+ ops->type_id, ops->name, vfio_global_cfg.ops->type_id, vfio_global_cfg.ops->name);
+ }
+
+ return 0;
+}
+
+int
+vfio_group_setup_device_fd(const char *dev_addr, struct vfio_group *grp, struct vfio_device *dev)
+{
+ rte_uuid_t vf_token;
+ int fd;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ if (!rte_uuid_is_null(vf_token)) {
+ char vf_token_str[RTE_UUID_STRLEN];
+ char devaddr[PATH_MAX];
+
+ rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str));
+ snprintf(devaddr, sizeof(devaddr),
+ "%s vf_token=%s", dev_addr, vf_token_str);
+
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, devaddr);
+ if (fd >= 0)
+ goto out;
+ }
+ /* get a file descriptor for the device */
+ fd = ioctl(grp->fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr);
+ if (fd < 0) {
+ /*
+ * if we cannot get a device fd, this implies a problem with the VFIO group or the
+ * container not having IOMMU configured.
+ */
+ EAL_LOG(WARNING, "Getting a vfio_dev_fd for %s failed", dev_addr);
+ return -1;
+ }
+out:
+ dev->fd = fd;
+ /* store backreference to group */
+ dev->group = grp->group_num;
+ /* increment number of devices in group */
+ grp->n_devices++;
+ return 0;
+}
+
+int
+vfio_group_get_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *tok[16], *group_tok, *end;
+ int ret, group_num;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* try to find out IOMMU group for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/iommu_group", sysfs_base, dev_addr);
+
+ ret = readlink(linkname, filename, sizeof(filename));
+
+ /* if the link doesn't exist, no VFIO for us */
+ if (ret < 0)
+ return 0;
+
+ ret = rte_strsplit(filename, sizeof(filename),
+ tok, RTE_DIM(tok), '/');
+
+ if (ret <= 0) {
+ EAL_LOG(ERR, "%s cannot get IOMMU group", dev_addr);
+ return -1;
+ }
+
+ /* IOMMU group is always the last token */
+ errno = 0;
+ group_tok = tok[ret - 1];
+ end = group_tok;
+ group_num = strtol(group_tok, &end, 10);
+ if (end == group_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing IOMMU number!", dev_addr);
+ return -1;
+ }
+ *iommu_group_num = group_num;
+
+ return 1;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index 66341a701a..e1f82ad77b 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -31,46 +31,65 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
memset(&reply, 0, sizeof(reply));
switch (m->req) {
- case SOCKET_REQ_GROUP:
- r->req = SOCKET_REQ_GROUP;
+ case VFIO_SOCKET_REQ_GROUP:
+ {
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
+ struct vfio_group *grp;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP &&
+ vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_GROUP;
r->group_num = m->group_num;
- fd = vfio_get_group_fd_by_num(m->group_num);
- if (fd < 0 && fd != -ENOENT)
- r->result = SOCKET_ERR;
- else if (fd == -ENOENT)
- /* if VFIO group exists but isn't bound to VFIO driver */
- r->result = SOCKET_NO_FD;
- else {
- /* if group exists and is bound to VFIO driver */
- r->result = SOCKET_OK;
+ grp = vfio_group_get_by_num(cfg, m->group_num);
+ if (grp == NULL) {
+ /* group doesn't exist in primary */
+ r->result = VFIO_SOCKET_NO_FD;
+ } else {
+ /* group exists and is bound to VFIO driver */
+ fd = grp->fd;
+ r->result = VFIO_SOCKET_OK;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
- case SOCKET_REQ_CONTAINER:
- r->req = SOCKET_REQ_CONTAINER;
+ }
+ case VFIO_SOCKET_REQ_CONTAINER:
+ r->req = VFIO_SOCKET_REQ_CONTAINER;
fd = dev_vfio_get_container_fd();
if (fd < 0)
- r->result = SOCKET_ERR;
+ r->result = VFIO_SOCKET_ERR;
else {
- r->result = SOCKET_OK;
+ r->result = VFIO_SOCKET_OK;
+ r->mode = vfio_global_cfg.mode;
reply.num_fds = 1;
reply.fds[0] = fd;
}
break;
- case SOCKET_REQ_IOMMU_TYPE:
+ case VFIO_SOCKET_REQ_IOMMU_TYPE:
{
int iommu_type_id;
- r->req = SOCKET_REQ_IOMMU_TYPE;
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_GROUP &&
+ vfio_global_cfg.mode != DEV_VFIO_MODE_NOIOMMU) {
+ EAL_LOG(ERR, "VFIO not initialized in group mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_IOMMU_TYPE;
iommu_type_id = vfio_get_iommu_type();
if (iommu_type_id < 0)
- r->result = SOCKET_ERR;
+ r->result = VFIO_SOCKET_ERR;
else {
r->iommu_type_id = iommu_type_id;
- r->result = SOCKET_OK;
+ r->result = VFIO_SOCKET_OK;
}
break;
}
@@ -90,8 +109,11 @@ vfio_mp_sync_setup(void)
{
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
int ret = rte_mp_action_register(EAL_VFIO_MP, vfio_mp_primary);
- if (ret && rte_errno != ENOTSUP)
+ if (ret && rte_errno != ENOTSUP) {
+ EAL_LOG(ERR, "Multiprocess sync setup failed: %d (%s)",
+ rte_errno, rte_strerror(rte_errno));
return -1;
+ }
}
return 0;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 29ba313218..5ec8eddaa2 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 20/25] bus/pci: use the new VFIO mode API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (18 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 19/25] vfio: cleanup and refactor Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 21/25] bus/fslmc: " Anatoly Burakov
` (5 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Chenbo Xia, Nipun Gupta
Use the new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/bus/pci/linux/pci.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c
index af37a65c45..b9fe4797fb 100644
--- a/drivers/bus/pci/linux/pci.c
+++ b/drivers/bus/pci/linux/pci.c
@@ -596,7 +596,7 @@ pci_device_iova_mode(const struct rte_pci_driver *pdrv,
static int is_vfio_noiommu_enabled = -1;
if (is_vfio_noiommu_enabled == -1) {
- if (dev_vfio_noiommu_is_enabled() == 1)
+ if (dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU)
is_vfio_noiommu_enabled = 1;
else
is_vfio_noiommu_enabled = 0;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 21/25] bus/fslmc: use the new VFIO mode API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (19 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 20/25] bus/pci: use the new VFIO mode API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 22/25] net/hinic3: " Anatoly Burakov
` (4 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Hemant Agrawal, Sachin Saxena
FSLMC bus only supports operating in group mode, and relies on no-IOMMU
mode checks. Use the new VFIO API's to query no-IOMMU status, as well as
protect the bus from initializing in non-group mode.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Acked-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
drivers/bus/fslmc/fslmc_bus.c | 11 ++++++++++-
drivers/bus/fslmc/fslmc_vfio.c | 2 +-
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c
index d55c414ead..8bfbdac52e 100644
--- a/drivers/bus/fslmc/fslmc_bus.c
+++ b/drivers/bus/fslmc/fslmc_bus.c
@@ -332,6 +332,14 @@ rte_fslmc_scan(void)
goto scan_fail;
}
+ /* for container groups to work, VFIO must be in group mode */
+ if (dev_vfio_get_mode() != DEV_VFIO_MODE_GROUP &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU) {
+ DPAA2_BUS_DEBUG("DPAA2: VFIO not in group mode");
+ ret = -EINVAL;
+ goto scan_fail;
+ }
+
ret = fslmc_get_container_group(group_name, &groupid);
if (ret != 0)
goto scan_fail;
@@ -500,7 +508,8 @@ rte_dpaa2_get_iommu_class(void)
return RTE_IOVA_DC;
/* check if all devices on the bus support Virtual addressing or not */
- if (fslmc_all_device_support_iova() != 0 && dev_vfio_noiommu_is_enabled() == 0)
+ if (fslmc_all_device_support_iova() != 0 &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU)
return RTE_IOVA_VA;
return RTE_IOVA_PA;
diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index c2b6ac1f3f..c416ef9f68 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -204,7 +204,7 @@ fslmc_vfio_add_group(int vfio_group_fd,
group->fd = vfio_group_fd;
group->groupid = iommu_group_num;
rte_strscpy(group->group_name, group_name, sizeof(group->group_name));
- if (dev_vfio_noiommu_is_enabled() > 0)
+ if (dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU)
group->iommu_type = VFIO_NOIOMMU_IOMMU;
else
group->iommu_type = VFIO_TYPE1_IOMMU;
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 22/25] net/hinic3: use the new VFIO mode API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (20 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 21/25] bus/fslmc: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 23/25] net/ntnic: " Anatoly Burakov
` (3 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Feifei Wang
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/hinic3/base/hinic3_hwdev.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hinic3/base/hinic3_hwdev.c b/drivers/net/hinic3/base/hinic3_hwdev.c
index 6a6cf44752..3486160ba1 100644
--- a/drivers/net/hinic3/base/hinic3_hwdev.c
+++ b/drivers/net/hinic3/base/hinic3_hwdev.c
@@ -78,7 +78,8 @@ hinic3_is_vfio_iommu_enable(const struct rte_eth_dev *eth_dev)
{
struct rte_pci_device *pci_dev = RTE_CLASS_TO_BUS_DEVICE(eth_dev, *pci_dev);
- return pci_dev->kdrv == RTE_PCI_KDRV_VFIO && dev_vfio_noiommu_is_enabled() != 1;
+ return pci_dev->kdrv == RTE_PCI_KDRV_VFIO &&
+ dev_vfio_get_mode() != DEV_VFIO_MODE_NOIOMMU;
}
int
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 23/25] net/ntnic: use the new VFIO mode API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (21 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 22/25] net/hinic3: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 24/25] vfio: remove no-IOMMU check API Anatoly Burakov
` (2 subsequent siblings)
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Christian Koue Muf, Serhii Iliushyk
Use new VFIO mode API to query no-IOMMU status.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
drivers/net/ntnic/ntnic_ethdev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ntnic/ntnic_ethdev.c b/drivers/net/ntnic/ntnic_ethdev.c
index d9744a7d05..45629a567d 100644
--- a/drivers/net/ntnic/ntnic_ethdev.c
+++ b/drivers/net/ntnic/ntnic_ethdev.c
@@ -2690,7 +2690,7 @@ nthw_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
(pci_dev->device.devargs->data ? pci_dev->device.devargs->data : "NULL"));
}
- const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_noiommu_is_enabled();
+ const int n_dev_vfio_no_io_mmu_enabled = dev_vfio_get_mode() == DEV_VFIO_MODE_NOIOMMU;
NT_LOG(DBG, NTNIC, "vfio_no_iommu_enabled=%d", n_dev_vfio_no_io_mmu_enabled);
if (n_dev_vfio_no_io_mmu_enabled) {
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 24/25] vfio: remove no-IOMMU check API
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (22 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 23/25] net/ntnic: " Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 25/25] vfio: introduce cdev mode Anatoly Burakov
2026-08-28 13:32 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Burakov, Anatoly
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson
The `dev_vfio_noiommu_is_enabled()` check has now been replaced by the new
mode API, so remove it.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/eal/freebsd/eal.c | 7 -------
lib/eal/include/dev_vfio.h | 14 --------------
lib/eal/linux/eal_vfio.c | 7 -------
3 files changed, 28 deletions(-)
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index 753aaf8656..e857ce7309 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -860,13 +860,6 @@ dev_vfio_is_enabled(__rte_unused const char *modname)
return 0;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
-int
-dev_vfio_noiommu_is_enabled(void)
-{
- return 0;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_group_num)
int
dev_vfio_get_group_num(__rte_unused const char *sysfs_base,
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 6404dd08c3..02978bf80a 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -177,20 +177,6 @@ __rte_internal
enum dev_vfio_mode
dev_vfio_get_mode(void);
-/**
- * @internal
- * Check if VFIO NOIOMMU mode is enabled.
- *
- * This function is only relevant on Linux in group mode.
- *
- * @return
- * 1 if enabled.
- * 0 if not enabled or not supported.
- */
-__rte_internal
-int
-dev_vfio_noiommu_is_enabled(void);
-
/**
* @internal
* Parse IOMMU group number for a device.
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 1cd1a44eef..df0cc1284c 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -1325,13 +1325,6 @@ container_dma_unmap(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
return ret;
}
-RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_noiommu_is_enabled)
-int
-dev_vfio_noiommu_is_enabled(void)
-{
- return vfio_global_cfg.mode == DEV_VFIO_MODE_NOIOMMU;
-}
-
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_container_create)
int
dev_vfio_container_create(void)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* [PATCH v14 25/25] vfio: introduce cdev mode
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (23 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 24/25] vfio: remove no-IOMMU check API Anatoly Burakov
@ 2026-08-28 13:06 ` Anatoly Burakov
2026-08-28 13:32 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Burakov, Anatoly
25 siblings, 0 replies; 203+ messages in thread
From: Anatoly Burakov @ 2026-08-28 13:06 UTC (permalink / raw)
To: dev, Bruce Richardson
Add support for VFIO cdev (also known as IOMMUFD) API. The group API is now
considered legacy in the kernel, and all further development is expected to
happen in IOMMUFD infrastructure.
The implementation largely follows the kernel reference code, with most
differences from group mode concentrated in how the container and the
devices are set up. The VFIO container allocates an "IOAS" (IO address
space), maps the memory into that space, and attaches devices to IOAS -
there is no IOMMU type discovery at device attach time.
Unlike group mode, the cdev mode is device-centric, and all DMA mappings
are made for an IOAS and not into VFIO container itself. As a result, while
group mode only maps memory during first device attach, cdev mode can map
memory without having any devices attached. However, current VFIO init has
moved to before bus scan, because we need VFIO mode information for buses
to decide on IOVA mode. DMA memory mapping, however, must be performed
after the DPDK memory was already initialized, so we add another internal
API to trigger VFIO DMA maps for cdev mode right after memory init.
To assist any future use of VFIO cdev mode for custom behavior, also
introduce "get device number" API, which is kind-of-but-not-really similar
to the concept of IOMMU group.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/linux_gsg/linux_drivers.rst | 30 ++
doc/guides/rel_notes/release_26_11.rst | 5 +
lib/eal/freebsd/eal.c | 16 +
lib/eal/include/dev_vfio.h | 45 +++
lib/eal/linux/eal.c | 7 +
lib/eal/linux/eal_vfio.c | 260 +++++++++++++++-
lib/eal/linux/eal_vfio.h | 39 ++-
lib/eal/linux/eal_vfio_cdev.c | 396 +++++++++++++++++++++++++
lib/eal/linux/eal_vfio_mp_sync.c | 42 +++
lib/eal/linux/meson.build | 1 +
10 files changed, 834 insertions(+), 7 deletions(-)
create mode 100644 lib/eal/linux/eal_vfio_cdev.c
diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst
index 12e9faf4a2..7dd38a69f8 100644
--- a/doc/guides/linux_gsg/linux_drivers.rst
+++ b/doc/guides/linux_gsg/linux_drivers.rst
@@ -133,6 +133,31 @@ For proper operation of VFIO when running DPDK applications as a non-privileged
For more information, please refer to :ref:`linux_gsg_running_without_root_privileges`.
+.. _linux_gsg_vfio_modes:
+
+VFIO group and cdev modes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Linux kernel provides two different interfaces for VFIO:
+
+group mode
+ The original interface, based on IOMMU groups.
+ It is accessed through the container device ``/dev/vfio/vfio`` and the per-group devices under ``/dev/vfio/``.
+
+cdev mode
+ The newer interface, based on iommufd.
+ It is accessed through ``/dev/iommu`` and the per-device character devices under ``/dev/vfio/devices/``.
+
+DPDK selects the mode automatically; there is no EAL option to force a particular mode.
+Currently, group mode is preferred and is used whenever the container device ``/dev/vfio/vfio`` is usable.
+Using cdev mode requires ``/dev/iommu`` to be present and accessible.
+
+.. note::
+
+ Using VF tokens in cdev mode requires Linux kernel version 6.17 or later.
+ On older kernels the device is bound without the token, so the kernel should be configured to use group mode if VF token support is required.
+
+
.. _linux_gsg_vfio_noiommu:
VFIO no-IOMMU mode
@@ -265,6 +290,11 @@ The token will be used for all PF and VF ports within the application.
Linux versions earlier than version 5.7 do not support the creation of
virtual functions within the VFIO framework.
+.. note::
+
+ When VFIO uses cdev mode, VF tokens require a newer kernel.
+ See :ref:`linux_gsg_vfio_modes` for details.
+
Troubleshooting VFIO
~~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 006b1d3e5a..ed56021c83 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Added VFIO cdev (IOMMUFD) mode.**
+
+ Added support for the VFIO character device (cdev) API, also known as IOMMUFD.
+ DPDK now supports both the group mode and the new cdev mode.
+
Removed Items
-------------
diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
index e857ce7309..32a6a7f654 100644
--- a/lib/eal/freebsd/eal.c
+++ b/lib/eal/freebsd/eal.c
@@ -853,6 +853,12 @@ dev_vfio_cleanup(void)
{
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_init_mem)
+int dev_vfio_init_mem(void)
+{
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
dev_vfio_is_enabled(__rte_unused const char *modname)
@@ -941,3 +947,13 @@ dev_vfio_get_mode(void)
{
return DEV_VFIO_MODE_NONE;
}
+
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_num)
+int
+dev_vfio_get_device_num(__rte_unused const char *sysfs_base,
+ __rte_unused const char *dev_addr,
+ __rte_unused int *vfio_device_num)
+{
+ rte_errno = ENOTSUP;
+ return -1;
+}
diff --git a/lib/eal/include/dev_vfio.h b/lib/eal/include/dev_vfio.h
index 02978bf80a..6f982d492d 100644
--- a/lib/eal/include/dev_vfio.h
+++ b/lib/eal/include/dev_vfio.h
@@ -28,6 +28,8 @@ extern "C" {
#define DEV_VFIO_DIR "/dev/vfio"
#define DEV_VFIO_CONTAINER_PATH "/dev/vfio/vfio"
+#define DEV_VFIO_IOMMUFD_PATH "/dev/iommu"
+#define DEV_VFIO_CDEV_DEVICES_PATH "/dev/vfio/devices"
#define DEV_VFIO_GROUP_FMT "/dev/vfio/%u"
#define DEV_VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define DEV_VFIO_NOIOMMU_MODE "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
@@ -48,11 +50,13 @@ struct vfio_device_info;
* - DEV_VFIO_MODE_NONE: VFIO is not enabled.
* - DEV_VFIO_MODE_GROUP: Legacy group mode.
* - DEV_VFIO_MODE_NOIOMMU: Unsafe no-IOMMU mode.
+ * - DEV_VFIO_MODE_CDEV: Character device mode.
*/
enum dev_vfio_mode {
DEV_VFIO_MODE_NONE = 0, /**< VFIO not enabled */
DEV_VFIO_MODE_GROUP, /**< Group mode */
DEV_VFIO_MODE_NOIOMMU, /**< Group mode with no IOMMU protection */
+ DEV_VFIO_MODE_CDEV, /**< Device mode */
};
/**
@@ -148,6 +152,20 @@ int dev_vfio_enable(void);
__rte_internal
void dev_vfio_cleanup(void);
+/**
+ * @internal
+ * Initialize VFIO memory mapping. Must be called after `dev_vfio_enable()` and
+ * after EAL memory initialization has completed.
+ *
+ * This function is only relevant on Linux.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure.
+ */
+__rte_internal
+int dev_vfio_init_mem(void);
+
/**
* @internal
* Check if VFIO subsystem is initialized and a specified kernel module is loaded.
@@ -204,6 +222,33 @@ __rte_internal
int
dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_group_num);
+/**
+ * @internal
+ * Parse VFIO cdev device number for a device.
+ *
+ * This function is only relevant on Linux in cdev mode.
+ *
+ * @param sysfs_base
+ * Sysfs path prefix.
+ * @param dev_addr
+ * Device identifier.
+ * @param vfio_device_num
+ * Pointer to where VFIO cdev device number will be stored.
+ *
+ * @return
+ * 0 on success.
+ * <0 on failure, rte_errno is set.
+ *
+ * Possible rte_errno values include:
+ * - ENODEV - Device not managed by VFIO.
+ * - EINVAL - Invalid parameters.
+ * - ENXIO - VFIO support not initialized.
+ * - ENOTSUP - Unsupported VFIO mode.
+ */
+__rte_internal
+int
+dev_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *vfio_device_num);
+
/**
* @internal
* Get device information.
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 722cacdb45..7cfb253824 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -818,6 +818,13 @@ rte_eal_init(int argc, char **argv)
goto err_out;
}
+ /* VFIO memory setup that requires DPDK memory to be available. */
+ if (dev_vfio_init_mem() < 0) {
+ rte_eal_init_alert("Cannot init VFIO memory");
+ rte_errno = EAGAIN;
+ goto err_out;
+ }
+
/* register multi-process action callbacks for hotplug after memory init */
if (eal_mp_dev_hotplug_init() < 0) {
rte_eal_init_alert("failed to register mp callback for hotplug");
diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index df0cc1284c..185de13674 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -349,6 +349,24 @@ vfio_container_get_by_group_num(int group_num)
return NULL;
}
+static struct vfio_container *
+vfio_container_get_by_dev_num(int dev_num)
+{
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+
+ VFIO_CONTAINER_FOREACH_ACTIVE(cfg) {
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ /* only cdev mode has dev_num */
+ if (dev->type != VFIO_DEVICE_TYPE_CDEV)
+ continue;
+ if (dev->dev_num == dev_num)
+ return cfg;
+ }
+ }
+ return NULL;
+}
+
static struct vfio_container *
vfio_container_create(void)
{
@@ -541,6 +559,55 @@ vfio_setup_dma_mem(struct vfio_container *cfg)
return 0;
}
+static enum vfio_result
+vfio_cdev_assign_device(struct vfio_container *cfg, const char *sysfs_base,
+ const char *dev_addr, struct vfio_device **out_dev)
+{
+ struct vfio_device *dev, *found_dev;
+ enum vfio_result res;
+ int dev_num, ret;
+
+ /* get the cdev device number from sysfs */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Failed to get cdev device number for %s", dev_addr);
+ return VFIO_ERROR;
+ } else if (ret == 0) {
+ EAL_LOG(ERR, "Device %s not bound to vfio-pci cdev", dev_addr);
+ return VFIO_NOT_MANAGED;
+ }
+
+ /* do we already have this device? */
+ found_dev = vfio_cdev_get_dev_by_num(cfg, dev_num);
+ if (found_dev != NULL) {
+ EAL_LOG(DEBUG, "Device %s already assigned to this container", dev_addr);
+ *out_dev = found_dev;
+ return VFIO_EXISTS;
+ }
+ /* create new device structure */
+ dev = vfio_device_create(cfg, VFIO_DEVICE_TYPE_CDEV);
+ if (dev == NULL) {
+ EAL_LOG(ERR, "No space to track new VFIO cdev device");
+ return VFIO_NO_SPACE;
+ }
+ /* store device number */
+ dev->dev_num = dev_num;
+
+ /* set up our device now and store it in config */
+ ret = vfio_cdev_setup_device(cfg, dev);
+ if (ret < 0) {
+ EAL_LOG(ERR, "Cannot setup cdev device %s", dev_addr);
+ res = VFIO_ERROR;
+ goto err;
+ }
+ *out_dev = dev;
+ return VFIO_SUCCESS;
+
+err:
+ vfio_device_erase(cfg, dev);
+ return res;
+}
+
static enum vfio_result
vfio_group_assign_device(struct vfio_container *cfg, const char *sysfs_base,
const char *dev_addr, struct vfio_device **out_dev)
@@ -704,6 +771,50 @@ dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
return -1;
}
+ /*
+ * The device-to-container assignment is a complex problem to solve, for the following
+ * reasons:
+ *
+ * 1. PCI infrastructure is decoupled from VFIO, so PCI does not know anything about VFIO
+ *
+ * This means that while 99% of VFIO usage is PCI-related, we cannot communicate to PCI that
+ * we want to map a particular device using a particular container. Previously, this was
+ * achieved using back-channel communication via IOMMU group binding, so that whenever PCI
+ * map actually happens, VFIO knows which container to use, so this is roughly the model we
+ * are going with.
+ *
+ * 2. VFIO cannot depend on PCI because VFIO is in EAL
+ *
+ * We cannot "assign" a PCI device to container using rte_pci_device pointer because VFIO
+ * cannot depend on PCI definitions, nor can't we even assume that our device is in fact a
+ * PCI device, even though in practice this is true (at the time of this writing, FSLMC is
+ * the only bus doing non-PCI VFIO mappings, but FSLMC manages all VFIO infrastructure by
+ * itself, so in practice even counting FSLMC bus, we're always dealing with PCI devices).
+ *
+ * 3. The "assignment" means different things for group and cdev mode
+ *
+ * In group mode, to "bind" a device to a specific container, it is enough to bind its
+ * IOMMU group, so that when dev_vfio_setup_device() is called, we simply retrieve already
+ * existing group, and through that we figure out which container to use.
+ *
+ * For cdev mode, there are no "groups", so "assignment" either means we store some kind of
+ * uniquely identifying token (such as device number, or an opaque pointer), or we simply
+ * open the device straight away, and when dev_vfio_setup_device() comes we simply return
+ * the fd that was already opened at assign.
+ *
+ * Doing it the latter way (opening the device at assign for both group and cdev modes)
+ * actually solves all of these problems, so that's what we're going to do - the device
+ * setup API call will actually just assign the device to default container, while release
+ * will automatically cleanup and unassign anything that needs unassigned. There will be no
+ * "unassign" call, as it is not necessary.
+ *
+ * The cdev mode can deduplicate on "device number", which is assigned by the kernel and is
+ * unique per device. The group mode keeps its deduplication mechanism based on a combination
+ * of sysfs path and device address, which is unique per device as well. This way we can
+ * ensure that assigning a device before setting it up works correctly, and that we do not
+ * open the same device fd twice or leak any resources on device release.
+ */
+
if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
EAL_LOG(ERR, "VFIO support not initialized");
rte_errno = ENXIO;
@@ -724,6 +835,9 @@ dev_vfio_container_assign_device(int container_fd, const char *sysfs_base, const
case DEV_VFIO_MODE_NOIOMMU:
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
+ case DEV_VFIO_MODE_CDEV:
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
res = VFIO_NOT_SUPPORTED;
@@ -800,6 +914,24 @@ dev_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
res = vfio_group_assign_device(cfg, sysfs_base, dev_addr, &dev);
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ int dev_num;
+
+ /* find device number */
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, &dev_num);
+ if (ret < 0)
+ goto assign_fail;
+ else if (ret == 0)
+ goto not_managed;
+
+ cfg = vfio_container_get_by_dev_num(dev_num);
+ if (cfg == NULL)
+ cfg = vfio_global_cfg.default_cfg;
+
+ res = vfio_cdev_assign_device(cfg, sysfs_base, dev_addr, &dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -924,6 +1056,12 @@ dev_vfio_release_device(const char *sysfs_base __rte_unused,
}
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ /* for cdev, just erase the device and we're done */
+ vfio_device_erase(cfg, dev);
+ break;
+ }
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -988,9 +1126,25 @@ vfio_select_mode(void)
if (vfio_sync_mode(cfg, &mode) < 0)
goto err;
- /* primary handles DMA setup for default containers */
- cfg->dma_setup_done = true;
- return (enum dev_vfio_mode)mode;
+ if (mode == DEV_VFIO_MODE_CDEV) {
+ /* set up ops and sync IOAS */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_sync_ioas(cfg) < 0)
+ goto err;
+
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else if (mode == DEV_VFIO_MODE_GROUP ||
+ mode == DEV_VFIO_MODE_NOIOMMU) {
+ /* primary handles DMA */
+ cfg->dma_setup_done = true;
+ } else {
+ /* shouldn't happen */
+ EAL_LOG(ERR, "Unknown VFIO mode %d", mode);
+ goto err;
+ }
+
+ return mode;
}
/* if we failed mp sync setup, we cannot initialize VFIO */
if (vfio_mp_sync_setup() < 0)
@@ -1006,10 +1160,17 @@ vfio_select_mode(void)
return DEV_VFIO_MODE_NOIOMMU;
return DEV_VFIO_MODE_GROUP;
}
+ EAL_LOG(DEBUG, "VFIO group mode not available, trying cdev mode...");
+ /* cdev ops, IOAS and DMA setup are deferred to dev_vfio_init_mem() */
+ if (vfio_cdev_enable(cfg) == 0)
+ return DEV_VFIO_MODE_CDEV;
+ EAL_LOG(DEBUG, "VFIO cdev mode not available");
err_mpsync:
vfio_mp_sync_cleanup();
err:
vfio_container_erase(cfg);
+ /* reset ops as well */
+ vfio_global_cfg.ops = NULL;
return DEV_VFIO_MODE_NONE;
}
@@ -1020,6 +1181,7 @@ vfio_mode_to_str(enum dev_vfio_mode mode)
switch (mode) {
case DEV_VFIO_MODE_GROUP: return "group";
case DEV_VFIO_MODE_NOIOMMU: return "noiommu";
+ case DEV_VFIO_MODE_CDEV: return "cdev";
default: return "not initialized";
}
}
@@ -1066,6 +1228,36 @@ dev_vfio_enable(void)
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_init_mem)
+int
+dev_vfio_init_mem(void)
+{
+ struct vfio_container *cfg = vfio_global_cfg.default_cfg;
+
+ /* secondary already synced everything during dev_vfio_enable() */
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ return 0;
+
+ switch (vfio_global_cfg.mode) {
+ case DEV_VFIO_MODE_CDEV:
+ /* iommufd IOAS is populated once DPDK memory is available */
+ vfio_cdev_setup_ops();
+ if (vfio_cdev_setup_ioas(cfg) < 0)
+ return -1;
+ if (vfio_setup_dma_mem(cfg) < 0)
+ return -1;
+ if (vfio_register_mem_event_callback() < 0)
+ return -1;
+ cfg->dma_setup_done = true;
+ break;
+ default:
+ /* group/noiommu map memory at device attach, or no VFIO */
+ break;
+ }
+
+ return 0;
+}
+
RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_is_enabled)
int
dev_vfio_is_enabled(const char *modname)
@@ -1158,6 +1350,40 @@ dev_vfio_get_group_num(const char *sysfs_base, const char *dev_addr, int *iommu_
return 0;
}
+RTE_EXPORT_INTERNAL_SYMBOL(dev_vfio_get_device_num)
+int
+dev_vfio_get_device_num(const char *sysfs_base, const char *dev_addr, int *device_num)
+{
+ int ret;
+
+ if (sysfs_base == NULL || dev_addr == NULL || device_num == NULL) {
+ rte_errno = EINVAL;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode == DEV_VFIO_MODE_NONE) {
+ EAL_LOG(ERR, "VFIO support not initialized");
+ rte_errno = ENXIO;
+ return -1;
+ }
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
+ ret = vfio_cdev_get_device_num(sysfs_base, dev_addr, device_num);
+ if (ret < 0) {
+ rte_errno = EINVAL;
+ return -1;
+ } else if (ret == 0) {
+ rte_errno = ENODEV;
+ return -1;
+ }
+ return 0;
+}
+
static int
vfio_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
uint64_t len, int do_map)
@@ -1357,6 +1583,27 @@ dev_vfio_container_create(void)
cfg->container_fd = container_fd;
break;
}
+ case DEV_VFIO_MODE_CDEV:
+ {
+ /* Open new iommufd for custom container */
+ container_fd = vfio_cdev_get_iommufd();
+ if (container_fd < 0) {
+ EAL_LOG(ERR, "Cannot open iommufd for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ cfg->container_fd = container_fd;
+
+ /* Set up IOAS for this container */
+ if (vfio_cdev_setup_ioas(cfg) < 0) {
+ EAL_LOG(ERR, "Cannot setup IOAS for cdev container");
+ rte_errno = EIO;
+ goto err;
+ }
+ /* we only need to set up IOAS for DMA to work */
+ cfg->dma_setup_done = true;
+ break;
+ }
default:
EAL_LOG(NOTICE, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
@@ -1416,6 +1663,13 @@ dev_vfio_container_destroy(int container_fd)
vfio_group_erase(cfg, grp);
}
break;
+ case DEV_VFIO_MODE_CDEV:
+ /* erase all devices */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ EAL_LOG(DEBUG, "Device vfio%d still open, closing", dev->dev_num);
+ vfio_device_erase(cfg, dev);
+ }
+ break;
default:
EAL_LOG(ERR, "Unsupported VFIO mode");
rte_errno = ENOTSUP;
diff --git a/lib/eal/linux/eal_vfio.h b/lib/eal/linux/eal_vfio.h
index 3cff4d4f91..586545f647 100644
--- a/lib/eal/linux/eal_vfio.h
+++ b/lib/eal/linux/eal_vfio.h
@@ -48,6 +48,7 @@ struct vfio_group {
enum vfio_device_type {
VFIO_DEVICE_TYPE_NONE = 0,
VFIO_DEVICE_TYPE_GROUP,
+ VFIO_DEVICE_TYPE_CDEV,
};
/* device tracking (common for group and cdev modes) */
@@ -55,9 +56,16 @@ struct vfio_device {
bool active;
enum vfio_device_type type;
int fd;
- int group; /**< back-reference to group list */
- char *sysfs_base; /**< sysfs path prefix */
- char *dev_addr; /**< device address */
+ union {
+ struct {
+ int dev_num; /**< device number, e.g., X in /dev/vfio/devices/vfioX */
+ };
+ struct {
+ int group; /**< back-reference to group list */
+ char *sysfs_base; /**< sysfs path prefix */
+ char *dev_addr; /**< device address */
+ };
+ };
};
/* group mode specific configuration */
@@ -68,13 +76,21 @@ struct vfio_group_config {
struct vfio_group groups[RTE_MAX_VFIO_GROUPS];
};
+/* cdev mode specific configuration */
+struct vfio_cdev_config {
+ uint32_t ioas_id;
+};
+
/* per-container configuration */
struct vfio_container {
bool active;
bool dma_setup_done;
int container_fd;
struct vfio_user_mem_maps mem_maps;
- struct vfio_group_config group_cfg;
+ union {
+ struct vfio_group_config group_cfg;
+ struct vfio_cdev_config cdev_cfg;
+ };
int n_devices;
struct vfio_device devices[RTE_MAX_VFIO_DEVICES];
};
@@ -169,6 +185,17 @@ int vfio_group_setup_iommu(struct vfio_container *cfg);
int vfio_group_setup_device_fd(const char *dev_addr,
struct vfio_group *grp, struct vfio_device *dev);
+/* cdev mode functions */
+int vfio_cdev_enable(struct vfio_container *cfg);
+void vfio_cdev_setup_ops(void);
+int vfio_cdev_setup_ioas(struct vfio_container *cfg);
+int vfio_cdev_sync_ioas(struct vfio_container *cfg);
+int vfio_cdev_get_iommufd(void);
+int vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr,
+ int *cdev_dev_num);
+struct vfio_device *vfio_cdev_get_dev_by_num(struct vfio_container *cfg, int cdev_dev_num);
+int vfio_cdev_setup_device(struct vfio_container *cfg, struct vfio_device *dev);
+
#define VFIO_MEM_EVENT_CLB_NAME "vfio_mem_event_clb"
#define VFIO_MODNAME "vfio"
#define EAL_VFIO_MP "eal_vfio_mp_sync"
@@ -176,6 +203,8 @@ int vfio_group_setup_device_fd(const char *dev_addr,
#define VFIO_SOCKET_REQ_CONTAINER 0x100
#define VFIO_SOCKET_REQ_GROUP 0x200
#define VFIO_SOCKET_REQ_IOMMU_TYPE 0x400
+#define VFIO_SOCKET_REQ_CDEV 0x800
+#define VFIO_SOCKET_REQ_IOAS_ID 0x1000
#define VFIO_SOCKET_OK 0x0
#define VFIO_SOCKET_NO_FD 0x1
#define VFIO_SOCKET_ERR 0xFF
@@ -186,6 +215,8 @@ struct vfio_mp_param {
union {
int group_num;
int iommu_type_id;
+ int cdev_dev_num;
+ int ioas_id;
enum dev_vfio_mode mode;
};
};
diff --git a/lib/eal/linux/eal_vfio_cdev.c b/lib/eal/linux/eal_vfio_cdev.c
new file mode 100644
index 0000000000..360d3b9306
--- /dev/null
+++ b/lib/eal/linux/eal_vfio_cdev.c
@@ -0,0 +1,396 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2025 Intel Corporation
+ */
+
+#include <dirent.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <unistd.h>
+
+#include <uapi/linux/iommufd.h>
+#include <uapi/linux/vfio.h>
+
+#include <rte_log.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_string_fns.h>
+
+#include "eal_vfio.h"
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+
+static int vfio_cdev_dma_map(struct vfio_container *cfg);
+static int vfio_cdev_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr,
+ uint64_t iova, uint64_t len, int do_map);
+
+/* IOMMUFD cdev mode IOMMU operations */
+static const struct vfio_iommu_ops iommufd_ops = {
+ .type_id = 0, /* cdev mode doesn't use type_id */
+ .name = "IOMMUFD",
+ .partial_unmap = false,
+ .dma_map_func = &vfio_cdev_dma_map,
+ .dma_user_map_func = &vfio_cdev_dma_mem_map
+};
+
+void
+vfio_cdev_setup_ops(void)
+{
+ vfio_global_cfg.ops = &iommufd_ops;
+}
+
+static int
+vfio_cdev_dma_mem_map(struct vfio_container *cfg, uint64_t vaddr, uint64_t iova,
+ uint64_t len, int do_map)
+{
+ struct iommu_ioas_map ioas_map;
+ struct iommu_ioas_unmap ioas_unmap;
+ int ret;
+
+ if (do_map != 0) {
+ memset(&ioas_map, 0, sizeof(ioas_map));
+ ioas_map.size = sizeof(struct iommu_ioas_map);
+ ioas_map.flags = IOMMU_IOAS_MAP_FIXED_IOVA |
+ IOMMU_IOAS_MAP_READABLE |
+ IOMMU_IOAS_MAP_WRITEABLE;
+ ioas_map.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_map.user_va = vaddr;
+ ioas_map.length = len;
+ ioas_map.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_MAP, &ioas_map);
+ if (ret) {
+ /**
+ * In case the mapping was already done EEXIST will be
+ * returned from kernel.
+ */
+ if (errno == EEXIST) {
+ EAL_LOG(DEBUG,
+ "Memory segment is already mapped, skipping");
+ } else {
+ EAL_LOG(ERR,
+ "Cannot set up DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+ } else {
+ memset(&ioas_unmap, 0, sizeof(ioas_unmap));
+ ioas_unmap.size = sizeof(struct iommu_ioas_unmap);
+ ioas_unmap.ioas_id = cfg->cdev_cfg.ioas_id;
+ ioas_unmap.length = len;
+ ioas_unmap.iova = iova;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_UNMAP, &ioas_unmap);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot clear DMA remapping, error "
+ "%i (%s)", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+cdev_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms,
+ void *arg)
+{
+ struct vfio_container *cfg = arg;
+
+ /* skip external memory that isn't a heap */
+ if (msl->external && !msl->heap)
+ return 0;
+
+ /* skip any segments with invalid IOVA addresses */
+ if (ms->iova == RTE_BAD_IOVA)
+ return 0;
+
+ return vfio_cdev_dma_mem_map(cfg, ms->addr_64, ms->iova, ms->len, 1);
+}
+
+static int
+vfio_cdev_dma_map(struct vfio_container *cfg)
+{
+ return rte_memseg_walk(cdev_map, cfg);
+}
+
+int
+vfio_cdev_sync_ioas(struct vfio_container *cfg)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+
+ p->req = VFIO_SOCKET_REQ_IOAS_ID;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 && mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 0) {
+ cfg->cdev_cfg.ioas_id = p->ioas_id;
+ free(mp_reply.msgs);
+ return 0;
+ }
+ }
+
+ free(mp_reply.msgs);
+ EAL_LOG(ERR, "Cannot request ioas_id");
+ return -1;
+}
+
+int
+vfio_cdev_setup_ioas(struct vfio_container *cfg)
+{
+ struct iommu_ioas_alloc ioas_alloc;
+ int ret;
+
+ /* Allocate an IOAS */
+ memset(&ioas_alloc, 0, sizeof(ioas_alloc));
+ ioas_alloc.size = sizeof(struct iommu_ioas_alloc);
+ ioas_alloc.flags = 0;
+
+ ret = ioctl(cfg->container_fd, IOMMU_IOAS_ALLOC, &ioas_alloc);
+ if (ret) {
+ EAL_LOG(ERR, "Cannot allocate IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+ cfg->cdev_cfg.ioas_id = ioas_alloc.out_ioas_id;
+
+ EAL_LOG(DEBUG, "Allocated IOAS with ID %u", cfg->cdev_cfg.ioas_id);
+ return 0;
+}
+
+int
+vfio_cdev_get_iommufd(void)
+{
+ int iommufd;
+
+ /* if not requesting via mp, open iommufd locally */
+ iommufd = open(DEV_VFIO_IOMMUFD_PATH, O_RDWR);
+ if (iommufd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s",
+ DEV_VFIO_IOMMUFD_PATH, strerror(errno));
+ return -1;
+ }
+
+ return iommufd;
+}
+
+int
+vfio_cdev_enable(struct vfio_container *cfg)
+{
+ int iommufd;
+
+ /* Check if iommufd device exists */
+ if (access(DEV_VFIO_IOMMUFD_PATH, F_OK) != 0) {
+ EAL_LOG(DEBUG,
+ "IOMMUFD device does not exist, skipping VFIO cdev support...");
+ return 1;
+ }
+
+ /* open iommufd */
+ iommufd = vfio_cdev_get_iommufd();
+ if (iommufd < 0)
+ return -1;
+
+ cfg->container_fd = iommufd;
+ return 0;
+}
+
+int
+vfio_cdev_get_device_num(const char *sysfs_base, const char *dev_addr, int *cdev_dev_num)
+{
+ char linkname[PATH_MAX];
+ char filename[PATH_MAX];
+ char *dev_tok, *end;
+ int dev_num;
+ DIR *dir;
+ struct dirent *entry;
+
+ memset(linkname, 0, sizeof(linkname));
+ memset(filename, 0, sizeof(filename));
+
+ /* check if vfio-dev directory exists for this device */
+ snprintf(linkname, sizeof(linkname),
+ "%s/%s/vfio-dev", sysfs_base, dev_addr);
+
+ dir = opendir(linkname);
+ if (dir == NULL) {
+ /* device doesn't have vfio-dev, not bound to vfio-pci cdev */
+ return 0;
+ }
+
+ /* find vfioX entry in vfio-dev directory */
+ while ((entry = readdir(dir)) != NULL) {
+ if (strncmp(entry->d_name, "vfio", 4) == 0) {
+ /* parse device number from vfioX */
+ errno = 0;
+ dev_tok = entry->d_name + 4; /* skip "vfio" prefix */
+ end = dev_tok;
+ dev_num = strtol(dev_tok, &end, 10);
+ if (end == dev_tok || *end != '\0' || errno != 0) {
+ EAL_LOG(ERR, "%s error parsing VFIO cdev device number!",
+ dev_addr);
+ closedir(dir);
+ return -1;
+ }
+ *cdev_dev_num = dev_num;
+ closedir(dir);
+ return 1;
+ }
+ }
+
+ closedir(dir);
+ /* no vfio device found */
+ return 0;
+}
+
+struct vfio_device *
+vfio_cdev_get_dev_by_num(struct vfio_container *cfg, int cdev_dev_num)
+{
+ struct vfio_device *dev;
+ /* find device handle */
+ VFIO_DEVICE_FOREACH_ACTIVE(cfg, dev) {
+ if (dev->dev_num != cdev_dev_num)
+ continue;
+ return dev;
+ }
+ return NULL;
+}
+
+static int
+cdev_open_device_fd(int cdev_dev_num)
+{
+ char devname[PATH_MAX] = {0};
+ int dev_fd;
+
+ snprintf(devname, sizeof(devname), "%s/vfio%d",
+ DEV_VFIO_CDEV_DEVICES_PATH, cdev_dev_num);
+
+ dev_fd = open(devname, O_RDWR);
+ if (dev_fd < 0) {
+ EAL_LOG(ERR, "Cannot open %s: %s", devname, strerror(errno));
+ return -1;
+ }
+
+ return dev_fd;
+}
+
+static int
+cdev_attach_device_to_iommufd(struct vfio_container *cfg, struct vfio_device *dev)
+{
+ struct vfio_device_bind_iommufd bind = {0};
+ struct vfio_device_attach_iommufd_pt attach = {0};
+ rte_uuid_t vf_token;
+
+ rte_eal_vfio_get_vf_token(vf_token);
+
+ /* try with token first */
+ if (!rte_uuid_is_null(vf_token)) {
+ bind.flags = VFIO_DEVICE_BIND_FLAG_TOKEN;
+ bind.token_uuid_ptr = (uintptr_t)&vf_token;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ /* this may fail because the kernel is too old */
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(DEBUG, "Failed to bind device %d with VF token", dev->dev_num);
+ EAL_LOG(NOTICE, "Unable to use VF tokens with current kernel version.");
+ EAL_LOG(NOTICE, "Please use kernel >=6.17 or use group mode.");
+ /* erase the bind structure */
+ bind = (struct vfio_device_bind_iommufd){0};
+ } else {
+ goto attach;
+ }
+ }
+ bind.flags = 0;
+ bind.argsz = sizeof(bind);
+ bind.iommufd = cfg->container_fd;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind) < 0) {
+ EAL_LOG(ERR, "Cannot bind device to IOMMUFD, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+attach:
+ /* attach device to IOAS */
+ attach.argsz = sizeof(attach);
+ attach.flags = 0;
+ attach.pt_id = cfg->cdev_cfg.ioas_id;
+
+ if (ioctl(dev->fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach) < 0) {
+ EAL_LOG(ERR, "Cannot attach device to IOAS, error %i (%s)",
+ errno, strerror(errno));
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+vfio_cdev_request_dev_fd(struct vfio_device *dev)
+{
+ struct rte_mp_msg mp_req, *mp_rep;
+ struct rte_mp_reply mp_reply = {0};
+ struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+ struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
+ int device_fd = -1;
+
+ /* secondary process requests device fd from primary */
+ p->req = VFIO_SOCKET_REQ_CDEV;
+ p->cdev_dev_num = dev->dev_num;
+ rte_strscpy(mp_req.name, EAL_VFIO_MP, sizeof(mp_req.name));
+ mp_req.len_param = sizeof(*p);
+ mp_req.num_fds = 0;
+
+ if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
+ mp_reply.nb_received == 1) {
+ mp_rep = &mp_reply.msgs[0];
+ p = (struct vfio_mp_param *)mp_rep->param;
+ if (p->result == VFIO_SOCKET_OK && mp_rep->num_fds == 1)
+ device_fd = mp_rep->fds[0];
+ }
+
+ free(mp_reply.msgs);
+
+ if (device_fd < 0) {
+ EAL_LOG(ERR, "Cannot request device fd for vfio%d", dev->dev_num);
+ return -1;
+ }
+ dev->fd = device_fd;
+
+ return 0;
+}
+
+int
+vfio_cdev_setup_device(struct vfio_container *cfg, struct vfio_device *dev)
+{
+ int device_fd;
+
+ /* get device fd - primary or custom container opens it, secondary requests from primary */
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY || !vfio_container_is_default(cfg)) {
+ device_fd = cdev_open_device_fd(dev->dev_num);
+ if (device_fd < 0)
+ return -1;
+ dev->fd = device_fd;
+
+ /* attach device to iommufd */
+ if (cdev_attach_device_to_iommufd(cfg, dev) < 0) {
+ close(device_fd);
+ dev->fd = -1;
+ return -1;
+ }
+ } else if (vfio_cdev_request_dev_fd(dev) < 0) {
+ return -1;
+ }
+ return 0;
+}
diff --git a/lib/eal/linux/eal_vfio_mp_sync.c b/lib/eal/linux/eal_vfio_mp_sync.c
index e1f82ad77b..8e989d17ab 100644
--- a/lib/eal/linux/eal_vfio_mp_sync.c
+++ b/lib/eal/linux/eal_vfio_mp_sync.c
@@ -93,6 +93,48 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
}
break;
}
+ case VFIO_SOCKET_REQ_CDEV:
+ {
+ struct vfio_container *cfg;
+ struct vfio_device *dev;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_CDEV;
+ r->cdev_dev_num = m->cdev_dev_num;
+
+ cfg = vfio_global_cfg.default_cfg;
+ dev = vfio_cdev_get_dev_by_num(cfg, m->cdev_dev_num);
+ if (dev == NULL) {
+ r->result = VFIO_SOCKET_NO_FD;
+ } else {
+ r->result = VFIO_SOCKET_OK;
+ reply.num_fds = 1;
+ reply.fds[0] = dev->fd;
+ }
+ break;
+ }
+ case VFIO_SOCKET_REQ_IOAS_ID:
+ {
+ struct vfio_container *cfg;
+
+ if (vfio_global_cfg.mode != DEV_VFIO_MODE_CDEV) {
+ EAL_LOG(ERR, "VFIO not initialized in cdev mode");
+ r->result = VFIO_SOCKET_ERR;
+ break;
+ }
+
+ r->req = VFIO_SOCKET_REQ_IOAS_ID;
+ cfg = vfio_global_cfg.default_cfg;
+ r->ioas_id = cfg->cdev_cfg.ioas_id;
+
+ r->result = VFIO_SOCKET_OK;
+ break;
+ }
default:
EAL_LOG(ERR, "vfio received invalid message!");
return -1;
diff --git a/lib/eal/linux/meson.build b/lib/eal/linux/meson.build
index 5ec8eddaa2..c164a30b49 100644
--- a/lib/eal/linux/meson.build
+++ b/lib/eal/linux/meson.build
@@ -16,6 +16,7 @@ sources += files(
'eal_thread.c',
'eal_timer.c',
'eal_vfio.c',
+ 'eal_vfio_cdev.c',
'eal_vfio_group.c',
'eal_vfio_mp_sync.c',
)
--
2.52.0
^ permalink raw reply related [flat|nested] 203+ messages in thread
* Re: [PATCH v14 00/25] Support VFIO cdev API in DPDK
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
` (24 preceding siblings ...)
2026-08-28 13:06 ` [PATCH v14 25/25] vfio: introduce cdev mode Anatoly Burakov
@ 2026-08-28 13:32 ` Burakov, Anatoly
25 siblings, 0 replies; 203+ messages in thread
From: Burakov, Anatoly @ 2026-08-28 13:32 UTC (permalink / raw)
To: dev
On 8/28/2026 3:06 PM, Anatoly Burakov wrote:
> This patchset depends on the following:
> * https://patches.dpdk.org/project/dpdk/list/?series=39072 ("net/nbl: fix use-after-free")
>
Question on CI: what is the proper way to specify patchset apply
dependencies? The first patch in this series contains a "Depends-on:"
tag which is supposed to trigger apply of that patch before applying
this patchset, but apparently it's not doing that. Do I also have to
specify something in the cover letter, not just in first patch?
https://doc.dpdk.org/guides/contributing/patches.html#patch-dependencies
This page suggests that it can be "commit log" (which I assume is
patches) *or* cover letter. This can be read as either being valid. Is
my understanding correct?
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 203+ messages in thread
end of thread, other threads:[~2026-08-28 13:32 UTC | newest]
Thread overview: 203+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <cover.1763141462.git.anatoly.burakov@intel.com>
2025-11-18 16:29 ` [PATCH v3 00/20] Support VFIO cdev API in DPDK Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 01/20] doc: add deprecation notice for VFIO API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 02/20] doc: add deprecation notice for vDPA driver API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 03/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2025-11-18 17:36 ` Stephen Hemminger
2025-11-18 16:29 ` [PATCH v3 04/20] vfio: make all functions internal Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 05/20] vfio: split get device info from setup Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 06/20] vfio: add container device assignment API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 07/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 08/20] net/ntnic: use container device assignment API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 09/20] vdpa/ifc: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 10/20] vdpa/nfp: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 11/20] vdpa/sfc: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 12/20] vhost: remove group-related API from drivers Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 13/20] vfio: remove group-based API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 14/20] vfio: cleanup and refactor Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 16/20] bus/fslmc: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 17/20] net/hinic3: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 18/20] net/ntnic: " Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
2025-11-18 16:29 ` [PATCH v3 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 02/18] vfio: make all functions internal Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 03/18] vfio: split get device info from setup Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 04/18] vfio: add container device assignment API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 06/18] net/ntnic: use container device assignment API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 07/18] vdpa/ifc: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 08/18] vdpa/nfp: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 09/18] vdpa/sfc: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 10/18] vhost: remove group-related API from drivers Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 11/18] vfio: remove group-based API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 12/18] vfio: cleanup and refactor Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 14/18] bus/fslmc: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 15/18] net/hinic3: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 16/18] net/ntnic: " Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
2026-02-26 14:17 ` [PATCH v7 18/18] vfio: introduce cdev mode Anatoly Burakov
2026-05-01 22:45 ` [PATCH v7 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
2026-06-11 15:08 ` [PATCH v8 " Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 01/18] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 02/18] vfio: make all functions internal Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 03/18] vfio: split get device info from setup Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 04/18] vfio: add container device assignment API Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 05/18] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 06/18] net/ntnic: use container device assignment API Anatoly Burakov
2026-06-11 15:08 ` [PATCH v8 07/18] vdpa/ifc: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 08/18] vdpa/nfp: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 09/18] vdpa/sfc: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 10/18] vhost: remove group-related API from drivers Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 11/18] vfio: remove group-based API Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 12/18] vfio: cleanup and refactor Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 13/18] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 14/18] bus/fslmc: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 15/18] net/hinic3: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 16/18] net/ntnic: " Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 17/18] vfio: remove no-IOMMU check API Anatoly Burakov
2026-06-11 15:09 ` [PATCH v8 18/18] vfio: introduce cdev mode Anatoly Burakov
2026-06-11 17:49 ` [PATCH v8 00/18] Support VFIO cdev API in DPDK Stephen Hemminger
2026-08-05 13:45 ` [PATCH v9 00/19] " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 01/19] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 02/19] vfio: make all functions internal Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 03/19] vfio: split get device info from setup Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 04/19] vfio: add container device assignment API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 05/19] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 06/19] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 07/19] vdpa/ifc: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 08/19] vdpa/nfp: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 09/19] vdpa/sfc: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 10/19] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 11/19] vhost: remove group-related API from driver Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 12/19] vfio: remove group-based API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 13/19] vfio: cleanup and refactor Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 14/19] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 15/19] bus/fslmc: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 16/19] net/hinic3: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 17/19] net/ntnic: " Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 18/19] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-05 13:45 ` [PATCH v9 19/19] vfio: introduce cdev mode Anatoly Burakov
2026-08-06 3:52 ` [PATCH v9 00/19] Support VFIO cdev API in DPDK Stephen Hemminger
2026-08-06 14:11 ` [PATCH v10 00/20] " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 02/20] vfio: make all functions internal Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 04/20] vfio: split get device info from setup Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 05/20] vfio: add container device assignment API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 07/20] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 08/20] vdpa/ifc: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 09/20] vdpa/nfp: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 10/20] vdpa/sfc: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 12/20] vhost: remove group-related API from driver Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 13/20] vfio: remove group-based API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 14/20] vfio: cleanup and refactor Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 16/20] bus/fslmc: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 17/20] net/hinic3: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 18/20] net/ntnic: " Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-06 14:11 ` [PATCH v10 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-06 17:13 ` [PATCH v10 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
2026-08-07 14:46 ` [PATCH v11 " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 02/20] vfio: make all functions internal Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 04/20] vfio: split get device info from setup Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 05/20] vfio: add container device assignment API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 07/20] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 08/20] vdpa/ifc: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 09/20] vdpa/nfp: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 10/20] vdpa/sfc: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 12/20] vhost: remove group-related API from driver Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 13/20] vfio: remove group-based API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 14/20] vfio: cleanup and refactor Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 16/20] bus/fslmc: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 17/20] net/hinic3: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 18/20] net/ntnic: " Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-07 14:46 ` [PATCH v11 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-11 0:30 ` [PATCH v11 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
2026-08-25 13:20 ` [PATCH v12 " Anatoly Burakov
2026-08-25 13:20 ` [PATCH v12 01/20] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-25 14:08 ` Stephen Hemminger
2026-08-25 13:20 ` [PATCH v12 02/20] vfio: make all functions internal Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 03/20] bus/pci: rename mismatching error labels Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 04/20] vfio: split get device info from setup Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 05/20] vfio: add container device assignment API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 06/20] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 07/20] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 08/20] vdpa/ifc: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 09/20] vdpa/nfp: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 10/20] vdpa/sfc: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 11/20] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 12/20] vhost: remove group-related API from driver Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 13/20] vfio: remove group-based API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 14/20] vfio: cleanup and refactor Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 15/20] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 16/20] bus/fslmc: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 17/20] net/hinic3: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 18/20] net/ntnic: " Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 19/20] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-25 13:21 ` [PATCH v12 20/20] vfio: introduce cdev mode Anatoly Burakov
2026-08-25 14:17 ` [PATCH v12 00/20] Support VFIO cdev API in DPDK Stephen Hemminger
2026-08-26 10:09 ` Burakov, Anatoly
2026-08-27 15:14 ` [PATCH v13 00/24] " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 01/24] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 02/24] vfio: make all functions internal Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 03/24] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 04/24] vfio: decouple EAL from VFIO internals Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 05/24] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 06/24] vfio: rename API to reflect internal status Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 07/24] bus/pci: rename mismatching error labels Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 08/24] vfio: split get device info from setup Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 09/24] vfio: add container device assignment API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 10/24] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 11/24] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 12/24] vdpa/ifc: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 13/24] vdpa/nfp: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 14/24] vdpa/sfc: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 15/24] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 16/24] vhost: remove group-related API from driver Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 17/24] vfio: remove group-based API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 18/24] vfio: cleanup and refactor Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 19/24] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 20/24] bus/fslmc: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 21/24] net/hinic3: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 22/24] net/ntnic: " Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 23/24] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-27 15:14 ` [PATCH v13 24/24] vfio: introduce cdev mode Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 01/25] uapi: update to v6.17 and add iommufd.h Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 02/25] vfio: make all functions internal Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 03/25] bus/fslmc: decouple from EAL VFIO Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 04/25] vfio: decouple EAL from VFIO internals Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 05/25] vfio: do proper teardown on VFIO cleanup Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 06/25] vfio: remove modname parameter from init Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 07/25] vfio: rename API to reflect internal status Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 08/25] bus/pci: rename mismatching error labels Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 09/25] vfio: split get device info from setup Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 10/25] vfio: add container device assignment API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 11/25] net/nbl: do not use VFIO group bind API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 12/25] net/ntnic: use container device assignment API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 13/25] vdpa/ifc: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 14/25] vdpa/nfp: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 15/25] vdpa/sfc: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 16/25] vdpa/mlx5: remove group-related API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 17/25] vhost: remove group-related API from driver Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 18/25] vfio: remove group-based API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 19/25] vfio: cleanup and refactor Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 20/25] bus/pci: use the new VFIO mode API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 21/25] bus/fslmc: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 22/25] net/hinic3: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 23/25] net/ntnic: " Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 24/25] vfio: remove no-IOMMU check API Anatoly Burakov
2026-08-28 13:06 ` [PATCH v14 25/25] vfio: introduce cdev mode Anatoly Burakov
2026-08-28 13:32 ` [PATCH v14 00/25] Support VFIO cdev API in DPDK Burakov, Anatoly
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox