* [RFC PATCH 1/3] vfio/pci: Introduce library allocating from Interrupt Message Store (IMS)
2023-08-24 16:15 [RFC PATCH 0/3] vfio/ims: Back guest interrupts from Interrupt Message Store (IMS) Reinette Chatre
@ 2023-08-24 16:15 ` Reinette Chatre
2023-08-24 16:15 ` [RFC PATCH 2/3] vfio/ims: Support emulated interrupts Reinette Chatre
2023-08-24 16:15 ` [RFC PATCH 3/3] vfio/ims: Add helper that returns IMS index Reinette Chatre
2 siblings, 0 replies; 10+ messages in thread
From: Reinette Chatre @ 2023-08-24 16:15 UTC (permalink / raw)
To: jgg, yishaih, shameerali.kolothum.thodi, kevin.tian,
alex.williamson
Cc: kvm, dave.jiang, jing2.liu, ashok.raj, fenghua.yu, tom.zanussi,
reinette.chatre, linux-kernel
With Interrupt Message Store (IMS) support introduced in
commit 0194425af0c8 ("PCI/MSI: Provide IMS (Interrupt Message Store)
support") a device can create a secondary interrupt domain that works
side by side with MSI-X on the same device. IMS allows for
implementation-specific interrupt storage that is managed by the
implementation specific interrupt chip associated with the IMS domain
at the time it (the IMS domain) is created for the device via
pci_create_ims_domain().
An example usage of IMS is for devices that can have their resources
assigned to guests with varying granularity. For example, an
accelerator device may support many workqueues and a single workqueue
can be composed into a virtual device for use by a guest. Using
IMS interrupts for the guest preserves MSI-X for host usage while
allowing a significantly larger number of interrupt vectors than
allowed by MSI-X. All while enabling usage of the same device driver
in the host and guest.
VFIO PCI IMS is a library that supports virtual devices with
MSI-X interrupts that are backed by IMS interrupts on the host.
Introduce core functionality of VFIO PCI IMS consisting of IMS interrupt
allocation and free, and ability to associate domain specific
information with each interrupt. With the generic term of "cookie"
(the same name used in the IMS core) this value is opaque to VFIO
PCI IMS but important to the virtual device, for example a PASID.
The core VFIO PCI IMS API is:
vfio_pci_ims_init(): Initialize the IMS interrupt context. VFIO PCI
IMS needs to allocate from a device specific IMS domain. The
PCI device owning the IMS domain as well as the default cookie are
provided using vfio_pci_ims_init(). vfio_pci_ims_init() is intended
to be called from the virtual device's vfio_device_ops->open_device().
vfio_pci_ims_free(): Free the IMS interrupt context. vfio_pci_ims_free()
is intended to be called from the virtual device's
vfio_device_ops->close_device() after the index as a whole has been
disabled via vfio_pci_set_ims_trigger().
vfio_pci_ims_set_cookie(): Set a unique cookie for a particular vector.
Must be called after vfio_pci_ims_init(). For example, the virtual
device driver may intercept the guest's MMIO write that configures
a new PASID. Calling vfio_pci_ims_set_cookie() with the new PASID
value enables subsequent interrupts to be allocated with accurate
data.
vfio_pci_set_ims_trigger(): Runtime helper intended to be called
via the VFIO_DEVICE_SET_IRQS ioctl() and more directly on device
close to disable the index as whole. vfio_pci_set_ims_trigger()
allocates and frees IMS interrupts on the host as MSI-X interrupts
in the guest change state. Interrupts are always dynamically
allocated, enabling a user of this helper to clear the
VFIO_IRQ_INFO_NORESIZE flag in response to user space's request
for interrupt information.
While interrupts are dynamically allocated and freed the interrupt
contexts are maintained from initial allocation to virtual device
shutdown. This is done to maintain the vector specific cookies.
The interrupt context can be modified via separate flows. For example,
vfio_pci_set_ims_trigger() can modify the context while handling an
ioctl(), while vfio_pci_ims_set_cookie() can modify the interrupt
context as part of handling an intercepted MMIO write. Concurrent
changes to the interrupt context are protected by a mutex (ctx_mutex).
VFIO PCI IMS manages IMS interrupts of a PCI device. This prompts
it to be located in drivers/vfio/pci, enabled with a new config option
CONFIG_VFIO_PCI_IMS. The library does not depend on the PCI
VFIO bus driver so there is not a dependency on CONFIG_VFIO_PCI.
Originally-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
drivers/vfio/pci/Kconfig | 12 +
drivers/vfio/pci/Makefile | 2 +
drivers/vfio/pci/vfio_pci_ims.c | 389 ++++++++++++++++++++++++++++++++
include/linux/vfio.h | 60 +++++
4 files changed, 463 insertions(+)
create mode 100644 drivers/vfio/pci/vfio_pci_ims.c
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 86bb7835cf3c..cf9e4ee14574 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -24,6 +24,18 @@ config VFIO_PCI
If you don't know what to do here, say N.
+config VFIO_PCI_IMS
+ tristate "VFIO library supporting Interrupt Message Store (IMS)"
+ depends on PCI_MSI
+ select EVENTFD
+ select IRQ_BYPASS_MANAGER
+ default n
+ help
+ VFIO library used by virtual devices with MSI-X interrupts
+ that are backed by IMS interrupts on the host.
+
+ If you don't know what to do here, say N.
+
if VFIO_PCI
config VFIO_PCI_VGA
bool "Generic VFIO PCI support for VGA devices"
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 24c524224da5..860e1ff5ef60 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -8,6 +8,8 @@ vfio-pci-y := vfio_pci.o
vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
+obj-$(CONFIG_VFIO_PCI_IMS) += vfio_pci_ims.o
+
obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/
obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
diff --git a/drivers/vfio/pci/vfio_pci_ims.c b/drivers/vfio/pci/vfio_pci_ims.c
new file mode 100644
index 000000000000..0926eb921351
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_ims.c
@@ -0,0 +1,389 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Interrupt Message Store (IMS) library
+ *
+ * Copyright (C) 2023 Intel Corporation
+ */
+
+#include <linux/device.h>
+#include <linux/eventfd.h>
+#include <linux/interrupt.h>
+#include <linux/irqbypass.h>
+#include <linux/irqreturn.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+#include <linux/vfio.h>
+#include <linux/xarray.h>
+
+/*
+ * IMS interrupt context.
+ * @name: Name of device associated with interrupt.
+ * Provided to request_irq().
+ * @trigger: eventfd associated with interrupt.
+ * @producer: Interrupt's registered IRQ bypass producer.
+ * @ims_id: Interrupt index associated with IMS interrupt.
+ * @virq: Linux IRQ number associated with IMS interrupt.
+ * @icookie: Cookie used by irqchip driver.
+ */
+struct vfio_pci_ims_ctx {
+ char *name;
+ struct eventfd_ctx *trigger;
+ struct irq_bypass_producer producer;
+ int ims_id;
+ int virq;
+ union msi_instance_cookie icookie;
+};
+
+static irqreturn_t vfio_pci_ims_irq_handler(int irq, void *arg)
+{
+ struct eventfd_ctx *trigger = arg;
+
+ eventfd_signal(trigger, 1);
+ return IRQ_HANDLED;
+}
+
+/*
+ * Free the interrupt associated with @ctx.
+ *
+ * Free interrupt from the underlying PCI device's IMS domain.
+ */
+static void vfio_pci_ims_irq_free(struct vfio_pci_ims *ims,
+ struct vfio_pci_ims_ctx *ctx)
+{
+ struct msi_map irq_map = {};
+
+ lockdep_assert_held(&ims->ctx_mutex);
+
+ irq_map.index = ctx->ims_id;
+ irq_map.virq = ctx->virq;
+ pci_ims_free_irq(ims->pdev, irq_map);
+ ctx->ims_id = -EINVAL;
+ ctx->virq = 0;
+}
+
+/*
+ * Allocate an interrupt for @ctx.
+ *
+ * Allocate interrupt from the underlying PCI device's IMS domain.
+ */
+static int vfio_pci_ims_irq_alloc(struct vfio_pci_ims *ims,
+ struct vfio_pci_ims_ctx *ctx)
+{
+ struct msi_map irq_map = {};
+
+ lockdep_assert_held(&ims->ctx_mutex);
+
+ irq_map = pci_ims_alloc_irq(ims->pdev, &ctx->icookie, NULL);
+ if (irq_map.index < 0)
+ return irq_map.index;
+
+ ctx->ims_id = irq_map.index;
+ ctx->virq = irq_map.virq;
+
+ return 0;
+}
+
+/*
+ * Return interrupt context for @vector.
+ *
+ * Interrupt contexts are not freed until shutdown so first
+ * check if there is a context associated with @vector that
+ * should be returned before allocating new context.
+ *
+ * Return: pointer to interrupt context, NULL on failure.
+ */
+static struct vfio_pci_ims_ctx *
+vfio_pci_ims_ctx_get(struct vfio_pci_ims *ims, unsigned int vector)
+{
+ struct vfio_pci_ims_ctx *ctx;
+ int ret;
+
+ lockdep_assert_held(&ims->ctx_mutex);
+
+ ctx = xa_load(&ims->ctx, vector);
+ if (ctx)
+ return ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
+ if (!ctx)
+ return NULL;
+
+ ctx->icookie = ims->default_cookie;
+ ret = xa_insert(&ims->ctx, vector, ctx, GFP_KERNEL_ACCOUNT);
+ if (ret) {
+ kfree(ctx);
+ return NULL;
+ }
+
+ return ctx;
+}
+
+static int vfio_pci_ims_set_vector_signal(struct vfio_device *vdev,
+ unsigned int vector, int fd)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct device *dev = &vdev->device;
+ struct vfio_pci_ims_ctx *ctx;
+ struct eventfd_ctx *trigger;
+ int ret;
+
+ lockdep_assert_held(&ims->ctx_mutex);
+
+ ctx = xa_load(&ims->ctx, vector);
+
+ if (ctx && ctx->trigger) {
+ irq_bypass_unregister_producer(&ctx->producer);
+ free_irq(ctx->virq, ctx->trigger);
+ vfio_pci_ims_irq_free(ims, ctx);
+ kfree(ctx->name);
+ ctx->name = NULL;
+ eventfd_ctx_put(ctx->trigger);
+ ctx->trigger = NULL;
+ }
+
+ if (fd < 0)
+ return 0;
+
+ /* Interrupt contexts remain allocated until shutdown. */
+ ctx = vfio_pci_ims_ctx_get(ims, vector);
+ if (!ctx)
+ return -EINVAL;
+
+ ctx->name = kasprintf(GFP_KERNEL, "vfio-ims[%d](%s)", vector,
+ dev_name(dev));
+ if (!ctx->name)
+ return -ENOMEM;
+
+ trigger = eventfd_ctx_fdget(fd);
+ if (IS_ERR(trigger)) {
+ ret = PTR_ERR(trigger);
+ goto out_free_name;
+ }
+
+ ctx->trigger = trigger;
+
+ ret = vfio_pci_ims_irq_alloc(ims, ctx);
+ if (ret < 0)
+ goto out_put_eventfd_ctx;
+
+ ret = request_irq(ctx->virq, vfio_pci_ims_irq_handler, 0, ctx->name,
+ ctx->trigger);
+ if (ret < 0)
+ goto out_free_irq;
+
+ ctx->producer.token = ctx->trigger;
+ ctx->producer.irq = ctx->virq;
+ ret = irq_bypass_register_producer(&ctx->producer);
+ if (unlikely(ret)) {
+ dev_info(&vdev->device,
+ "irq bypass producer (token %p) registration fails: %d\n",
+ &ctx->producer.token, ret);
+ ctx->producer.token = NULL;
+ }
+
+ return 0;
+
+out_free_irq:
+ vfio_pci_ims_irq_free(ims, ctx);
+out_put_eventfd_ctx:
+ eventfd_ctx_put(ctx->trigger);
+ ctx->trigger = NULL;
+out_free_name:
+ kfree(ctx->name);
+ ctx->name = NULL;
+ return ret;
+}
+
+static int vfio_pci_ims_set_block(struct vfio_device *vdev, unsigned int start,
+ unsigned int count, int *fds)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ unsigned int i, j;
+ int ret = 0;
+
+ lockdep_assert_held(&ims->ctx_mutex);
+
+ for (i = 0, j = start; i < count && !ret; i++, j++) {
+ int fd = fds ? fds[i] : -1;
+
+ ret = vfio_pci_ims_set_vector_signal(vdev, j, fd);
+ }
+
+ if (ret) {
+ for (i = start; i < j; i++)
+ vfio_pci_ims_set_vector_signal(vdev, i, -1);
+ }
+
+ return ret;
+}
+
+/*
+ * Manage Interrupt Message Store (IMS) interrupts on the host that are
+ * backing guest MSI-X vectors.
+ *
+ * @vdev: VFIO device
+ * @index: Type of guest vectors to set up. Must be
+ * VFIO_PCI_MSIX_IRQ_INDEX.
+ * @start: First vector index.
+ * @count: Number of vectors.
+ * @flags: Type of data provided in @data.
+ * @data: Data as specified by @flags.
+ *
+ * Caller is required to validate provided range for @vdev.
+ *
+ * Context: Interrupt context must be initialized via vfio_pci_ims_init()
+ * before any interrupts can be allocated.
+ * Can be called from vfio_device_ops->ioctl() or during shutdown via
+ * vfio_device_ops->close_device().
+ *
+ * Return: Error code on failure or 0 on success.
+ */
+int vfio_pci_set_ims_trigger(struct vfio_device *vdev, unsigned int index,
+ unsigned int start, unsigned int count, u32 flags,
+ void *data)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+ unsigned long i;
+ int ret;
+
+ if (index != VFIO_PCI_MSIX_IRQ_INDEX)
+ return -EINVAL;
+
+ mutex_lock(&ims->ctx_mutex);
+ if (!count && (flags & VFIO_IRQ_SET_DATA_NONE)) {
+ xa_for_each(&ims->ctx, i, ctx)
+ vfio_pci_ims_set_vector_signal(vdev, i, -1);
+ ret = 0;
+ goto out_unlock;
+ }
+
+ if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {
+ ret = vfio_pci_ims_set_block(vdev, start, count, (int *)data);
+ goto out_unlock;
+ }
+
+ for (i = start; i < start + count; i++) {
+ ctx = xa_load(&ims->ctx, i);
+ if (!ctx || !ctx->trigger)
+ continue;
+ if (flags & VFIO_IRQ_SET_DATA_NONE) {
+ eventfd_signal(ctx->trigger, 1);
+ } else if (flags & VFIO_IRQ_SET_DATA_BOOL) {
+ u8 *bools = data;
+
+ if (bools[i - start])
+ eventfd_signal(ctx->trigger, 1);
+ }
+ }
+
+ ret = 0;
+
+out_unlock:
+ mutex_unlock(&ims->ctx_mutex);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_set_ims_trigger);
+
+/*
+ * Initialize the IMS context associated with virtual device.
+ *
+ * @vdev: VFIO device
+ * @pdev: PCI device that owns the IMS domain from where IMS
+ * interrupts will be allocated.
+ * @default_cookie: The default cookie for new IMS instances that do
+ * not have an instance-specific cookie.
+ *
+ * Context: Must be called during vfio_device_ops->open_device().
+ */
+void vfio_pci_ims_init(struct vfio_device *vdev, struct pci_dev *pdev,
+ union msi_instance_cookie *default_cookie)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+
+ xa_init(&ims->ctx);
+ mutex_init(&ims->ctx_mutex);
+ ims->pdev = pdev;
+ ims->default_cookie = *default_cookie;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_init);
+
+/*
+ * Free the IMS context associated with virtual device.
+ *
+ * @vdev: VFIO device
+ *
+ * Virtual device has to free all allocated interrupts before freeing the
+ * IMS context. This is done by triggering a call to disable the index as
+ * a whole by triggering vfio_pci_set_ims_trigger() with
+ * flags = (DATA_NONE|ACTION_TRIGGER), count = 0.
+ *
+ * Context: Must be called during vfio_device_ops->close_device() after
+ * index as a whole has been disabled.
+ */
+void vfio_pci_ims_free(struct vfio_device *vdev)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+ unsigned long i;
+
+ /*
+ * All interrupts should be freed (including free of name and
+ * trigger) before context cleanup.
+ */
+ mutex_lock(&ims->ctx_mutex);
+ xa_for_each(&ims->ctx, i, ctx) {
+ WARN_ON_ONCE(ctx->trigger);
+ WARN_ON_ONCE(ctx->name);
+ xa_erase(&ims->ctx, i);
+ kfree(ctx);
+ }
+ mutex_unlock(&ims->ctx_mutex);
+ ims->pdev = NULL;
+ ims->default_cookie = (union msi_instance_cookie) { .value = 0 };
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_free);
+
+/*
+ * Set unique cookie for vector.
+ *
+ * Context: Must be called after vfio_pci_ims_init()
+ */
+int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
+ union msi_instance_cookie *icookie)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+ int ret = 0;
+
+ mutex_lock(&ims->ctx_mutex);
+ ctx = xa_load(&ims->ctx, vector);
+ if (ctx) {
+ ctx->icookie = *icookie;
+ goto out_unlock;
+ }
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
+ if (!ctx) {
+ ret = -ENOMEM;
+ goto out_unlock;
+ }
+
+ ctx->icookie = *icookie;
+ ret = xa_insert(&ims->ctx, vector, ctx, GFP_KERNEL_ACCOUNT);
+ if (ret) {
+ kfree(ctx);
+ goto out_unlock;
+ }
+
+ ret = 0;
+
+out_unlock:
+ mutex_unlock(&ims->ctx_mutex);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_set_cookie);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Intel Corporation");
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 2c137ea94a3e..aa54239bff4d 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -12,6 +12,7 @@
#include <linux/iommu.h>
#include <linux/mm.h>
#include <linux/workqueue.h>
+#include <linux/pci.h>
#include <linux/poll.h>
#include <uapi/linux/vfio.h>
#include <linux/iova_bitmap.h>
@@ -33,6 +34,24 @@ struct vfio_device_set {
unsigned int device_count;
};
+#if IS_ENABLED(CONFIG_VFIO_PCI_IMS)
+/*
+ * Interrupt Message Store (IMS) data
+ * @ctx: IMS interrupt context storage.
+ * @ctx_mutex: Protects the interrupt context storage.
+ * @pdev: PCI device owning the IMS domain from where
+ * interrupts are allocated.
+ * @default_cookie: Default cookie used for IMS interrupts without unique
+ * cookie.
+ */
+struct vfio_pci_ims {
+ struct xarray ctx;
+ struct mutex ctx_mutex;
+ struct pci_dev *pdev;
+ union msi_instance_cookie default_cookie;
+};
+#endif
+
struct vfio_device {
struct device *dev;
const struct vfio_device_ops *ops;
@@ -62,6 +81,9 @@ struct vfio_device {
struct iommufd_device *iommufd_device;
bool iommufd_attached;
#endif
+#if IS_ENABLED(CONFIG_VFIO_PCI_IMS)
+ struct vfio_pci_ims ims;
+#endif
};
/**
@@ -302,4 +324,42 @@ int vfio_virqfd_enable(void *opaque, int (*handler)(void *, void *),
struct virqfd **pvirqfd, int fd);
void vfio_virqfd_disable(struct virqfd **pvirqfd);
+/*
+ * Interrupt Message Store (IMS)
+ */
+#if IS_ENABLED(CONFIG_VFIO_PCI_IMS)
+int vfio_pci_set_ims_trigger(struct vfio_device *vdev, unsigned int index,
+ unsigned int start, unsigned int count, u32 flags,
+ void *data);
+void vfio_pci_ims_init(struct vfio_device *vdev, struct pci_dev *pdev,
+ union msi_instance_cookie *default_cookie);
+void vfio_pci_ims_free(struct vfio_device *vdev);
+int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
+ union msi_instance_cookie *icookie);
+#else
+static inline int vfio_pci_set_ims_trigger(struct vfio_device *vdev,
+ unsigned int index,
+ unsigned int start,
+ unsigned int count, u32 flags,
+ void *data)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline void vfio_pci_ims_init(struct vfio_device *vdev,
+ struct pci_dev *pdev,
+ union msi_instance_cookie *default_cookie)
+{}
+
+static inline void vfio_pci_ims_free(struct vfio_device *vdev) {}
+
+static inline int vfio_pci_ims_set_cookie(struct vfio_device *vdev,
+ unsigned int vector,
+ union msi_instance_cookie *icookie)
+{
+ return -EOPNOTSUPP;
+}
+
+#endif /* CONFIG_VFIO_PCI_IMS */
+
#endif /* VFIO_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-24 16:15 [RFC PATCH 0/3] vfio/ims: Back guest interrupts from Interrupt Message Store (IMS) Reinette Chatre
2023-08-24 16:15 ` [RFC PATCH 1/3] vfio/pci: Introduce library allocating " Reinette Chatre
@ 2023-08-24 16:15 ` Reinette Chatre
2023-08-24 16:33 ` Jason Gunthorpe
2023-08-24 16:15 ` [RFC PATCH 3/3] vfio/ims: Add helper that returns IMS index Reinette Chatre
2 siblings, 1 reply; 10+ messages in thread
From: Reinette Chatre @ 2023-08-24 16:15 UTC (permalink / raw)
To: jgg, yishaih, shameerali.kolothum.thodi, kevin.tian,
alex.williamson
Cc: kvm, dave.jiang, jing2.liu, ashok.raj, fenghua.yu, tom.zanussi,
reinette.chatre, linux-kernel
Access from a guest to a virtual device may be either 'direct-path',
where the guest interacts directly with the underlying hardware,
or 'intercepted path' where the virtual device emulates operations.
Support emulated interrupts that can be used to handle 'intercepted
path' operations. For example, a virtual device may use 'intercepted
path' for configuration. Doing so, configuration requests intercepted
by the virtual device driver are handled within the virtual device
driver with completion signaled to the guest without interacting with
the underlying hardware.
Add vfio_pci_ims_set_emulated() and vfio_pci_ims_send_signal() to
the VFIO PCI IMS API. vfio_pci_ims_set_emulated() configures a
range of interrupts that are emulated. Any range of interrupts
can be configured as emulated as long as no IMS interrupt has
previously been allocated at that vector. The virtual device
uses vfio_pci_ims_send_signal() to trigger interrupts in the guest.
Originally-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
drivers/vfio/pci/vfio_pci_ims.c | 117 ++++++++++++++++++++++++++++----
include/linux/vfio.h | 14 ++++
2 files changed, 117 insertions(+), 14 deletions(-)
diff --git a/drivers/vfio/pci/vfio_pci_ims.c b/drivers/vfio/pci/vfio_pci_ims.c
index 0926eb921351..fe5b3484ad34 100644
--- a/drivers/vfio/pci/vfio_pci_ims.c
+++ b/drivers/vfio/pci/vfio_pci_ims.c
@@ -17,16 +17,18 @@
#include <linux/xarray.h>
/*
- * IMS interrupt context.
- * @name: Name of device associated with interrupt.
+ * Interrupt context. Used for emulated as well as IMS interrupts.
+ * @emulated: (IMS and emulated) true if context belongs to emulated interrupt.
+ * @name: (IMS and emulated) Name of device associated with interrupt.
* Provided to request_irq().
- * @trigger: eventfd associated with interrupt.
- * @producer: Interrupt's registered IRQ bypass producer.
- * @ims_id: Interrupt index associated with IMS interrupt.
- * @virq: Linux IRQ number associated with IMS interrupt.
- * @icookie: Cookie used by irqchip driver.
+ * @trigger: (IMS and emulated) eventfd associated with interrupt.
+ * @producer: (IMS only) Interrupt's registered IRQ bypass producer.
+ * @ims_id: (IMS only) Interrupt index associated with IMS interrupt.
+ * @virq: (IMS only) Linux IRQ number associated with IMS interrupt.
+ * @icookie: (IMS only) Cookie used by irqchip driver.
*/
struct vfio_pci_ims_ctx {
+ bool emulated;
char *name;
struct eventfd_ctx *trigger;
struct irq_bypass_producer producer;
@@ -35,6 +37,31 @@ struct vfio_pci_ims_ctx {
union msi_instance_cookie icookie;
};
+/*
+ * Send signal to the eventfd.
+ * @vdev: VFIO device
+ * @vector: MSI-X vector of @vdev for which interrupt will be signaled
+ *
+ * Intended for use to send signal for emulated interrupts.
+ */
+void vfio_pci_ims_send_signal(struct vfio_device *vdev, unsigned int vector)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+
+ mutex_lock(&ims->ctx_mutex);
+ ctx = xa_load(&ims->ctx, vector);
+
+ if (WARN_ON_ONCE(!ctx || !ctx->emulated || !ctx->trigger)) {
+ mutex_unlock(&ims->ctx_mutex);
+ return;
+ }
+
+ eventfd_signal(ctx->trigger, 1);
+ mutex_unlock(&ims->ctx_mutex);
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_send_signal);
+
static irqreturn_t vfio_pci_ims_irq_handler(int irq, void *arg)
{
struct eventfd_ctx *trigger = arg;
@@ -46,7 +73,8 @@ static irqreturn_t vfio_pci_ims_irq_handler(int irq, void *arg)
/*
* Free the interrupt associated with @ctx.
*
- * Free interrupt from the underlying PCI device's IMS domain.
+ * For an emulated interrupt there is nothing to do. For an IMS interrupt
+ * the interrupt is freed from the underlying PCI device's IMS domain.
*/
static void vfio_pci_ims_irq_free(struct vfio_pci_ims *ims,
struct vfio_pci_ims_ctx *ctx)
@@ -55,6 +83,9 @@ static void vfio_pci_ims_irq_free(struct vfio_pci_ims *ims,
lockdep_assert_held(&ims->ctx_mutex);
+ if (ctx->emulated)
+ return;
+
irq_map.index = ctx->ims_id;
irq_map.virq = ctx->virq;
pci_ims_free_irq(ims->pdev, irq_map);
@@ -65,7 +96,8 @@ static void vfio_pci_ims_irq_free(struct vfio_pci_ims *ims,
/*
* Allocate an interrupt for @ctx.
*
- * Allocate interrupt from the underlying PCI device's IMS domain.
+ * For an emulated interrupt there is nothing to do. For an IMS interrupt
+ * the interrupt is allocated from the underlying PCI device's IMS domain.
*/
static int vfio_pci_ims_irq_alloc(struct vfio_pci_ims *ims,
struct vfio_pci_ims_ctx *ctx)
@@ -74,6 +106,9 @@ static int vfio_pci_ims_irq_alloc(struct vfio_pci_ims *ims,
lockdep_assert_held(&ims->ctx_mutex);
+ if (ctx->emulated)
+ return -EINVAL;
+
irq_map = pci_ims_alloc_irq(ims->pdev, &ctx->icookie, NULL);
if (irq_map.index < 0)
return irq_map.index;
@@ -133,9 +168,11 @@ static int vfio_pci_ims_set_vector_signal(struct vfio_device *vdev,
ctx = xa_load(&ims->ctx, vector);
if (ctx && ctx->trigger) {
- irq_bypass_unregister_producer(&ctx->producer);
- free_irq(ctx->virq, ctx->trigger);
- vfio_pci_ims_irq_free(ims, ctx);
+ if (!ctx->emulated) {
+ irq_bypass_unregister_producer(&ctx->producer);
+ free_irq(ctx->virq, ctx->trigger);
+ vfio_pci_ims_irq_free(ims, ctx);
+ }
kfree(ctx->name);
ctx->name = NULL;
eventfd_ctx_put(ctx->trigger);
@@ -163,6 +200,9 @@ static int vfio_pci_ims_set_vector_signal(struct vfio_device *vdev,
ctx->trigger = trigger;
+ if (ctx->emulated)
+ return 0;
+
ret = vfio_pci_ims_irq_alloc(ims, ctx);
if (ret < 0)
goto out_put_eventfd_ctx;
@@ -219,8 +259,8 @@ static int vfio_pci_ims_set_block(struct vfio_device *vdev, unsigned int start,
}
/*
- * Manage Interrupt Message Store (IMS) interrupts on the host that are
- * backing guest MSI-X vectors.
+ * Manage Interrupt Message Store (IMS) or emulated interrupts on the
+ * host that are backing guest MSI-X vectors.
*
* @vdev: VFIO device
* @index: Type of guest vectors to set up. Must be
@@ -360,6 +400,10 @@ int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
mutex_lock(&ims->ctx_mutex);
ctx = xa_load(&ims->ctx, vector);
if (ctx) {
+ if (WARN_ON_ONCE(ctx->emulated)) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
ctx->icookie = *icookie;
goto out_unlock;
}
@@ -385,5 +429,50 @@ int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
}
EXPORT_SYMBOL_GPL(vfio_pci_ims_set_cookie);
+/*
+ * Set range of interrupts that will be emulated instead of backed by IMS.
+ *
+ * Return: error code on failure (-EBUSY if the vector is not available,
+ * -ENOMEM on allocation failure), 0 on success
+ */
+int vfio_pci_ims_set_emulated(struct vfio_device *vdev, unsigned int start,
+ unsigned int count)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+ unsigned long i, j;
+ int ret = 0;
+
+ mutex_lock(&ims->ctx_mutex);
+
+ for (i = start; i < start + count; i++) {
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
+ if (!ctx) {
+ ret = -ENOMEM;
+ goto out_err;
+ }
+ ctx->emulated = true;
+ ret = xa_insert(&ims->ctx, i, ctx, GFP_KERNEL_ACCOUNT);
+ if (ret) {
+ kfree(ctx);
+ goto out_err;
+ }
+ }
+
+ mutex_unlock(&ims->ctx_mutex);
+ return 0;
+
+out_err:
+ for (j = start; j < i; j++) {
+ ctx = xa_load(&ims->ctx, j);
+ xa_erase(&ims->ctx, j);
+ kfree(ctx);
+ }
+ mutex_unlock(&ims->ctx_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_set_emulated);
+
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Intel Corporation");
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index aa54239bff4d..906220002ff4 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -334,6 +334,9 @@ int vfio_pci_set_ims_trigger(struct vfio_device *vdev, unsigned int index,
void vfio_pci_ims_init(struct vfio_device *vdev, struct pci_dev *pdev,
union msi_instance_cookie *default_cookie);
void vfio_pci_ims_free(struct vfio_device *vdev);
+int vfio_pci_ims_set_emulated(struct vfio_device *vdev, unsigned int start,
+ unsigned int count);
+void vfio_pci_ims_send_signal(struct vfio_device *vdev, unsigned int vector);
int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
union msi_instance_cookie *icookie);
#else
@@ -353,6 +356,17 @@ static inline void vfio_pci_ims_init(struct vfio_device *vdev,
static inline void vfio_pci_ims_free(struct vfio_device *vdev) {}
+static inline int vfio_pci_ims_set_emulated(struct vfio_device *vdev,
+ unsigned int start,
+ unsigned int count)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline void vfio_pci_ims_send_signal(struct vfio_device *vdev,
+ unsigned int vector)
+{}
+
static inline int vfio_pci_ims_set_cookie(struct vfio_device *vdev,
unsigned int vector,
union msi_instance_cookie *icookie)
--
2.34.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-24 16:15 ` [RFC PATCH 2/3] vfio/ims: Support emulated interrupts Reinette Chatre
@ 2023-08-24 16:33 ` Jason Gunthorpe
2023-08-24 17:19 ` Reinette Chatre
0 siblings, 1 reply; 10+ messages in thread
From: Jason Gunthorpe @ 2023-08-24 16:33 UTC (permalink / raw)
To: Reinette Chatre
Cc: yishaih, shameerali.kolothum.thodi, kevin.tian, alex.williamson,
kvm, dave.jiang, jing2.liu, ashok.raj, fenghua.yu, tom.zanussi,
linux-kernel
On Thu, Aug 24, 2023 at 09:15:21AM -0700, Reinette Chatre wrote:
> Access from a guest to a virtual device may be either 'direct-path',
> where the guest interacts directly with the underlying hardware,
> or 'intercepted path' where the virtual device emulates operations.
>
> Support emulated interrupts that can be used to handle 'intercepted
> path' operations. For example, a virtual device may use 'intercepted
> path' for configuration. Doing so, configuration requests intercepted
> by the virtual device driver are handled within the virtual device
> driver with completion signaled to the guest without interacting with
> the underlying hardware.
Why does this have anything to do with IMS? I thought the point here
was that IMS was some back end to the MSI-X emulation - should a
purely emulated interrupt logically be part of the MSI code, not IMS?
Jason
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-24 16:33 ` Jason Gunthorpe
@ 2023-08-24 17:19 ` Reinette Chatre
2023-08-25 3:05 ` Tian, Kevin
0 siblings, 1 reply; 10+ messages in thread
From: Reinette Chatre @ 2023-08-24 17:19 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: yishaih, shameerali.kolothum.thodi, kevin.tian, alex.williamson,
kvm, dave.jiang, jing2.liu, ashok.raj, fenghua.yu, tom.zanussi,
linux-kernel
Hi Jason,
On 8/24/2023 9:33 AM, Jason Gunthorpe wrote:
> On Thu, Aug 24, 2023 at 09:15:21AM -0700, Reinette Chatre wrote:
>> Access from a guest to a virtual device may be either 'direct-path',
>> where the guest interacts directly with the underlying hardware,
>> or 'intercepted path' where the virtual device emulates operations.
>>
>> Support emulated interrupts that can be used to handle 'intercepted
>> path' operations. For example, a virtual device may use 'intercepted
>> path' for configuration. Doing so, configuration requests intercepted
>> by the virtual device driver are handled within the virtual device
>> driver with completion signaled to the guest without interacting with
>> the underlying hardware.
>
> Why does this have anything to do with IMS? I thought the point here
> was that IMS was some back end to the MSI-X emulation - should a
> purely emulated interrupt logically be part of the MSI code, not IMS?
You are correct, an emulated interrupt is not unique to IMS.
The target usage of this library is by pure(?) VFIO devices (struct
vfio_device). These are virtual devices that are composed by separate
VFIO drivers. For example, a single resource of an accelerator device
can be composed into a stand-alone virtual device for use by a guest.
Through its API and implementation the current VFIO MSI
code expects to work with actual PCI devices (struct
vfio_pci_core_device). With the target usage not being an
actual PCI device the VFIO MSI code was not found to be a good
fit and thus this implementation does not build on current MSI
support.
Reinette
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-24 17:19 ` Reinette Chatre
@ 2023-08-25 3:05 ` Tian, Kevin
2023-08-25 16:55 ` Reinette Chatre
0 siblings, 1 reply; 10+ messages in thread
From: Tian, Kevin @ 2023-08-25 3:05 UTC (permalink / raw)
To: Chatre, Reinette, Jason Gunthorpe
Cc: yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com,
alex.williamson@redhat.com, kvm@vger.kernel.org, Jiang, Dave,
Liu, Jing2, Raj, Ashok, Yu, Fenghua, tom.zanussi@linux.intel.com,
linux-kernel@vger.kernel.org
> From: Chatre, Reinette <reinette.chatre@intel.com>
> Sent: Friday, August 25, 2023 1:19 AM
>
> Hi Jason,
>
> On 8/24/2023 9:33 AM, Jason Gunthorpe wrote:
> > On Thu, Aug 24, 2023 at 09:15:21AM -0700, Reinette Chatre wrote:
> >> Access from a guest to a virtual device may be either 'direct-path',
> >> where the guest interacts directly with the underlying hardware,
> >> or 'intercepted path' where the virtual device emulates operations.
> >>
> >> Support emulated interrupts that can be used to handle 'intercepted
> >> path' operations. For example, a virtual device may use 'intercepted
> >> path' for configuration. Doing so, configuration requests intercepted
> >> by the virtual device driver are handled within the virtual device
> >> driver with completion signaled to the guest without interacting with
> >> the underlying hardware.
> >
> > Why does this have anything to do with IMS? I thought the point here
> > was that IMS was some back end to the MSI-X emulation - should a
> > purely emulated interrupt logically be part of the MSI code, not IMS?
>
> You are correct, an emulated interrupt is not unique to IMS.
>
> The target usage of this library is by pure(?) VFIO devices (struct
> vfio_device). These are virtual devices that are composed by separate
> VFIO drivers. For example, a single resource of an accelerator device
> can be composed into a stand-alone virtual device for use by a guest.
>
> Through its API and implementation the current VFIO MSI
> code expects to work with actual PCI devices (struct
> vfio_pci_core_device). With the target usage not being an
> actual PCI device the VFIO MSI code was not found to be a good
> fit and thus this implementation does not build on current MSI
> support.
>
This might be achieved by creating a structure vfio_pci_intr_ctx
included by vfio_pci_core_device and other vfio device types. Then
move vfio_pci_intr.c to operate on vfio_pci_intr_ctx instead of
vfio_pci_core_device to make MSI frontend code sharable by both
PCI devices or virtual devices (mdev or SIOV).
Then there is only one irq_ctx. Within the ctx we can abstract
backend ops, e.g. enable/disble_msi(), alloc/free_ctx(), alloc/free_irq(), etc.
to accommodate pci MSI/MSI-X, IMS, or emulation.
The unknown risk is whether a clear abstraction can be defined. If
in the end the common library contains many if-else to handle subtle
backend differences then it might not be a good choice...
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-25 3:05 ` Tian, Kevin
@ 2023-08-25 16:55 ` Reinette Chatre
2023-08-25 16:59 ` Jason Gunthorpe
0 siblings, 1 reply; 10+ messages in thread
From: Reinette Chatre @ 2023-08-25 16:55 UTC (permalink / raw)
To: Tian, Kevin, Jason Gunthorpe
Cc: yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com,
alex.williamson@redhat.com, kvm@vger.kernel.org, Jiang, Dave,
Liu, Jing2, Raj, Ashok, Yu, Fenghua, tom.zanussi@linux.intel.com,
linux-kernel@vger.kernel.org
Hi Kevin,
On 8/24/2023 8:05 PM, Tian, Kevin wrote:
>> From: Chatre, Reinette <reinette.chatre@intel.com>
>> Sent: Friday, August 25, 2023 1:19 AM
>>
>> Hi Jason,
>>
>> On 8/24/2023 9:33 AM, Jason Gunthorpe wrote:
>>> On Thu, Aug 24, 2023 at 09:15:21AM -0700, Reinette Chatre wrote:
>>>> Access from a guest to a virtual device may be either 'direct-path',
>>>> where the guest interacts directly with the underlying hardware,
>>>> or 'intercepted path' where the virtual device emulates operations.
>>>>
>>>> Support emulated interrupts that can be used to handle 'intercepted
>>>> path' operations. For example, a virtual device may use 'intercepted
>>>> path' for configuration. Doing so, configuration requests intercepted
>>>> by the virtual device driver are handled within the virtual device
>>>> driver with completion signaled to the guest without interacting with
>>>> the underlying hardware.
>>>
>>> Why does this have anything to do with IMS? I thought the point here
>>> was that IMS was some back end to the MSI-X emulation - should a
>>> purely emulated interrupt logically be part of the MSI code, not IMS?
>>
>> You are correct, an emulated interrupt is not unique to IMS.
>>
>> The target usage of this library is by pure(?) VFIO devices (struct
>> vfio_device). These are virtual devices that are composed by separate
>> VFIO drivers. For example, a single resource of an accelerator device
>> can be composed into a stand-alone virtual device for use by a guest.
>>
>> Through its API and implementation the current VFIO MSI
>> code expects to work with actual PCI devices (struct
>> vfio_pci_core_device). With the target usage not being an
>> actual PCI device the VFIO MSI code was not found to be a good
>> fit and thus this implementation does not build on current MSI
>> support.
>>
>
> This might be achieved by creating a structure vfio_pci_intr_ctx
> included by vfio_pci_core_device and other vfio device types. Then
> move vfio_pci_intr.c to operate on vfio_pci_intr_ctx instead of
> vfio_pci_core_device to make MSI frontend code sharable by both
> PCI devices or virtual devices (mdev or SIOV).
Thank you very much Kevin.
For data there is the per-device context related to interrupts as
well as the per-interrupt context. These contexts are different between
the different device types and interrupt types and if I understand
correctly both context types (per-device as well as per-interrupt)
should be made opaque within the new vfio_pci_intr.c. Additionally,
with different mechanisms (for example, the locking required) to
interact with the different device types the code is also device
(PCI or virtual) specific. Considering how both the data and
code would be opaque to this new library it looks to me as
though in some aspects this library may thus appear as a skeleton
with interrupt and device specific logic contained in its users.
It is not obvious to me where the MSI frontend and backend boundary
is. If I understand correctly majority of the code in vfio_pci_intrs.c
would move to callbacks. Potentially leaving little to be shared,
vfio_pci_set_irqs_ioctl() and vfio_msihandler() seems like
candidates.
To clarify it sounds to me as though this design is motivated by the
requirement to bring emulated interrupt support to the current
INTx/MSI/MSI-X support on PCI devices? It is not clear to me how this
feature will be used since interrupts need to be signaled from the
driver instead of the hardware. I am missing something.
> Then there is only one irq_ctx. Within the ctx we can abstract
> backend ops, e.g. enable/disble_msi(), alloc/free_ctx(), alloc/free_irq(), etc.
> to accommodate pci MSI/MSI-X, IMS, or emulation.
>
> The unknown risk is whether a clear abstraction can be defined. If
> in the end the common library contains many if-else to handle subtle
> backend differences then it might not be a good choice...
Thank you very much for your guidance. Instead of Jason's expectation that
IMS would be a backend of MSI-X this will change to IMS and MSI-X both being
a backend to a new interface. It is difficult for me to envision the end
result so I will work on an implementation based on my understanding of
your proposal that we can use for further discussion.
Reinette
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-25 16:55 ` Reinette Chatre
@ 2023-08-25 16:59 ` Jason Gunthorpe
2023-08-25 17:14 ` Reinette Chatre
0 siblings, 1 reply; 10+ messages in thread
From: Jason Gunthorpe @ 2023-08-25 16:59 UTC (permalink / raw)
To: Reinette Chatre
Cc: Tian, Kevin, yishaih@nvidia.com,
shameerali.kolothum.thodi@huawei.com, alex.williamson@redhat.com,
kvm@vger.kernel.org, Jiang, Dave, Liu, Jing2, Raj, Ashok,
Yu, Fenghua, tom.zanussi@linux.intel.com,
linux-kernel@vger.kernel.org
On Fri, Aug 25, 2023 at 09:55:52AM -0700, Reinette Chatre wrote:
> Thank you very much for your guidance. Instead of Jason's expectation that
> IMS would be a backend of MSI-X this will change to IMS and MSI-X both being
> a backend to a new interface. It is difficult for me to envision the end
> result so I will work on an implementation based on my understanding of
> your proposal that we can use for further discussion.
I think the point is that emulating MSI-X is kind of tricky and should
be common VFIO code, regardless of what path it takes.
So I would expect some library code to do this, entry points the
vfio_device can hook into its callbacks (eg config space and rw of a
page)
Then the other side would connect to the physical implementation,
sw,ims,msi-x,future
Jason
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 2/3] vfio/ims: Support emulated interrupts
2023-08-25 16:59 ` Jason Gunthorpe
@ 2023-08-25 17:14 ` Reinette Chatre
0 siblings, 0 replies; 10+ messages in thread
From: Reinette Chatre @ 2023-08-25 17:14 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Tian, Kevin, yishaih@nvidia.com,
shameerali.kolothum.thodi@huawei.com, alex.williamson@redhat.com,
kvm@vger.kernel.org, Jiang, Dave, Liu, Jing2, Raj, Ashok,
Yu, Fenghua, tom.zanussi@linux.intel.com,
linux-kernel@vger.kernel.org
On 8/25/2023 9:59 AM, Jason Gunthorpe wrote:
> On Fri, Aug 25, 2023 at 09:55:52AM -0700, Reinette Chatre wrote:
>
>> Thank you very much for your guidance. Instead of Jason's expectation that
>> IMS would be a backend of MSI-X this will change to IMS and MSI-X both being
>> a backend to a new interface. It is difficult for me to envision the end
>> result so I will work on an implementation based on my understanding of
>> your proposal that we can use for further discussion.
>
> I think the point is that emulating MSI-X is kind of tricky and should
> be common VFIO code, regardless of what path it takes.
>
> So I would expect some library code to do this, entry points the
> vfio_device can hook into its callbacks (eg config space and rw of a
> page)
>
> Then the other side would connect to the physical implementation,
> sw,ims,msi-x,future
>
Thank you very much Jason. I am thus hearing the same from you and
Kevin. I'll work on this.
Reinette
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH 3/3] vfio/ims: Add helper that returns IMS index
2023-08-24 16:15 [RFC PATCH 0/3] vfio/ims: Back guest interrupts from Interrupt Message Store (IMS) Reinette Chatre
2023-08-24 16:15 ` [RFC PATCH 1/3] vfio/pci: Introduce library allocating " Reinette Chatre
2023-08-24 16:15 ` [RFC PATCH 2/3] vfio/ims: Support emulated interrupts Reinette Chatre
@ 2023-08-24 16:15 ` Reinette Chatre
2 siblings, 0 replies; 10+ messages in thread
From: Reinette Chatre @ 2023-08-24 16:15 UTC (permalink / raw)
To: jgg, yishaih, shameerali.kolothum.thodi, kevin.tian,
alex.williamson
Cc: kvm, dave.jiang, jing2.liu, ashok.raj, fenghua.yu, tom.zanussi,
reinette.chatre, linux-kernel
A virtual device driver needs to facilitate translation between
the guest's MSI-X interrupt and the backing IMS interrupt with
which the physical device is programmed. For example, the
guest may need to obtain the IMS index from the virtual device driver
that it needs to program into descriptors submitted to the device
to ensure that the completion interrupts are generated correctly.
Introduce vfio_pci_ims_hwirq() to VFIO PCI IMS as a helper that
returns the IMS interrupt index backing a provided MSI-X
interrupt index belonging to a guest.
Originally-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
---
drivers/vfio/pci/vfio_pci_ims.c | 21 +++++++++++++++++++++
include/linux/vfio.h | 7 +++++++
2 files changed, 28 insertions(+)
diff --git a/drivers/vfio/pci/vfio_pci_ims.c b/drivers/vfio/pci/vfio_pci_ims.c
index fe5b3484ad34..27711bc7a6c2 100644
--- a/drivers/vfio/pci/vfio_pci_ims.c
+++ b/drivers/vfio/pci/vfio_pci_ims.c
@@ -37,6 +37,27 @@ struct vfio_pci_ims_ctx {
union msi_instance_cookie icookie;
};
+/*
+ * Return IMS index of IMS interrupt backing MSI-X interrupt @vector
+ */
+int vfio_pci_ims_hwirq(struct vfio_device *vdev, unsigned int vector)
+{
+ struct vfio_pci_ims *ims = &vdev->ims;
+ struct vfio_pci_ims_ctx *ctx;
+ int id;
+
+ mutex_lock(&ims->ctx_mutex);
+ ctx = xa_load(&ims->ctx, vector);
+ if (!ctx || ctx->emulated)
+ id = -EINVAL;
+ else
+ id = ctx->ims_id;
+ mutex_unlock(&ims->ctx_mutex);
+
+ return id;
+}
+EXPORT_SYMBOL_GPL(vfio_pci_ims_hwirq);
+
/*
* Send signal to the eventfd.
* @vdev: VFIO device
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index 906220002ff4..1c0e1bddd95b 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -339,6 +339,7 @@ int vfio_pci_ims_set_emulated(struct vfio_device *vdev, unsigned int start,
void vfio_pci_ims_send_signal(struct vfio_device *vdev, unsigned int vector);
int vfio_pci_ims_set_cookie(struct vfio_device *vdev, unsigned int vector,
union msi_instance_cookie *icookie);
+int vfio_pci_ims_hwirq(struct vfio_device *vdev, unsigned int vector);
#else
static inline int vfio_pci_set_ims_trigger(struct vfio_device *vdev,
unsigned int index,
@@ -374,6 +375,12 @@ static inline int vfio_pci_ims_set_cookie(struct vfio_device *vdev,
return -EOPNOTSUPP;
}
+static inline int vfio_pci_ims_hwirq(struct vfio_device *vdev,
+ unsigned int index)
+{
+ return -EOPNOTSUPP;
+}
+
#endif /* CONFIG_VFIO_PCI_IMS */
#endif /* VFIO_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 10+ messages in thread