[PATCH 0/7] irqbypass: Cleanups and a perf improvement

[PATCH 0/7] irqbypass: Cleanups and a perf improvement

[Sean Christopherson]

The two primary goals of this series are to make the irqbypass concept
easier to understand, and to address the terrible performance that can
result from using a list to track connections.

For the first goal, track the producer/consumer "tokens" as eventfd context
pointers instead of opaque "void *".  Supporting arbitrary token types was
dead infrastructure when it was added 10 years ago, and nothing has changed
since.  Taking an opaque token makes a *very* simple concept (device signals
eventfd; KVM listens to eventfd) unnecessarily difficult to understand.

Burying that simple behind a layer of obfuscation also makes the overall
code more brittle, as callers can pass in literally anything. I.e. passing
in a token that will never be paired would go unnoticed.

For the performance issue, use an xarray.  I'm definitely not wedded to an
xarray, but IMO it doesn't add meaningful complexity (even requires less
code), and pretty much Just Works.  Like tried this a while back[1], but
the implementation had undesirable behavior changes and stalled out.

To address the use case where huge numbers of VMs are being created without
_any_ possibility for irqbypass, KVM should probably add a
KVM_IRQFD_FLAG_NO_IRQBYPASS flag so that userspace can opt-out on a per-IRQ
basis.  I already proposed a KVM module param[2] to let userspace disable
IRQ bypass, but that obviously affects all IRQs in all VMs.  It might
suffice for most use cases, but I can imagine scenarios where the VMM wants
to be more selective, e.g. when it *knows* a KVM_IRQFD isn't eligible for
bypass.  And both of those require userspace changes.

Note, I want to do more aggressive cleanups of irqbypass at some point,
e.g. not reporting an error to userspace if connect() fails is *awful*
behavior for environments that want/need irqbypass to always work.  But
that's a future problem.

[1] https://lore.kernel.org/all/20230801115646.33990-1-likexu@tencent.com
[2] https://lore.kernel.org/all/20250401161804.842968-1-seanjc@google.com

Sean Christopherson (7):
  irqbypass: Drop pointless and misleading THIS_MODULE get/put
  irqbypass: Drop superfluous might_sleep() annotations
  irqbypass: Take ownership of producer/consumer token tracking
  irqbypass: Explicitly track producer and consumer bindings
  irqbypass: Use paired consumer/producer to disconnect during
    unregister
  irqbypass: Use guard(mutex) in lieu of manual lock+unlock
  irqbypass: Use xarray to track producers and consumers

 drivers/vfio/pci/vfio_pci_intrs.c |   5 +-
 drivers/vhost/vdpa.c              |   4 +-
 include/linux/irqbypass.h         |  38 +++---
 virt/kvm/eventfd.c                |   3 +-
 virt/lib/irqbypass.c              | 185 ++++++++++--------------------
 5 files changed, 88 insertions(+), 147 deletions(-)


base-commit: 782f9feaa9517caf33186dcdd6b50a8f770ed29b
-- 
2.49.0.504.g3bcea36a83-goog

[PATCH 1/7] irqbypass: Drop pointless and misleading THIS_MODULE get/put

[Sean Christopherson]

Drop irqbypass.ko's superfluous and misleading get/put calls on
THIS_MODULE.  A module taking a reference to itself is useless; no amount
of checks will prevent doom and destruction if the caller hasn't already
guaranteed the liveliness of the module (this goes for any module).  E.g.
if try_module_get() fails because irqbypass.ko is being unloaded, then the
kernel has already hit a use-after-free by virtue of executing code whose
lifecycle is tied to irqbypass.ko.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/lib/irqbypass.c | 20 --------------------
 1 file changed, 20 deletions(-)

diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 28fda42e471b..080c706f3b01 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -92,9 +92,6 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer)
 
 	might_sleep();
 
-	if (!try_module_get(THIS_MODULE))
-		return -ENODEV;
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
@@ -120,7 +117,6 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer)
 	return 0;
 out_err:
 	mutex_unlock(&lock);
-	module_put(THIS_MODULE);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_producer);
@@ -142,9 +138,6 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 
 	might_sleep();
 
-	if (!try_module_get(THIS_MODULE))
-		return; /* nothing in the list anyway */
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
@@ -159,13 +152,10 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 		}
 
 		list_del(&producer->node);
-		module_put(THIS_MODULE);
 		break;
 	}
 
 	mutex_unlock(&lock);
-
-	module_put(THIS_MODULE);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
 
@@ -188,9 +178,6 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer)
 
 	might_sleep();
 
-	if (!try_module_get(THIS_MODULE))
-		return -ENODEV;
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
@@ -216,7 +203,6 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer)
 	return 0;
 out_err:
 	mutex_unlock(&lock);
-	module_put(THIS_MODULE);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_consumer);
@@ -238,9 +224,6 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 
 	might_sleep();
 
-	if (!try_module_get(THIS_MODULE))
-		return; /* nothing in the list anyway */
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
@@ -255,12 +238,9 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 		}
 
 		list_del(&consumer->node);
-		module_put(THIS_MODULE);
 		break;
 	}
 
 	mutex_unlock(&lock);
-
-	module_put(THIS_MODULE);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer);
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 1/7] irqbypass: Drop pointless and misleading THIS_MODULE get/put

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> Drop irqbypass.ko's superfluous and misleading get/put calls on
> THIS_MODULE.  A module taking a reference to itself is useless; no amount
> of checks will prevent doom and destruction if the caller hasn't already
> guaranteed the liveliness of the module (this goes for any module).  E.g.
> if try_module_get() fails because irqbypass.ko is being unloaded, then the
> kernel has already hit a use-after-free by virtue of executing code whose
> lifecycle is tied to irqbypass.ko.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

[PATCH 2/7] irqbypass: Drop superfluous might_sleep() annotations

[Sean Christopherson]

Drop superfluous might_sleep() annotations from irqbypass, mutex_lock()
provides all of the necessary tracking.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/lib/irqbypass.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 080c706f3b01..28a4d933569a 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -90,8 +90,6 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer)
 	if (!producer->token)
 		return -EINVAL;
 
-	might_sleep();
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
@@ -136,8 +134,6 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 	if (!producer->token)
 		return;
 
-	might_sleep();
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
@@ -176,8 +172,6 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer)
 	    !consumer->add_producer || !consumer->del_producer)
 		return -EINVAL;
 
-	might_sleep();
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
@@ -222,8 +216,6 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 	if (!consumer->token)
 		return;
 
-	might_sleep();
-
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 2/7] irqbypass: Drop superfluous might_sleep() annotations

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> Drop superfluous might_sleep() annotations from irqbypass, mutex_lock()
> provides all of the necessary tracking.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

[PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Sean Christopherson]

Move ownership of IRQ bypass token tracking into irqbypass.ko, and
explicitly require callers to pass an eventfd_ctx structure instead of a
completely opaque token.  Relying on producers and consumers to set the
token appropriately is error prone, and hiding the fact that the token must
be an eventfd_ctx pointer (for all intents and purposes) unnecessarily
obfuscates the code and makes it more brittle.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 drivers/vfio/pci/vfio_pci_intrs.c |  5 +----
 drivers/vhost/vdpa.c              |  4 +---
 include/linux/irqbypass.h         | 31 +++++++++++++++++--------------
 virt/kvm/eventfd.c                |  3 +--
 virt/lib/irqbypass.c              | 30 +++++++++++++++++++++---------
 5 files changed, 41 insertions(+), 32 deletions(-)

diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c
index 8382c5834335..c852643d8359 100644
--- a/drivers/vfio/pci/vfio_pci_intrs.c
+++ b/drivers/vfio/pci/vfio_pci_intrs.c
@@ -505,15 +505,12 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 	if (ret)
 		goto out_put_eventfd_ctx;
 
-	ctx->producer.token = trigger;
 	ctx->producer.irq = irq;
-	ret = irq_bypass_register_producer(&ctx->producer);
+	ret = irq_bypass_register_producer(&ctx->producer, trigger);
 	if (unlikely(ret)) {
 		dev_info(&pdev->dev,
 		"irq bypass producer (token %p) registration fails: %d\n",
 		ctx->producer.token, ret);
-
-		ctx->producer.token = NULL;
 	}
 	ctx->trigger = trigger;
 
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 5a49b5a6d496..2749290f892d 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -213,7 +213,7 @@ static void vhost_vdpa_setup_vq_irq(struct vhost_vdpa *v, u16 qid)
 		return;
 
 	vq->call_ctx.producer.irq = irq;
-	ret = irq_bypass_register_producer(&vq->call_ctx.producer);
+	ret = irq_bypass_register_producer(&vq->call_ctx.producer, vq->call_ctx.ctx);
 	if (unlikely(ret))
 		dev_info(&v->dev, "vq %u, irq bypass producer (token %p) registration fails, ret =  %d\n",
 			 qid, vq->call_ctx.producer.token, ret);
@@ -712,7 +712,6 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 			if (ops->get_status(vdpa) &
 			    VIRTIO_CONFIG_S_DRIVER_OK)
 				vhost_vdpa_unsetup_vq_irq(v, idx);
-			vq->call_ctx.producer.token = NULL;
 		}
 		break;
 	}
@@ -753,7 +752,6 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 			cb.callback = vhost_vdpa_virtqueue_cb;
 			cb.private = vq;
 			cb.trigger = vq->call_ctx.ctx;
-			vq->call_ctx.producer.token = vq->call_ctx.ctx;
 			if (ops->get_status(vdpa) &
 			    VIRTIO_CONFIG_S_DRIVER_OK)
 				vhost_vdpa_setup_vq_irq(v, idx);
diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h
index 9bdb2a781841..379725b9a003 100644
--- a/include/linux/irqbypass.h
+++ b/include/linux/irqbypass.h
@@ -10,6 +10,7 @@
 
 #include <linux/list.h>
 
+struct eventfd_ctx;
 struct irq_bypass_consumer;
 
 /*
@@ -18,20 +19,20 @@ struct irq_bypass_consumer;
  * The IRQ bypass manager is a simple set of lists and callbacks that allows
  * IRQ producers (ex. physical interrupt sources) to be matched to IRQ
  * consumers (ex. virtualization hardware that allows IRQ bypass or offload)
- * via a shared token (ex. eventfd_ctx).  Producers and consumers register
- * independently.  When a token match is found, the optional @stop callback
- * will be called for each participant.  The pair will then be connected via
- * the @add_* callbacks, and finally the optional @start callback will allow
- * any final coordination.  When either participant is unregistered, the
- * process is repeated using the @del_* callbacks in place of the @add_*
- * callbacks.  Match tokens must be unique per producer/consumer, 1:N pairings
- * are not supported.
+ * via a shared eventfd_ctx).  Producers and consumers register independently.
+ * When a producer and consumer are paired, i.e. a token match is found, the
+ * optional @stop callback will be called for each participant.  The pair will
+ * then be connected via the @add_* callbacks, and finally the optional @start
+ * callback will allow any final coordination.  When either participant is
+ * unregistered, the process is repeated using the @del_* callbacks in place of
+ * the @add_* callbacks.  Match tokens must be unique per producer/consumer,
+ * 1:N pairings are not supported.
  */
 
 /**
  * struct irq_bypass_producer - IRQ bypass producer definition
  * @node: IRQ bypass manager private list management
- * @token: opaque token to match between producer and consumer (non-NULL)
+ * @token: IRQ bypass manage private token to match producers and consumers
  * @irq: Linux IRQ number for the producer device
  * @add_consumer: Connect the IRQ producer to an IRQ consumer (optional)
  * @del_consumer: Disconnect the IRQ producer from an IRQ consumer (optional)
@@ -57,7 +58,7 @@ struct irq_bypass_producer {
 /**
  * struct irq_bypass_consumer - IRQ bypass consumer definition
  * @node: IRQ bypass manager private list management
- * @token: opaque token to match between producer and consumer (non-NULL)
+ * @token: IRQ bypass manage private token to match producers and consumers
  * @add_producer: Connect the IRQ consumer to an IRQ producer
  * @del_producer: Disconnect the IRQ consumer from an IRQ producer
  * @stop: Perform any quiesce operations necessary prior to add/del (optional)
@@ -79,9 +80,11 @@ struct irq_bypass_consumer {
 	void (*start)(struct irq_bypass_consumer *);
 };
 
-int irq_bypass_register_producer(struct irq_bypass_producer *);
-void irq_bypass_unregister_producer(struct irq_bypass_producer *);
-int irq_bypass_register_consumer(struct irq_bypass_consumer *);
-void irq_bypass_unregister_consumer(struct irq_bypass_consumer *);
+int irq_bypass_register_producer(struct irq_bypass_producer *producer,
+				 struct eventfd_ctx *eventfd);
+void irq_bypass_unregister_producer(struct irq_bypass_producer *producer);
+int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
+				 struct eventfd_ctx *eventfd);
+void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer);
 
 #endif /* IRQBYPASS_H */
diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 249ba5b72e9b..afdbac0d0b9f 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -426,12 +426,11 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
 
 #ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
 	if (kvm_arch_has_irq_bypass()) {
-		irqfd->consumer.token = (void *)irqfd->eventfd;
 		irqfd->consumer.add_producer = kvm_arch_irq_bypass_add_producer;
 		irqfd->consumer.del_producer = kvm_arch_irq_bypass_del_producer;
 		irqfd->consumer.stop = kvm_arch_irq_bypass_stop;
 		irqfd->consumer.start = kvm_arch_irq_bypass_start;
-		ret = irq_bypass_register_consumer(&irqfd->consumer);
+		ret = irq_bypass_register_consumer(&irqfd->consumer, irqfd->eventfd);
 		if (ret)
 			pr_info("irq bypass consumer (token %p) registration fails: %d\n",
 				irqfd->consumer.token, ret);
diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 28a4d933569a..98bf76d03078 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -77,30 +77,32 @@ static void __disconnect(struct irq_bypass_producer *prod,
 /**
  * irq_bypass_register_producer - register IRQ bypass producer
  * @producer: pointer to producer structure
+ * @eventfd: pointer to the eventfd context associated with the producer
  *
  * Add the provided IRQ producer to the list of producers and connect
  * with any matching token found on the IRQ consumers list.
  */
-int irq_bypass_register_producer(struct irq_bypass_producer *producer)
+int irq_bypass_register_producer(struct irq_bypass_producer *producer,
+				 struct eventfd_ctx *eventfd)
 {
 	struct irq_bypass_producer *tmp;
 	struct irq_bypass_consumer *consumer;
 	int ret;
 
-	if (!producer->token)
+	if (WARN_ON_ONCE(producer->token))
 		return -EINVAL;
 
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
-		if (tmp->token == producer->token) {
+		if (tmp->token == eventfd) {
 			ret = -EBUSY;
 			goto out_err;
 		}
 	}
 
 	list_for_each_entry(consumer, &consumers, node) {
-		if (consumer->token == producer->token) {
+		if (consumer->token == eventfd) {
 			ret = __connect(producer, consumer);
 			if (ret)
 				goto out_err;
@@ -108,6 +110,7 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer)
 		}
 	}
 
+	producer->token = eventfd;
 	list_add(&producer->node, &producers);
 
 	mutex_unlock(&lock);
@@ -147,10 +150,12 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 			}
 		}
 
+		producer->token = NULL;
 		list_del(&producer->node);
 		break;
 	}
 
+	WARN_ON_ONCE(producer->token);
 	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
@@ -158,31 +163,35 @@ EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
 /**
  * irq_bypass_register_consumer - register IRQ bypass consumer
  * @consumer: pointer to consumer structure
+ * @eventfd: pointer to the eventfd context associated with the consumer
  *
  * Add the provided IRQ consumer to the list of consumers and connect
  * with any matching token found on the IRQ producer list.
  */
-int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer)
+int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
+				 struct eventfd_ctx *eventfd)
 {
 	struct irq_bypass_consumer *tmp;
 	struct irq_bypass_producer *producer;
 	int ret;
 
-	if (!consumer->token ||
-	    !consumer->add_producer || !consumer->del_producer)
+	if (WARN_ON_ONCE(consumer->token))
+		return -EINVAL;
+
+	if (!consumer->add_producer || !consumer->del_producer)
 		return -EINVAL;
 
 	mutex_lock(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
-		if (tmp->token == consumer->token || tmp == consumer) {
+		if (tmp->token == eventfd || tmp == consumer) {
 			ret = -EBUSY;
 			goto out_err;
 		}
 	}
 
 	list_for_each_entry(producer, &producers, node) {
-		if (producer->token == consumer->token) {
+		if (producer->token == eventfd) {
 			ret = __connect(producer, consumer);
 			if (ret)
 				goto out_err;
@@ -190,6 +199,7 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer)
 		}
 	}
 
+	consumer->token = eventfd;
 	list_add(&consumer->node, &consumers);
 
 	mutex_unlock(&lock);
@@ -229,10 +239,12 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 			}
 		}
 
+		consumer->token = NULL;
 		list_del(&consumer->node);
 		break;
 	}
 
+	WARN_ON_ONCE(consumer->token);
 	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer);
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> @@ -505,15 +505,12 @@ static int vfio_msi_set_vector_signal(struct
> vfio_pci_core_device *vdev,
>  	if (ret)
>  		goto out_put_eventfd_ctx;
> 
> -	ctx->producer.token = trigger;
>  	ctx->producer.irq = irq;
> -	ret = irq_bypass_register_producer(&ctx->producer);
> +	ret = irq_bypass_register_producer(&ctx->producer, trigger);
>  	if (unlikely(ret)) {
>  		dev_info(&pdev->dev,
>  		"irq bypass producer (token %p) registration fails: %d\n",
>  		ctx->producer.token, ret);

Use 'trigger' as ctx->producer.token is NULL if registration fails. 

> @@ -18,20 +19,20 @@ struct irq_bypass_consumer;
>   * The IRQ bypass manager is a simple set of lists and callbacks that allows
>   * IRQ producers (ex. physical interrupt sources) to be matched to IRQ
>   * consumers (ex. virtualization hardware that allows IRQ bypass or offload)
> - * via a shared token (ex. eventfd_ctx).  Producers and consumers register
> - * independently.  When a token match is found, the optional @stop
> callback
> - * will be called for each participant.  The pair will then be connected via
> - * the @add_* callbacks, and finally the optional @start callback will allow
> - * any final coordination.  When either participant is unregistered, the
> - * process is repeated using the @del_* callbacks in place of the @add_*
> - * callbacks.  Match tokens must be unique per producer/consumer, 1:N
> pairings
> - * are not supported.
> + * via a shared eventfd_ctx).  Producers and consumers register
> independently.

s/eventfd_ctx)/eventfd_ctx/

> +int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
> +				 struct eventfd_ctx *eventfd)
>  {
>  	struct irq_bypass_consumer *tmp;
>  	struct irq_bypass_producer *producer;
>  	int ret;
> 
> -	if (!consumer->token ||
> -	    !consumer->add_producer || !consumer->del_producer)
> +	if (WARN_ON_ONCE(consumer->token))
> +		return -EINVAL;
> +
> +	if (!consumer->add_producer || !consumer->del_producer)
>  		return -EINVAL;
> 
>  	mutex_lock(&lock);
> 
>  	list_for_each_entry(tmp, &consumers, node) {
> -		if (tmp->token == consumer->token || tmp == consumer) {
> +		if (tmp->token == eventfd || tmp == consumer) {
>  			ret = -EBUSY;
>  			goto out_err;
>  		}
>  	}

the 2nd check 'tmp == consumer' is redundant. If they are equal 
consumer->token is not NULL then the earlier WARN_ON will be
triggered already.

otherwise,

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Sean Christopherson]

On Thu, Apr 10, 2025, Kevin Tian wrote:
> > From: Sean Christopherson <seanjc@google.com>
> > +int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
> > +				 struct eventfd_ctx *eventfd)
> >  {
> >  	struct irq_bypass_consumer *tmp;
> >  	struct irq_bypass_producer *producer;
> >  	int ret;
> > 
> > -	if (!consumer->token ||
> > -	    !consumer->add_producer || !consumer->del_producer)
> > +	if (WARN_ON_ONCE(consumer->token))
> > +		return -EINVAL;
> > +
> > +	if (!consumer->add_producer || !consumer->del_producer)
> >  		return -EINVAL;
> > 
> >  	mutex_lock(&lock);
> > 
> >  	list_for_each_entry(tmp, &consumers, node) {
> > -		if (tmp->token == consumer->token || tmp == consumer) {
> > +		if (tmp->token == eventfd || tmp == consumer) {
> >  			ret = -EBUSY;
> >  			goto out_err;
> >  		}
> >  	}
> 
> the 2nd check 'tmp == consumer' is redundant. If they are equal 
> consumer->token is not NULL then the earlier WARN_ON will be
> triggered already.

Oh, nice.  Good catch!  That check subtly gets dropped on the conversion to
xarray, so it definitely makes sense to remove it in this patch.

Re: [PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Alex Williamson]

On Fri,  4 Apr 2025 14:14:45 -0700
Sean Christopherson <seanjc@google.com> wrote:
> diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h
> index 9bdb2a781841..379725b9a003 100644
> --- a/include/linux/irqbypass.h
> +++ b/include/linux/irqbypass.h
> @@ -10,6 +10,7 @@
>  
>  #include <linux/list.h>
>  
> +struct eventfd_ctx;
>  struct irq_bypass_consumer;
>  
>  /*
> @@ -18,20 +19,20 @@ struct irq_bypass_consumer;
>   * The IRQ bypass manager is a simple set of lists and callbacks that allows
>   * IRQ producers (ex. physical interrupt sources) to be matched to IRQ
>   * consumers (ex. virtualization hardware that allows IRQ bypass or offload)
> - * via a shared token (ex. eventfd_ctx).  Producers and consumers register
> - * independently.  When a token match is found, the optional @stop callback
> - * will be called for each participant.  The pair will then be connected via
> - * the @add_* callbacks, and finally the optional @start callback will allow
> - * any final coordination.  When either participant is unregistered, the
> - * process is repeated using the @del_* callbacks in place of the @add_*
> - * callbacks.  Match tokens must be unique per producer/consumer, 1:N pairings
> - * are not supported.
> + * via a shared eventfd_ctx).  Producers and consumers register independently.
> + * When a producer and consumer are paired, i.e. a token match is found, the
> + * optional @stop callback will be called for each participant.  The pair will
> + * then be connected via the @add_* callbacks, and finally the optional @start
> + * callback will allow any final coordination.  When either participant is
> + * unregistered, the process is repeated using the @del_* callbacks in place of
> + * the @add_* callbacks.  Match tokens must be unique per producer/consumer,
> + * 1:N pairings are not supported.
>   */
>  
>  /**
>   * struct irq_bypass_producer - IRQ bypass producer definition
>   * @node: IRQ bypass manager private list management
> - * @token: opaque token to match between producer and consumer (non-NULL)
> + * @token: IRQ bypass manage private token to match producers and consumers

The "token" terminology seems a little out of place after all is said
and done in this series.  Should it just be an "index" in anticipation
of the usage with xarray and changed to an unsigned long?  Or at least
s/token/eventfd/ and changed to an eventfd_ctx pointer?  Thanks,

Alex

Re: [PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Sean Christopherson]

On Thu, Apr 10, 2025, Alex Williamson wrote:
> On Fri,  4 Apr 2025 14:14:45 -0700
> Sean Christopherson <seanjc@google.com> wrote:
> > diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h
> > index 9bdb2a781841..379725b9a003 100644
> > --- a/include/linux/irqbypass.h
> > +++ b/include/linux/irqbypass.h
> > @@ -10,6 +10,7 @@
> >  
> >  #include <linux/list.h>
> >  
> > +struct eventfd_ctx;
> >  struct irq_bypass_consumer;
> >  
> >  /*
> > @@ -18,20 +19,20 @@ struct irq_bypass_consumer;
> >   * The IRQ bypass manager is a simple set of lists and callbacks that allows
> >   * IRQ producers (ex. physical interrupt sources) to be matched to IRQ
> >   * consumers (ex. virtualization hardware that allows IRQ bypass or offload)
> > - * via a shared token (ex. eventfd_ctx).  Producers and consumers register
> > - * independently.  When a token match is found, the optional @stop callback
> > - * will be called for each participant.  The pair will then be connected via
> > - * the @add_* callbacks, and finally the optional @start callback will allow
> > - * any final coordination.  When either participant is unregistered, the
> > - * process is repeated using the @del_* callbacks in place of the @add_*
> > - * callbacks.  Match tokens must be unique per producer/consumer, 1:N pairings
> > - * are not supported.
> > + * via a shared eventfd_ctx).  Producers and consumers register independently.
> > + * When a producer and consumer are paired, i.e. a token match is found, the
> > + * optional @stop callback will be called for each participant.  The pair will
> > + * then be connected via the @add_* callbacks, and finally the optional @start
> > + * callback will allow any final coordination.  When either participant is
> > + * unregistered, the process is repeated using the @del_* callbacks in place of
> > + * the @add_* callbacks.  Match tokens must be unique per producer/consumer,
> > + * 1:N pairings are not supported.
> >   */
> >  
> >  /**
> >   * struct irq_bypass_producer - IRQ bypass producer definition
> >   * @node: IRQ bypass manager private list management
> > - * @token: opaque token to match between producer and consumer (non-NULL)
> > + * @token: IRQ bypass manage private token to match producers and consumers
> 
> The "token" terminology seems a little out of place after all is said
> and done in this series.  

Ugh, yeah, good point.  I don't know why I left it as "token".

> Should it just be an "index" in anticipation of the usage with xarray and
> changed to an unsigned long?  Or at least s/token/eventfd/ and changed to an
> eventfd_ctx pointer?

My strong vote is for "struct eventfd_ctx *eventfd;"

Re: [PATCH 3/7] irqbypass: Take ownership of producer/consumer token tracking

[Alex Williamson]

On Thu, 10 Apr 2025 15:04:35 -0700
Sean Christopherson <seanjc@google.com> wrote:
> On Thu, Apr 10, 2025, Alex Williamson wrote:
> > The "token" terminology seems a little out of place after all is said
> > and done in this series.    
> 
> Ugh, yeah, good point.  I don't know why I left it as "token".
> 
> > Should it just be an "index" in anticipation of the usage with xarray and
> > changed to an unsigned long?  Or at least s/token/eventfd/ and changed to an
> > eventfd_ctx pointer?  
> 
> My strong vote is for "struct eventfd_ctx *eventfd;"

WFM, thanks,

Alex

[PATCH 4/7] irqbypass: Explicitly track producer and consumer bindings

[Sean Christopherson]

Explicitly track IRQ bypass producer:consumer bindings.  This will allow
making removal an O(1) operation; searching through the list to find
information that is trivially tracked (and useful for debug) is wasteful.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 include/linux/irqbypass.h | 5 +++++
 virt/lib/irqbypass.c      | 9 +++++++++
 2 files changed, 14 insertions(+)

diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h
index 379725b9a003..6d4e4882843c 100644
--- a/include/linux/irqbypass.h
+++ b/include/linux/irqbypass.h
@@ -29,6 +29,8 @@ struct irq_bypass_consumer;
  * 1:N pairings are not supported.
  */
 
+struct irq_bypass_consumer;
+
 /**
  * struct irq_bypass_producer - IRQ bypass producer definition
  * @node: IRQ bypass manager private list management
@@ -46,6 +48,7 @@ struct irq_bypass_consumer;
 struct irq_bypass_producer {
 	struct list_head node;
 	void *token;
+	struct irq_bypass_consumer *consumer;
 	int irq;
 	int (*add_consumer)(struct irq_bypass_producer *,
 			    struct irq_bypass_consumer *);
@@ -72,6 +75,8 @@ struct irq_bypass_producer {
 struct irq_bypass_consumer {
 	struct list_head node;
 	void *token;
+	struct irq_bypass_producer *producer;
+
 	int (*add_producer)(struct irq_bypass_consumer *,
 			    struct irq_bypass_producer *);
 	void (*del_producer)(struct irq_bypass_consumer *,
diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 98bf76d03078..4c912c30b7e6 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -51,6 +51,10 @@ static int __connect(struct irq_bypass_producer *prod,
 	if (prod->start)
 		prod->start(prod);
 
+	if (!ret) {
+		prod->consumer = cons;
+		cons->producer = prod;
+	}
 	return ret;
 }
 
@@ -72,6 +76,9 @@ static void __disconnect(struct irq_bypass_producer *prod,
 		cons->start(cons);
 	if (prod->start)
 		prod->start(prod);
+
+	prod->consumer = NULL;
+	cons->producer = NULL;
 }
 
 /**
@@ -145,6 +152,7 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 
 		list_for_each_entry(consumer, &consumers, node) {
 			if (consumer->token == producer->token) {
+				WARN_ON_ONCE(producer->consumer != consumer);
 				__disconnect(producer, consumer);
 				break;
 			}
@@ -234,6 +242,7 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 
 		list_for_each_entry(producer, &producers, node) {
 			if (producer->token == consumer->token) {
+				WARN_ON_ONCE(consumer->producer != producer);
 				__disconnect(producer, consumer);
 				break;
 			}
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 4/7] irqbypass: Explicitly track producer and consumer bindings

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> Explicitly track IRQ bypass producer:consumer bindings.  This will allow
> making removal an O(1) operation; searching through the list to find
> information that is trivially tracked (and useful for debug) is wasteful.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

[PATCH 5/7] irqbypass: Use paired consumer/producer to disconnect during unregister

[Sean Christopherson]

Use the paired consumer/producer information to disconnect IRQ bypass
producers/consumers in O(1) time (ignoring the cost of __disconnect()).

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/lib/irqbypass.c | 50 +++++++-------------------------------------
 1 file changed, 8 insertions(+), 42 deletions(-)

diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 4c912c30b7e6..6d68a0f71dd9 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -138,32 +138,16 @@ EXPORT_SYMBOL_GPL(irq_bypass_register_producer);
  */
 void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 {
-	struct irq_bypass_producer *tmp;
-	struct irq_bypass_consumer *consumer;
-
 	if (!producer->token)
 		return;
 
 	mutex_lock(&lock);
 
-	list_for_each_entry(tmp, &producers, node) {
-		if (tmp->token != producer->token)
-			continue;
+	if (producer->consumer)
+		__disconnect(producer, producer->consumer);
 
-		list_for_each_entry(consumer, &consumers, node) {
-			if (consumer->token == producer->token) {
-				WARN_ON_ONCE(producer->consumer != consumer);
-				__disconnect(producer, consumer);
-				break;
-			}
-		}
-
-		producer->token = NULL;
-		list_del(&producer->node);
-		break;
-	}
-
-	WARN_ON_ONCE(producer->token);
+	producer->token = NULL;
+	list_del(&producer->node);
 	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
@@ -173,8 +157,6 @@ EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
  * @consumer: pointer to consumer structure
  * @eventfd: pointer to the eventfd context associated with the consumer
  *
- * Add the provided IRQ consumer to the list of consumers and connect
- * with any matching token found on the IRQ producer list.
  */
 int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
 				 struct eventfd_ctx *eventfd)
@@ -228,32 +210,16 @@ EXPORT_SYMBOL_GPL(irq_bypass_register_consumer);
  */
 void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 {
-	struct irq_bypass_consumer *tmp;
-	struct irq_bypass_producer *producer;
-
 	if (!consumer->token)
 		return;
 
 	mutex_lock(&lock);
 
-	list_for_each_entry(tmp, &consumers, node) {
-		if (tmp != consumer)
-			continue;
+	if (consumer->producer)
+		__disconnect(consumer->producer, consumer);
 
-		list_for_each_entry(producer, &producers, node) {
-			if (producer->token == consumer->token) {
-				WARN_ON_ONCE(consumer->producer != producer);
-				__disconnect(producer, consumer);
-				break;
-			}
-		}
-
-		consumer->token = NULL;
-		list_del(&consumer->node);
-		break;
-	}
-
-	WARN_ON_ONCE(consumer->token);
+	consumer->token = NULL;
+	list_del(&consumer->node);
 	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer);
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 5/7] irqbypass: Use paired consumer/producer to disconnect during unregister

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
>
> @@ -173,8 +157,6 @@
> EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
>   * @consumer: pointer to consumer structure
>   * @eventfd: pointer to the eventfd context associated with the consumer
>   *
> - * Add the provided IRQ consumer to the list of consumers and connect
> - * with any matching token found on the IRQ producer list.
>   */
>  int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
>  				 struct eventfd_ctx *eventfd)

above belongs to patch7.

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

[PATCH 6/7] irqbypass: Use guard(mutex) in lieu of manual lock+unlock

[Sean Christopherson]

Use guard(mutex) to clean up irqbypass's error handling.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/lib/irqbypass.c | 38 ++++++++++----------------------------
 1 file changed, 10 insertions(+), 28 deletions(-)

diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 6d68a0f71dd9..261ef77f6364 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -99,33 +99,25 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer,
 	if (WARN_ON_ONCE(producer->token))
 		return -EINVAL;
 
-	mutex_lock(&lock);
+	guard(mutex)(&lock);
 
 	list_for_each_entry(tmp, &producers, node) {
-		if (tmp->token == eventfd) {
-			ret = -EBUSY;
-			goto out_err;
-		}
+		if (tmp->token == eventfd)
+			return -EBUSY;
 	}
 
 	list_for_each_entry(consumer, &consumers, node) {
 		if (consumer->token == eventfd) {
 			ret = __connect(producer, consumer);
 			if (ret)
-				goto out_err;
+				return ret;
 			break;
 		}
 	}
 
 	producer->token = eventfd;
 	list_add(&producer->node, &producers);
-
-	mutex_unlock(&lock);
-
 	return 0;
-out_err:
-	mutex_unlock(&lock);
-	return ret;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_producer);
 
@@ -141,14 +133,13 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 	if (!producer->token)
 		return;
 
-	mutex_lock(&lock);
+	guard(mutex)(&lock);
 
 	if (producer->consumer)
 		__disconnect(producer, producer->consumer);
 
 	producer->token = NULL;
 	list_del(&producer->node);
-	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
 
@@ -171,33 +162,25 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
 	if (!consumer->add_producer || !consumer->del_producer)
 		return -EINVAL;
 
-	mutex_lock(&lock);
+	guard(mutex)(&lock);
 
 	list_for_each_entry(tmp, &consumers, node) {
-		if (tmp->token == eventfd || tmp == consumer) {
-			ret = -EBUSY;
-			goto out_err;
-		}
+		if (tmp->token == eventfd || tmp == consumer)
+			return -EBUSY;
 	}
 
 	list_for_each_entry(producer, &producers, node) {
 		if (producer->token == eventfd) {
 			ret = __connect(producer, consumer);
 			if (ret)
-				goto out_err;
+				return ret;
 			break;
 		}
 	}
 
 	consumer->token = eventfd;
 	list_add(&consumer->node, &consumers);
-
-	mutex_unlock(&lock);
-
 	return 0;
-out_err:
-	mutex_unlock(&lock);
-	return ret;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_consumer);
 
@@ -213,13 +196,12 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 	if (!consumer->token)
 		return;
 
-	mutex_lock(&lock);
+	guard(mutex)(&lock);
 
 	if (consumer->producer)
 		__disconnect(consumer->producer, consumer);
 
 	consumer->token = NULL;
 	list_del(&consumer->node);
-	mutex_unlock(&lock);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer);
-- 
2.49.0.504.g3bcea36a83-goog

RE: [PATCH 6/7] irqbypass: Use guard(mutex) in lieu of manual lock+unlock

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> Use guard(mutex) to clean up irqbypass's error handling.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

[PATCH 7/7] irqbypass: Use xarray to track producers and consumers

[Sean Christopherson]

Track IRQ bypass produsers and consumers using an xarray to avoid the O(2n)
insertion time associated with walking a list to check for duplicate
entries, and to search for an partner.

At low (tens or few hundreds) total producer/consumer counts, using a list
is faster due to the need to allocate backing storage for xarray.  But as
count creeps into the thousands, xarray wins easily, and can provide
several orders of magnitude better latency at high counts.  E.g. hundreds
of nanoseconds vs. hundreds of milliseconds.

Cc: Oliver Upton <oliver.upton@linux.dev>
Cc: David Matlack <dmatlack@google.com>
Cc: Like Xu <like.xu.linux@gmail.com>
Reported-by: Yong He <alexyonghe@tencent.com>
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217379
Link: https://lore.kernel.org/all/20230801115646.33990-1-likexu@tencent.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 include/linux/irqbypass.h |  2 --
 virt/lib/irqbypass.c      | 68 +++++++++++++++++++--------------------
 2 files changed, 34 insertions(+), 36 deletions(-)

diff --git a/include/linux/irqbypass.h b/include/linux/irqbypass.h
index 6d4e4882843c..e9d9485eaf99 100644
--- a/include/linux/irqbypass.h
+++ b/include/linux/irqbypass.h
@@ -46,7 +46,6 @@ struct irq_bypass_consumer;
  * for a physical device assigned to a VM.
  */
 struct irq_bypass_producer {
-	struct list_head node;
 	void *token;
 	struct irq_bypass_consumer *consumer;
 	int irq;
@@ -73,7 +72,6 @@ struct irq_bypass_producer {
  * portions of the interrupt handling to the VM.
  */
 struct irq_bypass_consumer {
-	struct list_head node;
 	void *token;
 	struct irq_bypass_producer *producer;
 
diff --git a/virt/lib/irqbypass.c b/virt/lib/irqbypass.c
index 261ef77f6364..3f7734e63d0f 100644
--- a/virt/lib/irqbypass.c
+++ b/virt/lib/irqbypass.c
@@ -22,8 +22,8 @@
 MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("IRQ bypass manager utility module");
 
-static LIST_HEAD(producers);
-static LIST_HEAD(consumers);
+static DEFINE_XARRAY(producers);
+static DEFINE_XARRAY(consumers);
 static DEFINE_MUTEX(lock);
 
 /* @lock must be held when calling connect */
@@ -86,13 +86,13 @@ static void __disconnect(struct irq_bypass_producer *prod,
  * @producer: pointer to producer structure
  * @eventfd: pointer to the eventfd context associated with the producer
  *
- * Add the provided IRQ producer to the list of producers and connect
- * with any matching token found on the IRQ consumers list.
+ * Add the provided IRQ producer to the set of producers and connect with the
+ * consumer with a matching token, if one exists.
  */
 int irq_bypass_register_producer(struct irq_bypass_producer *producer,
 				 struct eventfd_ctx *eventfd)
 {
-	struct irq_bypass_producer *tmp;
+	unsigned long token = (unsigned long)eventfd;
 	struct irq_bypass_consumer *consumer;
 	int ret;
 
@@ -101,22 +101,20 @@ int irq_bypass_register_producer(struct irq_bypass_producer *producer,
 
 	guard(mutex)(&lock);
 
-	list_for_each_entry(tmp, &producers, node) {
-		if (tmp->token == eventfd)
-			return -EBUSY;
-	}
+	ret = xa_insert(&producers, token, producer, GFP_KERNEL);
+	if (ret)
+		return ret;
 
-	list_for_each_entry(consumer, &consumers, node) {
-		if (consumer->token == eventfd) {
-			ret = __connect(producer, consumer);
-			if (ret)
-				return ret;
-			break;
+	consumer = xa_load(&consumers, token);
+	if (consumer) {
+		ret = __connect(producer, consumer);
+		if (ret) {
+			WARN_ON_ONCE(xa_erase(&producers, token) != producer);
+			return ret;
 		}
 	}
 
 	producer->token = eventfd;
-	list_add(&producer->node, &producers);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_producer);
@@ -125,8 +123,9 @@ EXPORT_SYMBOL_GPL(irq_bypass_register_producer);
  * irq_bypass_unregister_producer - unregister IRQ bypass producer
  * @producer: pointer to producer structure
  *
- * Remove a previously registered IRQ producer from the list of producers
- * and disconnect it from any connected IRQ consumer.
+ * Remove a previously registered IRQ producer (note, it's safe to call this
+ * even if registration was unsuccessful).  Disconnect from the associated
+ * consumer, if one exists.
  */
 void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 {
@@ -138,8 +137,8 @@ void irq_bypass_unregister_producer(struct irq_bypass_producer *producer)
 	if (producer->consumer)
 		__disconnect(producer, producer->consumer);
 
+	WARN_ON_ONCE(xa_erase(&producers, (unsigned long)producer->token) != producer);
 	producer->token = NULL;
-	list_del(&producer->node);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
 
@@ -148,11 +147,13 @@ EXPORT_SYMBOL_GPL(irq_bypass_unregister_producer);
  * @consumer: pointer to consumer structure
  * @eventfd: pointer to the eventfd context associated with the consumer
  *
+ * Add the provided IRQ consumer to the set of consumer and connect with the
+ * producer with a matching token, if one exists.
  */
 int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
 				 struct eventfd_ctx *eventfd)
 {
-	struct irq_bypass_consumer *tmp;
+	unsigned long token = (unsigned long)eventfd;
 	struct irq_bypass_producer *producer;
 	int ret;
 
@@ -164,22 +165,20 @@ int irq_bypass_register_consumer(struct irq_bypass_consumer *consumer,
 
 	guard(mutex)(&lock);
 
-	list_for_each_entry(tmp, &consumers, node) {
-		if (tmp->token == eventfd || tmp == consumer)
-			return -EBUSY;
-	}
+	ret = xa_insert(&consumers, token, consumer, GFP_KERNEL);
+	if (ret)
+		return ret;
 
-	list_for_each_entry(producer, &producers, node) {
-		if (producer->token == eventfd) {
-			ret = __connect(producer, consumer);
-			if (ret)
-				return ret;
-			break;
+	producer = xa_load(&producers, token);
+	if (producer) {
+		ret = __connect(producer, consumer);
+		if (ret) {
+			WARN_ON_ONCE(xa_erase(&consumers, token) != consumer);
+			return ret;
 		}
 	}
 
 	consumer->token = eventfd;
-	list_add(&consumer->node, &consumers);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(irq_bypass_register_consumer);
@@ -188,8 +187,9 @@ EXPORT_SYMBOL_GPL(irq_bypass_register_consumer);
  * irq_bypass_unregister_consumer - unregister IRQ bypass consumer
  * @consumer: pointer to consumer structure
  *
- * Remove a previously registered IRQ consumer from the list of consumers
- * and disconnect it from any connected IRQ producer.
+ * Remove a previously registered IRQ consumer (note, it's safe to call this
+ * even if registration was unsuccessful).  Disconnect from the associated
+ * producer, if one exists.
  */
 void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 {
@@ -201,7 +201,7 @@ void irq_bypass_unregister_consumer(struct irq_bypass_consumer *consumer)
 	if (consumer->producer)
 		__disconnect(consumer->producer, consumer);
 
+	WARN_ON_ONCE(xa_erase(&consumers, (unsigned long)consumer->token) != consumer);
 	consumer->token = NULL;
-	list_del(&consumer->node);
 }
 EXPORT_SYMBOL_GPL(irq_bypass_unregister_consumer);
-- 
2.49.0.504.g3bcea36a83-goog

Re: [PATCH 7/7] irqbypass: Use xarray to track producers and consumers

[Binbin Wu]

On 4/5/2025 5:14 AM, Sean Christopherson wrote:
> Track IRQ bypass produsers and consumers using an xarray to avoid the O(2n)
produsers -> producers

> insertion time associated with walking a list to check for duplicate
> entries, and to search for an partner.
>
> At low (tens or few hundreds) total producer/consumer counts, using a list
> is faster due to the need to allocate backing storage for xarray.  But as
> count creeps into the thousands, xarray wins easily, and can provide
> several orders of magnitude better latency at high counts.  E.g. hundreds
> of nanoseconds vs. hundreds of milliseconds.
>
[...]

RE: [PATCH 7/7] irqbypass: Use xarray to track producers and consumers

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Saturday, April 5, 2025 5:15 AM
> 
> Track IRQ bypass produsers and consumers using an xarray to avoid the
> O(2n)
> insertion time associated with walking a list to check for duplicate
> entries, and to search for an partner.
> 
> At low (tens or few hundreds) total producer/consumer counts, using a list
> is faster due to the need to allocate backing storage for xarray.  But as
> count creeps into the thousands, xarray wins easily, and can provide
> several orders of magnitude better latency at high counts.  E.g. hundreds
> of nanoseconds vs. hundreds of milliseconds.

add a link to the original data collected by Like.

> 
> Cc: Oliver Upton <oliver.upton@linux.dev>
> Cc: David Matlack <dmatlack@google.com>
> Cc: Like Xu <like.xu.linux@gmail.com>
> Reported-by: Yong He <alexyonghe@tencent.com>
> Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217379
> Link: https://lore.kernel.org/all/20230801115646.33990-1-
> likexu@tencent.com
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

Re: [PATCH 7/7] irqbypass: Use xarray to track producers and consumers

[Sean Christopherson]

On Thu, Apr 10, 2025, Kevin Tian wrote:
> > From: Sean Christopherson <seanjc@google.com>
> > Sent: Saturday, April 5, 2025 5:15 AM
> > 
> > Track IRQ bypass produsers and consumers using an xarray to avoid the
> > O(2n)
> > insertion time associated with walking a list to check for duplicate
> > entries, and to search for an partner.
> > 
> > At low (tens or few hundreds) total producer/consumer counts, using a list
> > is faster due to the need to allocate backing storage for xarray.  But as
> > count creeps into the thousands, xarray wins easily, and can provide
> > several orders of magnitude better latency at high counts.  E.g. hundreds
> > of nanoseconds vs. hundreds of milliseconds.
> 
> add a link to the original data collected by Like.
> 
> > 
> > Cc: Oliver Upton <oliver.upton@linux.dev>
> > Cc: David Matlack <dmatlack@google.com>
> > Cc: Like Xu <like.xu.linux@gmail.com>
> > Reported-by: Yong He <alexyonghe@tencent.com>
> > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217379
> > Link: https://lore.kernel.org/all/20230801115646.33990-1-likexu@tencent.com

I linked Like's submission here, which has his numbers.  Would it be helpful to
explictly call this out in the meat of the changelog?

RE: [PATCH 7/7] irqbypass: Use xarray to track producers and consumers

[Tian, Kevin]

> From: Sean Christopherson <seanjc@google.com>
> Sent: Thursday, April 10, 2025 10:52 PM
> 
> On Thu, Apr 10, 2025, Kevin Tian wrote:
> > > From: Sean Christopherson <seanjc@google.com>
> > > Sent: Saturday, April 5, 2025 5:15 AM
> > >
> > > Track IRQ bypass produsers and consumers using an xarray to avoid the
> > > O(2n)
> > > insertion time associated with walking a list to check for duplicate
> > > entries, and to search for an partner.
> > >
> > > At low (tens or few hundreds) total producer/consumer counts, using a
> list
> > > is faster due to the need to allocate backing storage for xarray.  But as
> > > count creeps into the thousands, xarray wins easily, and can provide
> > > several orders of magnitude better latency at high counts.  E.g. hundreds
> > > of nanoseconds vs. hundreds of milliseconds.
> >
> > add a link to the original data collected by Like.
> >
> > >
> > > Cc: Oliver Upton <oliver.upton@linux.dev>
> > > Cc: David Matlack <dmatlack@google.com>
> > > Cc: Like Xu <like.xu.linux@gmail.com>
> > > Reported-by: Yong He <alexyonghe@tencent.com>
> > > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217379
> > > Link: https://lore.kernel.org/all/20230801115646.33990-1-
> likexu@tencent.com
> 
> I linked Like's submission here, which has his numbers.  Would it be helpful
> to
> explictly call this out in the meat of the changelog?

No. I just overlooked that line. 😊

Re: [PATCH 0/7] irqbypass: Cleanups and a perf improvement

[Michael S. Tsirkin]

On Fri, Apr 04, 2025 at 02:14:42PM -0700, Sean Christopherson wrote:
> The two primary goals of this series are to make the irqbypass concept
> easier to understand, and to address the terrible performance that can
> result from using a list to track connections.
> 
> For the first goal, track the producer/consumer "tokens" as eventfd context
> pointers instead of opaque "void *".  Supporting arbitrary token types was
> dead infrastructure when it was added 10 years ago, and nothing has changed
> since.  Taking an opaque token makes a *very* simple concept (device signals
> eventfd; KVM listens to eventfd) unnecessarily difficult to understand.
> 
> Burying that simple behind a layer of obfuscation also makes the overall
> code more brittle, as callers can pass in literally anything. I.e. passing
> in a token that will never be paired would go unnoticed.
> 
> For the performance issue, use an xarray.  I'm definitely not wedded to an
> xarray, but IMO it doesn't add meaningful complexity (even requires less
> code), and pretty much Just Works.  Like tried this a while back[1], but
> the implementation had undesirable behavior changes and stalled out.
> 
> To address the use case where huge numbers of VMs are being created without
> _any_ possibility for irqbypass, KVM should probably add a
> KVM_IRQFD_FLAG_NO_IRQBYPASS flag so that userspace can opt-out on a per-IRQ
> basis.  I already proposed a KVM module param[2] to let userspace disable
> IRQ bypass, but that obviously affects all IRQs in all VMs.  It might
> suffice for most use cases, but I can imagine scenarios where the VMM wants
> to be more selective, e.g. when it *knows* a KVM_IRQFD isn't eligible for
> bypass.  And both of those require userspace changes.
> 
> Note, I want to do more aggressive cleanups of irqbypass at some point,
> e.g. not reporting an error to userspace if connect() fails is *awful*
> behavior for environments that want/need irqbypass to always work.  But
> that's a future problem.
> 
> [1] https://lore.kernel.org/all/20230801115646.33990-1-likexu@tencent.com
> [2] https://lore.kernel.org/all/20250401161804.842968-1-seanjc@google.com

vdpa changes seem minor, so

Acked-by: Michael S. Tsirkin <mst@redhat.com>


> Sean Christopherson (7):
>   irqbypass: Drop pointless and misleading THIS_MODULE get/put
>   irqbypass: Drop superfluous might_sleep() annotations
>   irqbypass: Take ownership of producer/consumer token tracking
>   irqbypass: Explicitly track producer and consumer bindings
>   irqbypass: Use paired consumer/producer to disconnect during
>     unregister
>   irqbypass: Use guard(mutex) in lieu of manual lock+unlock
>   irqbypass: Use xarray to track producers and consumers
> 
>  drivers/vfio/pci/vfio_pci_intrs.c |   5 +-
>  drivers/vhost/vdpa.c              |   4 +-
>  include/linux/irqbypass.h         |  38 +++---
>  virt/kvm/eventfd.c                |   3 +-
>  virt/lib/irqbypass.c              | 185 ++++++++++--------------------
>  5 files changed, 88 insertions(+), 147 deletions(-)
> 
> 
> base-commit: 782f9feaa9517caf33186dcdd6b50a8f770ed29b
> -- 
> 2.49.0.504.g3bcea36a83-goog