[PATCH] virtio_blk: add use_irq_affinity module parameter

[PATCH] virtio_blk: add use_irq_affinity module parameter

[Liu, Changcheng]

When creating many virtio-blk devices, probe starts failing with
-ENOSPC (-28) because the system runs out of interrupt vectors:

  virtio_blk virtioNNN: probe with driver virtio_blk failed with error -28

By default virtio-blk uses managed IRQ affinity, which reserves an
interrupt vector on every CPU for each device (about nr_cpus vectors per
device). On a host with many CPUs and many devices this exhausts the
vectors long before all devices are probed.

Add use_irq_affinity (default true, no behaviour change). Set it to 0 to
use unmanaged interrupts, so each device only uses a couple of vectors
instead of one per CPU, allowing far more devices to probe.

Signed-off-by: Liu, Changcheng <jerrliu@nvidia.com>

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 32bf3ba07a9d..184f1c1f4485 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -41,6 +41,11 @@ static unsigned int poll_queues;
 module_param(poll_queues, uint, 0644);
 MODULE_PARM_DESC(poll_queues, "The number of dedicated virtqueues for polling I/O");
 
+static bool use_irq_affinity = true;
+module_param(use_irq_affinity, bool, 0644);
+MODULE_PARM_DESC(use_irq_affinity,
+		 "Use managed IRQ affinity for virtqueues (default: true)");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -1016,7 +1021,8 @@ static int init_vq(struct virtio_blk *vblk)
 	}
 
 	/* Discover virtqueues and write information to configuration.  */
-	err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
+	err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info,
+			      use_irq_affinity ? &desc : NULL);
 	if (err)
 		goto out;
 
-- 
2.43.7

Re: [PATCH] virtio_blk: add use_irq_affinity module parameter

[Stefan Hajnoczi]

On Sat, Aug 1, 2026 at 2:21 AM Liu, Changcheng <jerrliu@nvidia.com> wrote:
>
> When creating many virtio-blk devices, probe starts failing with
> -ENOSPC (-28) because the system runs out of interrupt vectors:
>
>   virtio_blk virtioNNN: probe with driver virtio_blk failed with error -28
>
> By default virtio-blk uses managed IRQ affinity, which reserves an
> interrupt vector on every CPU for each device (about nr_cpus vectors per
> device). On a host with many CPUs and many devices this exhausts the
> vectors long before all devices are probed.
>
> Add use_irq_affinity (default true, no behaviour change). Set it to 0 to
> use unmanaged interrupts, so each device only uses a couple of vectors
> instead of one per CPU, allowing far more devices to probe.
>
> Signed-off-by: Liu, Changcheng <jerrliu@nvidia.com>

There is already a num_request_queues module parameter for cases where
the user wishes to reduce the number of virtqueues. Did you benchmark
that and decide the performance of many queues sharing a single irq
makes it worth adding another module parameter?

Stefan

> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 32bf3ba07a9d..184f1c1f4485 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -41,6 +41,11 @@ static unsigned int poll_queues;
>  module_param(poll_queues, uint, 0644);
>  MODULE_PARM_DESC(poll_queues, "The number of dedicated virtqueues for polling I/O");
>
> +static bool use_irq_affinity = true;
> +module_param(use_irq_affinity, bool, 0644);
> +MODULE_PARM_DESC(use_irq_affinity,
> +                "Use managed IRQ affinity for virtqueues (default: true)");
> +
>  static int major;
>  static DEFINE_IDA(vd_index_ida);
>
> @@ -1016,7 +1021,8 @@ static int init_vq(struct virtio_blk *vblk)
>         }
>
>         /* Discover virtqueues and write information to configuration.  */
> -       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
> +       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info,
> +                             use_irq_affinity ? &desc : NULL);
>         if (err)
>                 goto out;
>
> --
> 2.43.7
>
>

Re: [PATCH] virtio_blk: add use_irq_affinity module parameter

[Liu, Changcheng]

On Wed, Aug 05, 2026 at 01:45:58PM -0400, Stefan Hajnoczi wrote:
> 
> On Sat, Aug 1, 2026 at 2:21 AM Liu, Changcheng <jerrliu@nvidia.com> wrote:
> >
> > When creating many virtio-blk devices, probe starts failing with
> > -ENOSPC (-28) because the system runs out of interrupt vectors:
> >
> >   virtio_blk virtioNNN: probe with driver virtio_blk failed with error -28
> >
> > By default virtio-blk uses managed IRQ affinity, which reserves an
> > interrupt vector on every CPU for each device (about nr_cpus vectors per
> > device). On a host with many CPUs and many devices this exhausts the
> > vectors long before all devices are probed.
> >
> > Add use_irq_affinity (default true, no behaviour change). Set it to 0 to
> > use unmanaged interrupts, so each device only uses a couple of vectors
> > instead of one per CPU, allowing far more devices to probe.
> >
> > Signed-off-by: Liu, Changcheng <jerrliu@nvidia.com>
> 
> There is already a num_request_queues module parameter for cases where
> the user wishes to reduce the number of virtqueues. Did you benchmark
> that and decide the performance of many queues sharing a single irq
> makes it worth adding another module parameter?
> 
> Stefan

num_request_queues does not address this case because each virtio-blk device
already has only one request virtqueue when the failure occurs, so the queue
count cannot be reduced further.

The issue is reproduced after probing approximately 800 virtio-blk devices on
a 64-core bare-metal host. With managed IRQ affinity, the IRQ-vector reservations
are eventually exhausted and probing additional devices fails with -ENOSPC.
Disabling managed IRQ affinity avoids these per-CPU vector reservations and
allows more devices to be probed successfully.

This patch does not cause multiple virtqueues to share a single IRQ. Passing a
NULL affinity descriptor only disables managed IRQ affinity. Regardless of the
number of virtqueues, virtio-pci still first attempts to allocate a separate
interrupt vector for each interrupt-driven virtqueue. Only if per-VQ vector
allocation fails does the existing virtio-pci fallback share a vector among
the virtqueues, and that fallback is independent of this patch.

The new parameter defaults to enabled, so the existing driver behavior remains
unchanged unless the option is explicitly disabled. The option is intended for
environments with a very large number of devices, particularly for functional
device testing.

- Changcheng

> 
> > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > index 32bf3ba07a9d..184f1c1f4485 100644
> > --- a/drivers/block/virtio_blk.c
> > +++ b/drivers/block/virtio_blk.c
> > @@ -41,6 +41,11 @@ static unsigned int poll_queues;
> >  module_param(poll_queues, uint, 0644);
> >  MODULE_PARM_DESC(poll_queues, "The number of dedicated virtqueues for polling I/O");
> >
> > +static bool use_irq_affinity = true;
> > +module_param(use_irq_affinity, bool, 0644);
> > +MODULE_PARM_DESC(use_irq_affinity,
> > +                "Use managed IRQ affinity for virtqueues (default: true)");
> > +
> >  static int major;
> >  static DEFINE_IDA(vd_index_ida);
> >
> > @@ -1016,7 +1021,8 @@ static int init_vq(struct virtio_blk *vblk)
> >         }
> >
> >         /* Discover virtqueues and write information to configuration.  */
> > -       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
> > +       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info,
> > +                             use_irq_affinity ? &desc : NULL);
> >         if (err)
> >                 goto out;
> >
> > --
> > 2.43.7
> >
> >

Re: [PATCH] virtio_blk: add use_irq_affinity module parameter

[Stefan Hajnoczi]

On Wed, Aug 5, 2026 at 10:40 PM Liu, Changcheng <jerrliu@nvidia.com> wrote:
>
> On Wed, Aug 05, 2026 at 01:45:58PM -0400, Stefan Hajnoczi wrote:
> >
> > On Sat, Aug 1, 2026 at 2:21 AM Liu, Changcheng <jerrliu@nvidia.com> wrote:
> > >
> > > When creating many virtio-blk devices, probe starts failing with
> > > -ENOSPC (-28) because the system runs out of interrupt vectors:
> > >
> > >   virtio_blk virtioNNN: probe with driver virtio_blk failed with error -28
> > >
> > > By default virtio-blk uses managed IRQ affinity, which reserves an
> > > interrupt vector on every CPU for each device (about nr_cpus vectors per
> > > device). On a host with many CPUs and many devices this exhausts the
> > > vectors long before all devices are probed.
> > >
> > > Add use_irq_affinity (default true, no behaviour change). Set it to 0 to
> > > use unmanaged interrupts, so each device only uses a couple of vectors
> > > instead of one per CPU, allowing far more devices to probe.
> > >
> > > Signed-off-by: Liu, Changcheng <jerrliu@nvidia.com>
> >
> > There is already a num_request_queues module parameter for cases where
> > the user wishes to reduce the number of virtqueues. Did you benchmark
> > that and decide the performance of many queues sharing a single irq
> > makes it worth adding another module parameter?
> >
> > Stefan
>
> num_request_queues does not address this case because each virtio-blk device
> already has only one request virtqueue when the failure occurs, so the queue
> count cannot be reduced further.

Okay.

> The issue is reproduced after probing approximately 800 virtio-blk devices on
> a 64-core bare-metal host. With managed IRQ affinity, the IRQ-vector reservations
> are eventually exhausted and probing additional devices fails with -ENOSPC.
> Disabling managed IRQ affinity avoids these per-CPU vector reservations and
> allows more devices to be probed successfully.

I don't follow. My understanding was that non-managed IRQ vectors are
allocated by request_irq(), which is called during virtio_find_vqs().
Probing 800 devices would still require 800 * (1 virtqueue irq + 1
config change irq) = 1,600 vectors. How come non-managed IRQs do not
hit the limit here?

Thanks,
Stefan

> This patch does not cause multiple virtqueues to share a single IRQ. Passing a
> NULL affinity descriptor only disables managed IRQ affinity. Regardless of the
> number of virtqueues, virtio-pci still first attempts to allocate a separate
> interrupt vector for each interrupt-driven virtqueue. Only if per-VQ vector
> allocation fails does the existing virtio-pci fallback share a vector among
> the virtqueues, and that fallback is independent of this patch.
>
> The new parameter defaults to enabled, so the existing driver behavior remains
> unchanged unless the option is explicitly disabled. The option is intended for
> environments with a very large number of devices, particularly for functional
> device testing.
>
> - Changcheng
>
> >
> > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > > index 32bf3ba07a9d..184f1c1f4485 100644
> > > --- a/drivers/block/virtio_blk.c
> > > +++ b/drivers/block/virtio_blk.c
> > > @@ -41,6 +41,11 @@ static unsigned int poll_queues;
> > >  module_param(poll_queues, uint, 0644);
> > >  MODULE_PARM_DESC(poll_queues, "The number of dedicated virtqueues for polling I/O");
> > >
> > > +static bool use_irq_affinity = true;
> > > +module_param(use_irq_affinity, bool, 0644);
> > > +MODULE_PARM_DESC(use_irq_affinity,
> > > +                "Use managed IRQ affinity for virtqueues (default: true)");
> > > +
> > >  static int major;
> > >  static DEFINE_IDA(vd_index_ida);
> > >
> > > @@ -1016,7 +1021,8 @@ static int init_vq(struct virtio_blk *vblk)
> > >         }
> > >
> > >         /* Discover virtqueues and write information to configuration.  */
> > > -       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
> > > +       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info,
> > > +                             use_irq_affinity ? &desc : NULL);
> > >         if (err)
> > >                 goto out;
> > >
> > > --
> > > 2.43.7
> > >
> > >

Re: [PATCH] virtio_blk: add use_irq_affinity module parameter

[Liu, Changcheng]

On Tue, Aug 11, 2026 at 10:37:28AM -0400, Stefan Hajnoczi wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Wed, Aug 5, 2026 at 10:40 PM Liu, Changcheng <jerrliu@nvidia.com> wrote:
> >
> > On Wed, Aug 05, 2026 at 01:45:58PM -0400, Stefan Hajnoczi wrote:
> > >
> > > On Sat, Aug 1, 2026 at 2:21 AM Liu, Changcheng <jerrliu@nvidia.com> wrote:
> > > >
> > > > When creating many virtio-blk devices, probe starts failing with
> > > > -ENOSPC (-28) because the system runs out of interrupt vectors:
> > > >
> > > >   virtio_blk virtioNNN: probe with driver virtio_blk failed with error -28
> > > >
> > > > By default virtio-blk uses managed IRQ affinity, which reserves an
> > > > interrupt vector on every CPU for each device (about nr_cpus vectors per
> > > > device). On a host with many CPUs and many devices this exhausts the
> > > > vectors long before all devices are probed.
> > > >
> > > > Add use_irq_affinity (default true, no behaviour change). Set it to 0 to
> > > > use unmanaged interrupts, so each device only uses a couple of vectors
> > > > instead of one per CPU, allowing far more devices to probe.
> > > >
> > > > Signed-off-by: Liu, Changcheng <jerrliu@nvidia.com>
> > >
> > > There is already a num_request_queues module parameter for cases where
> > > the user wishes to reduce the number of virtqueues. Did you benchmark
> > > that and decide the performance of many queues sharing a single irq
> > > makes it worth adding another module parameter?
> > >
> > > Stefan
> >
> > num_request_queues does not address this case because each virtio-blk device
> > already has only one request virtqueue when the failure occurs, so the queue
> > count cannot be reduced further.
> 
> Okay.
> 
> > The issue is reproduced after probing approximately 800 virtio-blk devices on
> > a 64-core bare-metal host. With managed IRQ affinity, the IRQ-vector reservations
> > are eventually exhausted and probing additional devices fails with -ENOSPC.
> > Disabling managed IRQ affinity avoids these per-CPU vector reservations and
> > allows more devices to be probed successfully.
> 
> I don't follow. My understanding was that non-managed IRQ vectors are
> allocated by request_irq(), which is called during virtio_find_vqs().
> Probing 800 devices would still require 800 * (1 virtqueue irq + 1
> config change irq) = 1,600 vectors. How come non-managed IRQs do not
> hit the limit here?
> 
> Thanks,
> Stefan
> 

With one request queue, virtio-blk allocates two MSI-X interrupts: an
unmanaged configuration interrupt and a managed request-queue interrupt.

The request-queue interrupt's managed affinity mask covers all possible
CPUs. On x86, irq_matrix_reserve_managed() reserves one vector slot on
every CPU in that mask so the interrupt can migrate safely during CPU
hotplug. Consequently, every virtio-blk device consumes one managed
vector slot on every CPU for its lifetime.

Once any CPU exhausts its roughly 200 usable vector slots, managed vector
allocation fails even if other CPUs still have capacity. virtio-pci then
falls back to its shared-vector MSI-X allocation, for which the affinity
descriptor is discarded and the interrupts are unmanaged. Allocation can
continue using capacity on the other CPUs, explaining the observed limit
of approximately 800 devices.

When managed affinity is disabled, both interrupts are activated on
individual CPUs and distributed by the vector allocator. APIC vector
numbers are per-CPU, so the same vector number can be reused on different
CPUs. On a 64-CPU system, 800 devices therefore consume approximately
800 * 2 / 64 = 25 vector slots per CPU.

Thanks
Changcheng

>
> > This patch does not cause multiple virtqueues to share a single IRQ. Passing a
> > NULL affinity descriptor only disables managed IRQ affinity. Regardless of the
> > number of virtqueues, virtio-pci still first attempts to allocate a separate
> > interrupt vector for each interrupt-driven virtqueue. Only if per-VQ vector
> > allocation fails does the existing virtio-pci fallback share a vector among
> > the virtqueues, and that fallback is independent of this patch.
> >
> > The new parameter defaults to enabled, so the existing driver behavior remains
> > unchanged unless the option is explicitly disabled. The option is intended for
> > environments with a very large number of devices, particularly for functional
> > device testing.
> >
> > - Changcheng
> >
> > >
> > > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> > > > index 32bf3ba07a9d..184f1c1f4485 100644
> > > > --- a/drivers/block/virtio_blk.c
> > > > +++ b/drivers/block/virtio_blk.c
> > > > @@ -41,6 +41,11 @@ static unsigned int poll_queues;
> > > >  module_param(poll_queues, uint, 0644);
> > > >  MODULE_PARM_DESC(poll_queues, "The number of dedicated virtqueues for polling I/O");
> > > >
> > > > +static bool use_irq_affinity = true;
> > > > +module_param(use_irq_affinity, bool, 0644);
> > > > +MODULE_PARM_DESC(use_irq_affinity,
> > > > +                "Use managed IRQ affinity for virtqueues (default: true)");
> > > > +
> > > >  static int major;
> > > >  static DEFINE_IDA(vd_index_ida);
> > > >
> > > > @@ -1016,7 +1021,8 @@ static int init_vq(struct virtio_blk *vblk)
> > > >         }
> > > >
> > > >         /* Discover virtqueues and write information to configuration.  */
> > > > -       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info, &desc);
> > > > +       err = virtio_find_vqs(vdev, num_vqs, vqs, vqs_info,
> > > > +                             use_irq_affinity ? &desc : NULL);
> > > >         if (err)
> > > >                 goto out;
> > > >
> > > > --
> > > > 2.43.7
> > > >
> > > >