Re: [PATCH virtio 0/8] virtio_pci_modern: allow parallel admin queue commands execution

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Jun 24, 2024 at 05:53:52PM +0800, Heng Qi wrote:
> On Mon, 24 Jun 2024 11:04:43 +0200, Jiri Pirko <jiri@resnulli.us> wrote:
> > From: Jiri Pirko <jiri@nvidia.com>
> > 
> > Currently the admin queue command execution is serialized by a lock.
> > This patchsets lifts this limitation allowing to execute admin queue
> > commands in parallel. To do that, admin queue processing needs to be
> > converted from polling to interrupt based completion.
> > 
> > Patches #1-#6 are preparations, making things a bit smoother as well.
> > Patch #7 implements interrupt based completion for admin queue.
> 
> Hi, Jiri
> 
> Before this set, I pushed the cvq irq set [1], and the discussion focused on the
> fact that the newly added irq vector may cause the IO queue to fall back to
> shared interrupt mode.
> But it is true that devices implemented according to the specification should
> not encounter this problem. So what do you think?
> 
> [1] https://lore.kernel.org/all/20240619171708-mutt-send-email-mst@kernel.org/

It's true - this can cause guest to run out of vectors for a variety of
reasons.

First we have guest irqs - I am guessing avq could use IRQF_SHARED ?
I am not sure why we don't allow IRQF_SHARED for the config
interrupt though. So I think addressing this part can be deferred.

Second, we might not have enough msix vectors on the device. Here sharing
with e.g. cvq and further with config interrupt would make sense.

Jiri do you think you can help Heng Qi hammer out a solution for cvq?
I feel this will work will then benefit in a similar way,
and having us poll aggressively for cvq but not admin commands
does not make much sense, right?

> > Patch #8 finally removes the admin queue serialization lock.
> > 
> > Jiri Pirko (8):
> >   virtio_pci: push out single vq find code to vp_find_one_vq_msix()
> >   virtio_pci_modern: treat vp_dev->admin_vq.info.vq pointer as static
> >   virtio: push out code to vp_avq_index()
> >   virtio: create admin queues alongside other virtqueues
> >   virtio_pci_modern: create admin queue of queried size
> >   virtio_pci_modern: pass cmd as an identification token
> >   virtio_pci_modern: use completion instead of busy loop to wait on
> >     admin cmd result
> >   virtio_pci_modern: remove admin queue serialization lock
> > 
> >  drivers/virtio/virtio.c            |  28 +----
> >  drivers/virtio/virtio_pci_common.c | 109 ++++++++++++++------
> >  drivers/virtio/virtio_pci_common.h |   9 +-
> >  drivers/virtio/virtio_pci_modern.c | 160 ++++++++++++-----------------
> >  include/linux/virtio.h             |   2 +
> >  include/linux/virtio_config.h      |   2 -
> >  6 files changed, 150 insertions(+), 160 deletions(-)
> > 
> > -- 
> > 2.45.1
> > 
> >