Re: [PATCH virtio 0/8] virtio_pci_modern: allow parallel admin queue commands execution

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Jun 24, 2024 at 04:51:37PM +0200, Jiri Pirko wrote:
> Mon, Jun 24, 2024 at 03:55:53PM CEST, mst@redhat.com wrote:
> >On Mon, Jun 24, 2024 at 03:46:19PM +0200, Jiri Pirko wrote:
> >> Mon, Jun 24, 2024 at 01:23:01PM CEST, mst@redhat.com wrote:
> >> >On Mon, Jun 24, 2024 at 05:53:52PM +0800, Heng Qi wrote:
> >> >> On Mon, 24 Jun 2024 11:04:43 +0200, Jiri Pirko <jiri@resnulli.us> wrote:
> >> >> > From: Jiri Pirko <jiri@nvidia.com>
> >> >> > 
> >> >> > Currently the admin queue command execution is serialized by a lock.
> >> >> > This patchsets lifts this limitation allowing to execute admin queue
> >> >> > commands in parallel. To do that, admin queue processing needs to be
> >> >> > converted from polling to interrupt based completion.
> >> >> > 
> >> >> > Patches #1-#6 are preparations, making things a bit smoother as well.
> >> >> > Patch #7 implements interrupt based completion for admin queue.
> >> >> 
> >> >> Hi, Jiri
> >> >> 
> >> >> Before this set, I pushed the cvq irq set [1], and the discussion focused on the
> >> >> fact that the newly added irq vector may cause the IO queue to fall back to
> >> >> shared interrupt mode.
> >> >> But it is true that devices implemented according to the specification should
> >> >> not encounter this problem. So what do you think?
> >> 
> >> Wait. Please note that admin queue is only created and used by PF virtio
> >> device. And most probably, this is on hypervisor managing the VFs that
> >> are passed to guest VMs. These VFs does not have admin queue.
> >> 
> >> Therefore, this is hardly comparable to control vq.
> >
> >
> >Well Parav recently posted patches adding admin queue
> >to VFs, with new "self" group type.
> 
> Right, but even so, when device implementation decides to implement and
> enable admin queue, it should also make sure to provide correct amount
> of vectors. My point is, there should not be any breakage in user
> expectation, or am I missing something?


Hmm, I think you are right that cvq is an existing capability
and adminq is newer.

Gimme a couple of days to think all this over, hopefully we'll also see
a new version of the cvq patch, making it easier to see whether they
interact and if so, how.



> 
> >
> >
> >> 
> >> >> 
> >> >> [1] https://lore.kernel.org/all/20240619171708-mutt-send-email-mst@kernel.org/
> >> >
> >> >It's true - this can cause guest to run out of vectors for a variety of
> >> >reasons.
> >> >
> >> >First we have guest irqs - I am guessing avq could use IRQF_SHARED ?
> >> 
> >> There is no avq in quest, under normal circumstances. Unless for some
> >> reason somebody passes trough virtio PF into guest.
> >
> >
> >At the moment, but this will change soon.
> >
> >
> >> 
> >> >I am not sure why we don't allow IRQF_SHARED for the config
> >> >interrupt though. So I think addressing this part can be deferred.
> >> >
> >> >Second, we might not have enough msix vectors on the device. Here sharing
> >> >with e.g. cvq and further with config interrupt would make sense.
> >> 
> >> For cvq irq vector, I believe that sharing with config irq makes sense.
> >> Even for admin queue maybe. But again, admin queue is on PF. I don't
> >> think this is a real concern.
> >> 
> >> 
> >> >
> >> >Jiri do you think you can help Heng Qi hammer out a solution for cvq?
> >> >I feel this will work will then benefit in a similar way,
> >> >and having us poll aggressively for cvq but not admin commands
> >> >does not make much sense, right?
> >> >
> >> >> > Patch #8 finally removes the admin queue serialization lock.
> >> >> > 
> >> >> > Jiri Pirko (8):
> >> >> >   virtio_pci: push out single vq find code to vp_find_one_vq_msix()
> >> >> >   virtio_pci_modern: treat vp_dev->admin_vq.info.vq pointer as static
> >> >> >   virtio: push out code to vp_avq_index()
> >> >> >   virtio: create admin queues alongside other virtqueues
> >> >> >   virtio_pci_modern: create admin queue of queried size
> >> >> >   virtio_pci_modern: pass cmd as an identification token
> >> >> >   virtio_pci_modern: use completion instead of busy loop to wait on
> >> >> >     admin cmd result
> >> >> >   virtio_pci_modern: remove admin queue serialization lock
> >> >> > 
> >> >> >  drivers/virtio/virtio.c            |  28 +----
> >> >> >  drivers/virtio/virtio_pci_common.c | 109 ++++++++++++++------
> >> >> >  drivers/virtio/virtio_pci_common.h |   9 +-
> >> >> >  drivers/virtio/virtio_pci_modern.c | 160 ++++++++++++-----------------
> >> >> >  include/linux/virtio.h             |   2 +
> >> >> >  include/linux/virtio_config.h      |   2 -
> >> >> >  6 files changed, 150 insertions(+), 160 deletions(-)
> >> >> > 
> >> >> > -- 
> >> >> > 2.45.1
> >> >> > 
> >> >> > 
> >> >
> >