Re: [PATCH net-next v4 2/5] virtio_net: enable irq for the control vq

[Heng Qi <hengqi@linux.alibaba.com>]

On Mon, 8 Jul 2024 13:40:42 +0200, Jiri Pirko <jiri@resnulli.us> wrote:
> Wed, Jun 26, 2024 at 01:51:00PM CEST, jiri@resnulli.us wrote:
> >Wed, Jun 26, 2024 at 11:58:08AM CEST, mst@redhat.com wrote:
> >>On Wed, Jun 26, 2024 at 10:43:11AM +0200, Jiri Pirko wrote:
> >>> Wed, Jun 26, 2024 at 10:08:14AM CEST, mst@redhat.com wrote:
> >>> >On Wed, Jun 26, 2024 at 09:52:58AM +0200, Jiri Pirko wrote:
> >>> >> Thu, Jun 20, 2024 at 12:31:34PM CEST, hengqi@linux.alibaba.com wrote:
> >>> >> >On Thu, 20 Jun 2024 06:11:40 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>> >> >> On Thu, Jun 20, 2024 at 06:10:51AM -0400, Michael S. Tsirkin wrote:
> >>> >> >> > On Thu, Jun 20, 2024 at 05:53:15PM +0800, Heng Qi wrote:
> >>> >> >> > > On Thu, 20 Jun 2024 16:26:05 +0800, Jason Wang <jasowang@redhat.com> wrote:
> >>> >> >> > > > On Thu, Jun 20, 2024 at 4:21 PM Jason Wang <jasowang@redhat.com> wrote:
> >>> >> >> > > > >
> >>> >> >> > > > > On Thu, Jun 20, 2024 at 3:35 PM Heng Qi <hengqi@linux.alibaba.com> wrote:
> >>> >> >> > > > > >
> >>> >> >> > > > > > On Wed, 19 Jun 2024 17:19:12 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>> >> >> > > > > > > On Thu, Jun 20, 2024 at 12:19:05AM +0800, Heng Qi wrote:
> >>> >> >> > > > > > > > @@ -5312,7 +5315,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi)
> >>> >> >> > > > > > > >
> >>> >> >> > > > > > > >     /* Parameters for control virtqueue, if any */
> >>> >> >> > > > > > > >     if (vi->has_cvq) {
> >>> >> >> > > > > > > > -           callbacks[total_vqs - 1] = NULL;
> >>> >> >> > > > > > > > +           callbacks[total_vqs - 1] = virtnet_cvq_done;
> >>> >> >> > > > > > > >             names[total_vqs - 1] = "control";
> >>> >> >> > > > > > > >     }
> >>> >> >> > > > > > > >
> >>> >> >> > > > > > >
> >>> >> >> > > > > > > If the # of MSIX vectors is exactly for data path VQs,
> >>> >> >> > > > > > > this will cause irq sharing between VQs which will degrade
> >>> >> >> > > > > > > performance significantly.
> >>> >> >> > > > > > >
> >>> >> >> > > > >
> >>> >> >> > > > > Why do we need to care about buggy management? I think libvirt has
> >>> >> >> > > > > been teached to use 2N+2 since the introduction of the multiqueue[1].
> >>> >> >> > > > 
> >>> >> >> > > > And Qemu can calculate it correctly automatically since:
> >>> >> >> > > > 
> >>> >> >> > > > commit 51a81a2118df0c70988f00d61647da9e298483a4
> >>> >> >> > > > Author: Jason Wang <jasowang@redhat.com>
> >>> >> >> > > > Date:   Mon Mar 8 12:49:19 2021 +0800
> >>> >> >> > > > 
> >>> >> >> > > >     virtio-net: calculating proper msix vectors on init
> >>> >> >> > > > 
> >>> >> >> > > >     Currently, the default msix vectors for virtio-net-pci is 3 which is
> >>> >> >> > > >     obvious not suitable for multiqueue guest, so we depends on the user
> >>> >> >> > > >     or management tools to pass a correct vectors parameter. In fact, we
> >>> >> >> > > >     can simplifying this by calculating the number of vectors on realize.
> >>> >> >> > > > 
> >>> >> >> > > >     Consider we have N queues, the number of vectors needed is 2*N + 2
> >>> >> >> > > >     (#queue pairs + plus one config interrupt and control vq). We didn't
> >>> >> >> > > >     check whether or not host support control vq because it was added
> >>> >> >> > > >     unconditionally by qemu to avoid breaking legacy guests such as Minix.
> >>> >> >> > > > 
> >>> >> >> > > >     Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com
> >>> >> >> > > >     Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
> >>> >> >> > > >     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> >>> >> >> > > >     Signed-off-by: Jason Wang <jasowang@redhat.com>
> >>> >> >> > > 
> >>> >> >> > > Yes, devices designed according to the spec need to reserve an interrupt
> >>> >> >> > > vector for ctrlq. So, Michael, do we want to be compatible with buggy devices?
> >>> >> >> > > 
> >>> >> >> > > Thanks.
> >>> >> >> > 
> >>> >> >> > These aren't buggy, the spec allows this. So don't fail, but
> >>> >> >> > I'm fine with using polling if not enough vectors.
> >>> >> >> 
> >>> >> >> sharing with config interrupt is easier code-wise though, FWIW -
> >>> >> >> we don't need to maintain two code-paths.
> >>> >> >
> >>> >> >Yes, it works well - config change irq is used less before - and will not fail.
> >>> >> 
> >>> >> Please note I'm working on such fallback for admin queue. I would Like
> >>> >> to send the patchset by the end of this week. You can then use it easily
> >>> >> for cvq.
> >>> >> 
> >>> >> Something like:
> >>> >> /* the config->find_vqs() implementation */
> >>> >> int vp_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
> >>> >>                 struct virtqueue *vqs[], vq_callback_t *callbacks[],
> >>> >>                 const char * const names[], const bool *ctx,
> >>> >>                 struct irq_affinity *desc)
> >>> >> {
> >>> >>         int err;
> >>> >> 
> >>> >>         /* Try MSI-X with one vector per queue. */
> >>> >>         err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
> >>> >>                                VP_VQ_VECTOR_POLICY_EACH, ctx, desc);
> >>> >>         if (!err)
> >>> >>                 return 0;
> >>> >>         /* Fallback: MSI-X with one shared vector for config and
> >>> >>          * slow path queues, one vector per queue for the rest. */
> >>> >>         err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
> >>> >>                                VP_VQ_VECTOR_POLICY_SHARED_SLOW, ctx, desc);
> >>> >>         if (!err)
> >>> >>                 return 0;
> >>> >>         /* Fallback: MSI-X with one vector for config, one shared for queues. */
> >>> >>         err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names,
> >>> >>                                VP_VQ_VECTOR_POLICY_SHARED, ctx, desc);
> >>> >>         if (!err)
> >>> >>                 return 0;
> >>> >>         /* Is there an interrupt? If not give up. */
> >>> >>         if (!(to_vp_device(vdev)->pci_dev->irq))
> >>> >>                 return err;
> >>> >>         /* Finally fall back to regular interrupts. */
> >>> >>         return vp_find_vqs_intx(vdev, nvqs, vqs, callbacks, names, ctx);
> >>> >> }
> >>> >> 
> >>> >> 
> >>> >
> >>> >
> >>> >Well for cvq, we'll need to adjust the API so core
> >>> >knows cvq interrupts are be shared with config not
> >>> >datapath.
> >>> 
> >>> Agreed. I was thinking about introducing some info struct and pass array
> >>> of it instead of callbacks[] and names[]. Then the struct can contain
> >>> flag indication. Something like:
> >>> 
> >>> struct vq_info {
> >>> 	vq_callback_t *callback;
> >>> 	const char *name;
> >>> 	bool slow_path;
> >>> };
> >>> 
> >>
> >>Yes. Add ctx too? There were attempts at it already btw.
> >
> >Yep, ctx too. I can take a stab at it if noone else is interested.
> 
> Since this work is in v4, and I hope it will get merged soon, I plan to

I've seen your set and will help review it tomorrow.

> send v2 of admin queue parallelization patchset after that. Here it is:
> https://github.com/jpirko/linux_mlxsw/tree/wip_virtio_parallel_aq2
> 
> Heng, note the last patch:
> virtio_pci: allow to indicate virtqueue being slow path
> 
> That is not part of my set, it is ment to be merged in your control
> queue patchset. Then you can just indicate cvq to be slow like this:
> 
>         /* Parameters for control virtqueue, if any */
>         if (vi->has_cvq) {
>                 vqs_info[total_vqs - 1].callback = virtnet_cvq_done;
>                 vqs_info[total_vqs - 1].name = "control";
>                 vqs_info[total_vqs - 1].slow_path = true;
>         }
> 
> I just wanted to let you know this is in process so you may prepare.
> Will keep you informed.

Thanks for letting me know!

Regards,
Heng

> 
> Thanks.