Re: [virtio-comment] Re: [PATCH RESEND] virtio-net: support setting coalescing params for multiple vqs

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Dec 21, 2023 at 03:01:57PM +0800, Heng Qi wrote:
> 
> 
> 在 2023/12/21 下午2:56, Michael S. Tsirkin 写道:
> > On Wed, Dec 20, 2023 at 10:40:34PM +0800, Heng Qi wrote:
> > > Currently, when each time the driver attempts to update the coalescing parameters
> > > for a vq, it needs to kick the device and wait for the ctrlq response to return.
> > But there's no fundamental reason for it to do so.
> 
> Indeed, please look at the current upstream netdim code:
> 
> static void virtnet_rx_dim_work(struct work_struct *work)
> {
>         struct dim *dim = container_of(work, struct dim, work);
>         struct receive_queue *rq = container_of(dim,
>                         struct receive_queue, dim);
>         struct virtnet_info *vi = rq->vq->vdev->priv;
>         struct net_device *dev = vi->dev;
>         struct dim_cq_moder update_moder;
>         int i, qnum, err;
> 
>         if (!rtnl_trylock())
>                 return;
> 
>         /* Each rxq's work is queued by "net_dim()->schedule_work()"
>          * in response to NAPI traffic changes. Note that dim->profile_ix
>          * for each rxq is updated prior to the queuing action.
>          * So we only need to traverse and update profiles for all rxqs
>          * in the work which is holding rtnl_lock.
>          */
>         for (i = 0; i < vi->curr_queue_pairs; i++) {            
> <-----------------
>                 rq = &vi->rq[i];
>                 dim = &rq->dim;
>                 qnum = rq - vi->rq;
> 
>                 if (!rq->dim_enabled)
>                         continue;
> 
>                 update_moder = net_dim_get_rx_moderation(dim->mode,
> dim->profile_ix);
>                 if (update_moder.usec != rq->intr_coal.max_usecs ||
>                     update_moder.pkts != rq->intr_coal.max_packets) {
>                         err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, qnum,
> update_moder.usec,
> update_moder.pkts);
>                         if (err)
>                                 pr_debug("%s: Failed to send dim parameters
> on rxq%d\n",
>                                          dev->name, qnum);
>                         dim->state = DIM_START_MEASURE;
>                 }
>         }
> 
>         rtnl_unlock();
> }
> 
> > It can just as well submit multiple commands and then wait.
> 
> Sending multiple commands and then waiting reduces the number of kicks.
> But it does not reduce the number of device DMAs. I already responded to
> this in a thread to Jason.
> please check:
> 
> https://lists.oasis-open.org/archives/virtio-comment/202312/msg00142.html
> 
> "
> Overall, batch reqs are sufficient. Because the current major overhead is
> the number of DMAs.
> For example, for a device with 256 queues,
> 
> For the current upstream code, the overhead is 256 kicks + 256*8 DMA times.
> The overhead of batch cmds is 1 kick + 256*8 DMA times.
> The overhead of batch reqs is 1 kick + 8 DMA times.
> 
> Below is 8 DMA times:
> - get avail idx 1 time
> - Pull available ring information 1 time
> - Pull the desc pointed to by the avail ring 3 times
> - Pull the hdr and out bufs pointed to by avail ring desc 2 times
> - Write once to the status buf pointed to by status 1 time "
> 
> Thanks.

So there's more DMA but it's all slow path.
Why do we need to micro-optimize it?
What is the overhead practically, in milliseconds?

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/