* [PATCH] virtio_net: suppress cpu stall when free_unused_bufs @ 2023-04-27 4:34 Wenliang Wang 2023-04-27 6:20 ` Xuan Zhuo 0 siblings, 1 reply; 13+ messages in thread From: Wenliang Wang @ 2023-04-27 4:34 UTC (permalink / raw) To: mst, jasowang, davem, edumazet, kuba, pabeni Cc: virtualization, netdev, linux-kernel, Wenliang Wang For multi-queue and large rx-ring-size use case, the following error occurred when free_unused_bufs: rcu: INFO: rcu_sched self-detected stall on CPU. Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> --- drivers/net/virtio_net.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ea1bd4bb326d..21d8382fd2c7 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) struct virtqueue *vq = vi->rq[i].vq; while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) virtnet_rq_free_unused_buf(vq, buf); + schedule(); } } -- 2.20.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 4:34 [PATCH] virtio_net: suppress cpu stall when free_unused_bufs Wenliang Wang @ 2023-04-27 6:20 ` Xuan Zhuo 2023-04-27 7:02 ` Wenliang Wang 0 siblings, 1 reply; 13+ messages in thread From: Xuan Zhuo @ 2023-04-27 6:20 UTC (permalink / raw) To: Wenliang Wang Cc: virtualization, netdev, linux-kernel, Wenliang Wang, mst, jasowang, davem, edumazet, kuba, pabeni On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > For multi-queue and large rx-ring-size use case, the following error Cound you give we one number for example? > occurred when free_unused_bufs: > rcu: INFO: rcu_sched self-detected stall on CPU. > > Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > --- > drivers/net/virtio_net.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index ea1bd4bb326d..21d8382fd2c7 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > struct virtqueue *vq = vi->rq[i].vq; > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > virtnet_rq_free_unused_buf(vq, buf); > + schedule(); Just for rq? Do we need to do the same thing for sq? Thanks. > } > } > > -- > 2.20.1 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 6:20 ` Xuan Zhuo @ 2023-04-27 7:02 ` Wenliang Wang 2023-04-27 7:13 ` Xuan Zhuo 0 siblings, 1 reply; 13+ messages in thread From: Wenliang Wang @ 2023-04-27 7:02 UTC (permalink / raw) To: Xuan Zhuo Cc: virtualization, netdev, linux-kernel, mst, jasowang, davem, edumazet, kuba, pabeni On 4/27/23 2:20 PM, Xuan Zhuo wrote: > On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: >> For multi-queue and large rx-ring-size use case, the following error > > Cound you give we one number for example? 128 queues and 16K queue_size is typical. > >> occurred when free_unused_bufs: >> rcu: INFO: rcu_sched self-detected stall on CPU. >> >> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> >> --- >> drivers/net/virtio_net.c | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >> index ea1bd4bb326d..21d8382fd2c7 100644 >> --- a/drivers/net/virtio_net.c >> +++ b/drivers/net/virtio_net.c >> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) >> struct virtqueue *vq = vi->rq[i].vq; >> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) >> virtnet_rq_free_unused_buf(vq, buf); >> + schedule(); > > Just for rq? > > Do we need to do the same thing for sq? Rq buffers are pre-allocated, take seconds to free rq unused buffers. Sq unused buffers are much less, so do the same for sq is optional. > > Thanks. > > >> } >> } >> >> -- >> 2.20.1 >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 7:02 ` Wenliang Wang @ 2023-04-27 7:13 ` Xuan Zhuo 2023-04-27 8:12 ` Michael S. Tsirkin 0 siblings, 1 reply; 13+ messages in thread From: Xuan Zhuo @ 2023-04-27 7:13 UTC (permalink / raw) To: Wenliang Wang Cc: virtualization, netdev, linux-kernel, mst, jasowang, davem, edumazet, kuba, pabeni On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > On 4/27/23 2:20 PM, Xuan Zhuo wrote: > > On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > >> For multi-queue and large rx-ring-size use case, the following error > > > > Cound you give we one number for example? > > 128 queues and 16K queue_size is typical. > > > > >> occurred when free_unused_bufs: > >> rcu: INFO: rcu_sched self-detected stall on CPU. > >> > >> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > >> --- > >> drivers/net/virtio_net.c | 1 + > >> 1 file changed, 1 insertion(+) > >> > >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > >> index ea1bd4bb326d..21d8382fd2c7 100644 > >> --- a/drivers/net/virtio_net.c > >> +++ b/drivers/net/virtio_net.c > >> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > >> struct virtqueue *vq = vi->rq[i].vq; > >> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > >> virtnet_rq_free_unused_buf(vq, buf); > >> + schedule(); > > > > Just for rq? > > > > Do we need to do the same thing for sq? > Rq buffers are pre-allocated, take seconds to free rq unused buffers. > > Sq unused buffers are much less, so do the same for sq is optional. I got. I think we should look for a way, compatible with the less queues or the smaller rings. Calling schedule() directly may be not a good way. Thanks. > > > > > Thanks. > > > > > >> } > >> } > >> > >> -- > >> 2.20.1 > >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 7:13 ` Xuan Zhuo @ 2023-04-27 8:12 ` Michael S. Tsirkin 2023-04-27 8:13 ` Xuan Zhuo 0 siblings, 1 reply; 13+ messages in thread From: Michael S. Tsirkin @ 2023-04-27 8:12 UTC (permalink / raw) To: Xuan Zhuo Cc: Wenliang Wang, virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: > On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > > > > On 4/27/23 2:20 PM, Xuan Zhuo wrote: > > > On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > >> For multi-queue and large rx-ring-size use case, the following error > > > > > > Cound you give we one number for example? > > > > 128 queues and 16K queue_size is typical. > > > > > > > >> occurred when free_unused_bufs: > > >> rcu: INFO: rcu_sched self-detected stall on CPU. > > >> > > >> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > > >> --- > > >> drivers/net/virtio_net.c | 1 + > > >> 1 file changed, 1 insertion(+) > > >> > > >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > >> index ea1bd4bb326d..21d8382fd2c7 100644 > > >> --- a/drivers/net/virtio_net.c > > >> +++ b/drivers/net/virtio_net.c > > >> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > > >> struct virtqueue *vq = vi->rq[i].vq; > > >> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > > >> virtnet_rq_free_unused_buf(vq, buf); > > >> + schedule(); > > > > > > Just for rq? > > > > > > Do we need to do the same thing for sq? > > Rq buffers are pre-allocated, take seconds to free rq unused buffers. > > > > Sq unused buffers are much less, so do the same for sq is optional. > > I got. > > I think we should look for a way, compatible with the less queues or the smaller > rings. Calling schedule() directly may be not a good way. > > Thanks. Why isn't it a good way? > > > > > > > > > Thanks. > > > > > > > > >> } > > >> } > > >> > > >> -- > > >> 2.20.1 > > >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:12 ` Michael S. Tsirkin @ 2023-04-27 8:13 ` Xuan Zhuo 2023-04-27 8:23 ` Michael S. Tsirkin 0 siblings, 1 reply; 13+ messages in thread From: Xuan Zhuo @ 2023-04-27 8:13 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Wenliang Wang, virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: > > On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > > > > > > > On 4/27/23 2:20 PM, Xuan Zhuo wrote: > > > > On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > >> For multi-queue and large rx-ring-size use case, the following error > > > > > > > > Cound you give we one number for example? > > > > > > 128 queues and 16K queue_size is typical. > > > > > > > > > > >> occurred when free_unused_bufs: > > > >> rcu: INFO: rcu_sched self-detected stall on CPU. > > > >> > > > >> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > > > >> --- > > > >> drivers/net/virtio_net.c | 1 + > > > >> 1 file changed, 1 insertion(+) > > > >> > > > >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > > >> index ea1bd4bb326d..21d8382fd2c7 100644 > > > >> --- a/drivers/net/virtio_net.c > > > >> +++ b/drivers/net/virtio_net.c > > > >> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > > > >> struct virtqueue *vq = vi->rq[i].vq; > > > >> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > > > >> virtnet_rq_free_unused_buf(vq, buf); > > > >> + schedule(); > > > > > > > > Just for rq? > > > > > > > > Do we need to do the same thing for sq? > > > Rq buffers are pre-allocated, take seconds to free rq unused buffers. > > > > > > Sq unused buffers are much less, so do the same for sq is optional. > > > > I got. > > > > I think we should look for a way, compatible with the less queues or the smaller > > rings. Calling schedule() directly may be not a good way. > > > > Thanks. > > Why isn't it a good way? For the small ring, I don't think it is a good way, maybe we only deal with one buf, then call schedule(). We can call the schedule() after processing a certain number of buffers, or check need_resched () first. Thanks. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > >> } > > > >> } > > > >> > > > >> -- > > > >> 2.20.1 > > > >> > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:13 ` Xuan Zhuo @ 2023-04-27 8:23 ` Michael S. Tsirkin 2023-04-27 8:49 ` Wenliang Wang 2023-04-27 10:45 ` [PATCH] " Qi Zheng 0 siblings, 2 replies; 13+ messages in thread From: Michael S. Tsirkin @ 2023-04-27 8:23 UTC (permalink / raw) To: Xuan Zhuo Cc: Wenliang Wang, virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni On Thu, Apr 27, 2023 at 04:13:45PM +0800, Xuan Zhuo wrote: > On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: > > > On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > > > > > > > > > > On 4/27/23 2:20 PM, Xuan Zhuo wrote: > > > > > On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > > > > >> For multi-queue and large rx-ring-size use case, the following error > > > > > > > > > > Cound you give we one number for example? > > > > > > > > 128 queues and 16K queue_size is typical. > > > > > > > > > > > > > >> occurred when free_unused_bufs: > > > > >> rcu: INFO: rcu_sched self-detected stall on CPU. > > > > >> > > > > >> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > > > > >> --- > > > > >> drivers/net/virtio_net.c | 1 + > > > > >> 1 file changed, 1 insertion(+) > > > > >> > > > > >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > > > >> index ea1bd4bb326d..21d8382fd2c7 100644 > > > > >> --- a/drivers/net/virtio_net.c > > > > >> +++ b/drivers/net/virtio_net.c > > > > >> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > > > > >> struct virtqueue *vq = vi->rq[i].vq; > > > > >> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > > > > >> virtnet_rq_free_unused_buf(vq, buf); > > > > >> + schedule(); > > > > > > > > > > Just for rq? > > > > > > > > > > Do we need to do the same thing for sq? > > > > Rq buffers are pre-allocated, take seconds to free rq unused buffers. > > > > > > > > Sq unused buffers are much less, so do the same for sq is optional. > > > > > > I got. > > > > > > I think we should look for a way, compatible with the less queues or the smaller > > > rings. Calling schedule() directly may be not a good way. > > > > > > Thanks. > > > > Why isn't it a good way? > > For the small ring, I don't think it is a good way, maybe we only deal with one > buf, then call schedule(). > > We can call the schedule() after processing a certain number of buffers, > or check need_resched () first. > > Thanks. Wenliang, does if (need_resched()) schedule(); fix the issue for you? > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > >> } > > > > >> } > > > > >> > > > > >> -- > > > > >> 2.20.1 > > > > >> > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:23 ` Michael S. Tsirkin @ 2023-04-27 8:49 ` Wenliang Wang 2023-04-27 8:51 ` Xuan Zhuo 2023-04-27 10:45 ` [PATCH] " Qi Zheng 1 sibling, 1 reply; 13+ messages in thread From: Wenliang Wang @ 2023-04-27 8:49 UTC (permalink / raw) To: Michael S. Tsirkin, Xuan Zhuo Cc: virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni On 4/27/23 4:23 PM, Michael S. Tsirkin wrote: > On Thu, Apr 27, 2023 at 04:13:45PM +0800, Xuan Zhuo wrote: >> On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: >>> On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: >>>> On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: >>>>> >>>>> >>>>> On 4/27/23 2:20 PM, Xuan Zhuo wrote: >>>>>> On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: >>>>>>> For multi-queue and large rx-ring-size use case, the following error >>>>>> >>>>>> Cound you give we one number for example? >>>>> >>>>> 128 queues and 16K queue_size is typical. >>>>> >>>>>> >>>>>>> occurred when free_unused_bufs: >>>>>>> rcu: INFO: rcu_sched self-detected stall on CPU. >>>>>>> >>>>>>> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> >>>>>>> --- >>>>>>> drivers/net/virtio_net.c | 1 + >>>>>>> 1 file changed, 1 insertion(+) >>>>>>> >>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>>>>>> index ea1bd4bb326d..21d8382fd2c7 100644 >>>>>>> --- a/drivers/net/virtio_net.c >>>>>>> +++ b/drivers/net/virtio_net.c >>>>>>> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) >>>>>>> struct virtqueue *vq = vi->rq[i].vq; >>>>>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) >>>>>>> virtnet_rq_free_unused_buf(vq, buf); >>>>>>> + schedule(); >>>>>> >>>>>> Just for rq? >>>>>> >>>>>> Do we need to do the same thing for sq? >>>>> Rq buffers are pre-allocated, take seconds to free rq unused buffers. >>>>> >>>>> Sq unused buffers are much less, so do the same for sq is optional. >>>> >>>> I got. >>>> >>>> I think we should look for a way, compatible with the less queues or the smaller >>>> rings. Calling schedule() directly may be not a good way. >>>> >>>> Thanks. >>> >>> Why isn't it a good way? >> >> For the small ring, I don't think it is a good way, maybe we only deal with one >> buf, then call schedule(). >> >> We can call the schedule() after processing a certain number of buffers, >> or check need_resched () first. >> >> Thanks. > > > Wenliang, does > if (need_resched()) > schedule(); > fix the issue for you? > Yeah, it works better. > >> >> >>> >>>> >>>>> >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> -- >>>>>>> 2.20.1 >>>>>>> >>> > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:49 ` Wenliang Wang @ 2023-04-27 8:51 ` Xuan Zhuo 2023-04-27 10:46 ` [PATCH v2] " Wenliang Wang 0 siblings, 1 reply; 13+ messages in thread From: Xuan Zhuo @ 2023-04-27 8:51 UTC (permalink / raw) To: Wenliang Wang Cc: virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni, Michael S. Tsirkin On Thu, 27 Apr 2023 16:49:58 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > On 4/27/23 4:23 PM, Michael S. Tsirkin wrote: > > On Thu, Apr 27, 2023 at 04:13:45PM +0800, Xuan Zhuo wrote: > >> On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > >>> On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: > >>>> On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > >>>>> > >>>>> > >>>>> On 4/27/23 2:20 PM, Xuan Zhuo wrote: > >>>>>> On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > >>>>>>> For multi-queue and large rx-ring-size use case, the following error > >>>>>> > >>>>>> Cound you give we one number for example? > >>>>> > >>>>> 128 queues and 16K queue_size is typical. > >>>>> > >>>>>> > >>>>>>> occurred when free_unused_bufs: > >>>>>>> rcu: INFO: rcu_sched self-detected stall on CPU. > >>>>>>> > >>>>>>> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > >>>>>>> --- > >>>>>>> drivers/net/virtio_net.c | 1 + > >>>>>>> 1 file changed, 1 insertion(+) > >>>>>>> > >>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > >>>>>>> index ea1bd4bb326d..21d8382fd2c7 100644 > >>>>>>> --- a/drivers/net/virtio_net.c > >>>>>>> +++ b/drivers/net/virtio_net.c > >>>>>>> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > >>>>>>> struct virtqueue *vq = vi->rq[i].vq; > >>>>>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > >>>>>>> virtnet_rq_free_unused_buf(vq, buf); > >>>>>>> + schedule(); > >>>>>> > >>>>>> Just for rq? > >>>>>> > >>>>>> Do we need to do the same thing for sq? > >>>>> Rq buffers are pre-allocated, take seconds to free rq unused buffers. > >>>>> > >>>>> Sq unused buffers are much less, so do the same for sq is optional. > >>>> > >>>> I got. > >>>> > >>>> I think we should look for a way, compatible with the less queues or the smaller > >>>> rings. Calling schedule() directly may be not a good way. > >>>> > >>>> Thanks. > >>> > >>> Why isn't it a good way? > >> > >> For the small ring, I don't think it is a good way, maybe we only deal with one > >> buf, then call schedule(). > >> > >> We can call the schedule() after processing a certain number of buffers, > >> or check need_resched () first. > >> > >> Thanks. > > > > > > Wenliang, does > > if (need_resched()) > > schedule(); > > fix the issue for you? > > > Yeah, it works better. I prefer to use it in combination with a fixed number(such as 256). Every time 256 buffers are processed, check need_resched(). This can accommodate large rings and small rings. Also, it is necessary to add similar logic to sq. Although the possibility is low, it is possible that the same problem will occur. Thanks. > > > >> > >> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> Thanks. > >>>>>> > >>>>>> > >>>>>>> } > >>>>>>> } > >>>>>>> > >>>>>>> -- > >>>>>>> 2.20.1 > >>>>>>> > >>> > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:51 ` Xuan Zhuo @ 2023-04-27 10:46 ` Wenliang Wang 2023-04-28 1:09 ` Michael S. Tsirkin 0 siblings, 1 reply; 13+ messages in thread From: Wenliang Wang @ 2023-04-27 10:46 UTC (permalink / raw) To: mst, jasowang, davem, edumazet, kuba, pabeni Cc: virtualization, netdev, linux-kernel, Wenliang Wang For multi-queue and large ring-size use case, the following error occurred when free_unused_bufs: rcu: INFO: rcu_sched self-detected stall on CPU. Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> --- v2: -add need_resched check. -apply same logic to sq. --- drivers/net/virtio_net.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ea1bd4bb326d..573558b69a60 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -3559,12 +3559,16 @@ static void free_unused_bufs(struct virtnet_info *vi) struct virtqueue *vq = vi->sq[i].vq; while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) virtnet_sq_free_unused_buf(vq, buf); + if (need_resched()) + schedule(); } for (i = 0; i < vi->max_queue_pairs; i++) { struct virtqueue *vq = vi->rq[i].vq; while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) virtnet_rq_free_unused_buf(vq, buf); + if (need_resched()) + schedule(); } } -- 2.20.1 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v2] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 10:46 ` [PATCH v2] " Wenliang Wang @ 2023-04-28 1:09 ` Michael S. Tsirkin 0 siblings, 0 replies; 13+ messages in thread From: Michael S. Tsirkin @ 2023-04-28 1:09 UTC (permalink / raw) To: Wenliang Wang Cc: jasowang, davem, edumazet, kuba, pabeni, virtualization, netdev, linux-kernel On Thu, Apr 27, 2023 at 06:46:18PM +0800, Wenliang Wang wrote: > For multi-queue and large ring-size use case, the following error > occurred when free_unused_bufs: > rcu: INFO: rcu_sched self-detected stall on CPU. > > Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> pls send vN+1 as a new thread not as a reply in existing thread of vN. > --- > v2: > -add need_resched check. > -apply same logic to sq. > --- > drivers/net/virtio_net.c | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index ea1bd4bb326d..573558b69a60 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -3559,12 +3559,16 @@ static void free_unused_bufs(struct virtnet_info *vi) > struct virtqueue *vq = vi->sq[i].vq; > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > virtnet_sq_free_unused_buf(vq, buf); > + if (need_resched()) > + schedule(); > } > > for (i = 0; i < vi->max_queue_pairs; i++) { > struct virtqueue *vq = vi->rq[i].vq; > while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > virtnet_rq_free_unused_buf(vq, buf); > + if (need_resched()) > + schedule(); > } > } > > -- > 2.20.1 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 8:23 ` Michael S. Tsirkin 2023-04-27 8:49 ` Wenliang Wang @ 2023-04-27 10:45 ` Qi Zheng 2023-04-28 13:56 ` Willem de Bruijn 1 sibling, 1 reply; 13+ messages in thread From: Qi Zheng @ 2023-04-27 10:45 UTC (permalink / raw) To: Michael S. Tsirkin, Xuan Zhuo Cc: Wenliang Wang, virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni On 2023/4/27 16:23, Michael S. Tsirkin wrote: > On Thu, Apr 27, 2023 at 04:13:45PM +0800, Xuan Zhuo wrote: >> On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: >>> On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: >>>> On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: >>>>> >>>>> >>>>> On 4/27/23 2:20 PM, Xuan Zhuo wrote: >>>>>> On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: >>>>>>> For multi-queue and large rx-ring-size use case, the following error >>>>>> >>>>>> Cound you give we one number for example? >>>>> >>>>> 128 queues and 16K queue_size is typical. >>>>> >>>>>> >>>>>>> occurred when free_unused_bufs: >>>>>>> rcu: INFO: rcu_sched self-detected stall on CPU. >>>>>>> >>>>>>> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> >>>>>>> --- >>>>>>> drivers/net/virtio_net.c | 1 + >>>>>>> 1 file changed, 1 insertion(+) >>>>>>> >>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >>>>>>> index ea1bd4bb326d..21d8382fd2c7 100644 >>>>>>> --- a/drivers/net/virtio_net.c >>>>>>> +++ b/drivers/net/virtio_net.c >>>>>>> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) >>>>>>> struct virtqueue *vq = vi->rq[i].vq; >>>>>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) >>>>>>> virtnet_rq_free_unused_buf(vq, buf); >>>>>>> + schedule(); >>>>>> >>>>>> Just for rq? >>>>>> >>>>>> Do we need to do the same thing for sq? >>>>> Rq buffers are pre-allocated, take seconds to free rq unused buffers. >>>>> >>>>> Sq unused buffers are much less, so do the same for sq is optional. >>>> >>>> I got. >>>> >>>> I think we should look for a way, compatible with the less queues or the smaller >>>> rings. Calling schedule() directly may be not a good way. >>>> >>>> Thanks. >>> >>> Why isn't it a good way? >> >> For the small ring, I don't think it is a good way, maybe we only deal with one >> buf, then call schedule(). >> >> We can call the schedule() after processing a certain number of buffers, >> or check need_resched () first. >> >> Thanks. > > > Wenliang, does > if (need_resched()) > schedule(); Can we just use cond_resched()? > fix the issue for you? > > >> >> >>> >>>> >>>>> >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> -- >>>>>>> 2.20.1 >>>>>>> >>> > -- Thanks, Qi ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] virtio_net: suppress cpu stall when free_unused_bufs 2023-04-27 10:45 ` [PATCH] " Qi Zheng @ 2023-04-28 13:56 ` Willem de Bruijn 0 siblings, 0 replies; 13+ messages in thread From: Willem de Bruijn @ 2023-04-28 13:56 UTC (permalink / raw) To: Qi Zheng, Michael S. Tsirkin, Xuan Zhuo Cc: Wenliang Wang, virtualization, netdev, linux-kernel, jasowang, davem, edumazet, kuba, pabeni Qi Zheng wrote: > > > On 2023/4/27 16:23, Michael S. Tsirkin wrote: > > On Thu, Apr 27, 2023 at 04:13:45PM +0800, Xuan Zhuo wrote: > >> On Thu, 27 Apr 2023 04:12:44 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > >>> On Thu, Apr 27, 2023 at 03:13:44PM +0800, Xuan Zhuo wrote: > >>>> On Thu, 27 Apr 2023 15:02:26 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > >>>>> > >>>>> > >>>>> On 4/27/23 2:20 PM, Xuan Zhuo wrote: > >>>>>> On Thu, 27 Apr 2023 12:34:33 +0800, Wenliang Wang <wangwenliang.1995@bytedance.com> wrote: > >>>>>>> For multi-queue and large rx-ring-size use case, the following error > >>>>>> > >>>>>> Cound you give we one number for example? > >>>>> > >>>>> 128 queues and 16K queue_size is typical. > >>>>> > >>>>>> > >>>>>>> occurred when free_unused_bufs: > >>>>>>> rcu: INFO: rcu_sched self-detected stall on CPU. > >>>>>>> > >>>>>>> Signed-off-by: Wenliang Wang <wangwenliang.1995@bytedance.com> > >>>>>>> --- > >>>>>>> drivers/net/virtio_net.c | 1 + > >>>>>>> 1 file changed, 1 insertion(+) > >>>>>>> > >>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > >>>>>>> index ea1bd4bb326d..21d8382fd2c7 100644 > >>>>>>> --- a/drivers/net/virtio_net.c > >>>>>>> +++ b/drivers/net/virtio_net.c > >>>>>>> @@ -3565,6 +3565,7 @@ static void free_unused_bufs(struct virtnet_info *vi) > >>>>>>> struct virtqueue *vq = vi->rq[i].vq; > >>>>>>> while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) > >>>>>>> virtnet_rq_free_unused_buf(vq, buf); > >>>>>>> + schedule(); > >>>>>> > >>>>>> Just for rq? > >>>>>> > >>>>>> Do we need to do the same thing for sq? > >>>>> Rq buffers are pre-allocated, take seconds to free rq unused buffers. > >>>>> > >>>>> Sq unused buffers are much less, so do the same for sq is optional. > >>>> > >>>> I got. > >>>> > >>>> I think we should look for a way, compatible with the less queues or the smaller > >>>> rings. Calling schedule() directly may be not a good way. > >>>> > >>>> Thanks. > >>> > >>> Why isn't it a good way? > >> > >> For the small ring, I don't think it is a good way, maybe we only deal with one > >> buf, then call schedule(). > >> > >> We can call the schedule() after processing a certain number of buffers, > >> or check need_resched () first. > >> > >> Thanks. > > > > > > Wenliang, does > > if (need_resched()) > > schedule(); > > Can we just use cond_resched()? I believe that is preferred. But v2 still calls schedule directly. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-04-28 13:56 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-04-27 4:34 [PATCH] virtio_net: suppress cpu stall when free_unused_bufs Wenliang Wang 2023-04-27 6:20 ` Xuan Zhuo 2023-04-27 7:02 ` Wenliang Wang 2023-04-27 7:13 ` Xuan Zhuo 2023-04-27 8:12 ` Michael S. Tsirkin 2023-04-27 8:13 ` Xuan Zhuo 2023-04-27 8:23 ` Michael S. Tsirkin 2023-04-27 8:49 ` Wenliang Wang 2023-04-27 8:51 ` Xuan Zhuo 2023-04-27 10:46 ` [PATCH v2] " Wenliang Wang 2023-04-28 1:09 ` Michael S. Tsirkin 2023-04-27 10:45 ` [PATCH] " Qi Zheng 2023-04-28 13:56 ` Willem de Bruijn
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).