Re: Re: Re: Re: Re: [PATCH v1 0/7] introduce virtnet_mon for monitor virtio_net

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, Dec 16, 2025 at 03:47:55PM +0800, Lange Tang wrote:
> At 2025-12-16 11:59:14, "Jason Wang" <jasowang@redhat.com> wrote:
> >On Mon, Dec 15, 2025 at 4:12 PM Lange Tang <lange_tang@163.com> wrote:
> >>
> >> At 2025-12-15 14:41:42, "Jason Wang" <jasowang@redhat.com> wrote:
> >> >On Sat, Dec 13, 2025 at 10:41 AM Lange Tang <lange_tang@163.com> wrote:
> >> >>
> >> >> At 2025-12-11 16:32:15, "Jason Wang" <jasowang@redhat.com> wrote:
> >> >> >On Thu, Dec 11, 2025 at 10:52 AM Lange Tang <lange_tang@163.com> wrote:
> >> >> >>
> >> >> >> At 2025-12-10 17:04:04, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> >> >On Thu, Nov 27, 2025 at 11:24:00AM +0800, Longjun Tang wrote:
> >> >> >> >> From: Tang Longjun <tanglongjun@kylinos.cn>
> >> >> >> >>
> >> >> >> >> hi,
> >> >> >> >> virtnet_mon is used to monitor the data packets of the virtio_net driver
> >> >> >> >> and the related parameters of virtqueue, useful for tracking its status
> >> >> >> >> and troubleshooting faults.
> >> >> >> >>
> >> >> >> >> pls review. tks
> >> >> >> >>
> >> >> >> >> Best regard.
> >> >> >> >
> >> >> >> >what does this achieve that direct use of tracing would not?
> >> >> >>
> >> >> >> I apologize that my explanation of virtnet_mon was not detailed enough.
> >> >> >> virtnet_mon uses kprobe and buffers to monitor virtio_net.
> >> >> >> To monitor virtio_net, it is necessary to track the member parameters of the virtqueue corresponding to each data packet and output them.
> >> >> >> When PPS very high, other tracing techniques, such as ebpf, may not be able to handle it, resulting in data loss because they do not have sufficiently large buffers to batch export log data.
> >> >> >
> >> >> >Can you expand more about this? For example, in which kind of setup
> >> >> >and what do you want to trace and why ebpf can't handle that. Note
> >> >> >that the most lightweight stuff is the counter, have you tried that?
> >> >>
> >> >> For example, when there is occasional latency in data transmission between the
> >> >> virtual network frontend (virtio_net) and backend (such as vhost_net),
> >> >> we may need to track the time taken for each packet received and sent in the virtio_net driver.
> >> >> Typically, we accomplish this using eBPF, such as bpftrace. The pseudocode might include the following:
> >> >> """
> >> >> kprobe:skb_recv_done {
> >> >>         printf("%ld skb_recv_done Cpu:%d ...\n",...);
> >> >> }
> >> >> kprobe:skb_xmit_done {
> >> >>         printf("%ld skb_xmit_done Cpu:%d ...\n",...);
> >> >> }
> >> >> kprobe:virtnet_poll {
> >> >>         printf("%ld virtnet_poll Cpu:%d budget:%d ...\n",...);
> >> >> }
> >> >> kprobe:start_xmit {
> >> >>   ...
> >> >>   printf("%ld start_xmit Cpu:%d type:%s seq:%ld ...\n",...)
> >> >> }
> >> >> kprobe:gro_receive_skb {
> >> >>   ...
> >> >>   printf("%ld gro_receive_skb Cpu:%d type:%s seq:%ld ...\n",...)
> >> >> }
> >> >> kprobe:receive_buf {
> >> >>   ...
> >> >>   printf("%ld receive_buf Cpu:%d name:%s avali_idx:%d used_idx:%d ...\n",...);
> >> >> }
> >> >> """
> >> >> Using the above bpftrace code, we can track the timestamps of the data as it passes through these functions,
> >> >> along with skb and virtqueue information, and output logs via printf for further diagnosis of the causes of the latency.
> >> >> Interestingly, a significant amount of logs were found to be missing when executing these bpftrace codes.
> >> >> Below is the testing environment:
> >> >> VM: 8G8C，virtio_net mq=4, kernel 6.18-rc7, iperf3 -s -p 1314
> >> >> HOST: iperf3 -c 192.168.122.218 -t 100 -p 1314 -P 4
> >> >> It was also found that when testing with mq=1, there was no log loss.
> >> >>
> >> >> Compared to mq=1, the reason for log loss at mq=4 is suspected to be due to data being sent or received
> >> >> by different CPUs. Additionally, under the 4-thread iperf testing scenario with PPS > 150,000,
> >> >> the log data is asynchronously output from different CPUs, leading to excessive IO pressure that causes log data loss.
> >> >
> >> >I think what I don't understand is how the things you introduced here
> >> >may help in this case?
> >>
> >> The virtnet_mon module introduced here abandons eBPF and uses kprobe + kfifo.
> >> In the aforementioned cases, all the information that needs to be tracked first enters kfifo,
> >> then is formatted into logs and cached in a large buffer.
> >> Finally, it is exported to user space in batches through the virtnet_mon_read function,
> >> thereby reducing IO pressure and preventing log loss.
> >
> >Well, this "problem" seems not virtio-net specific. Have you tried
> >with BPF ringbuf or perfbuf?
> 
> Concerning the ringbuf and perfbuf in bpf, I may need some time to verify whether this can resolve this "problem".
>  I will get back to you with the results. 
> 
> On the other hand, I did not find any tracepoints in virtio_net that track the virtqueue, 
> such as name, num_free, avail.idx, used.idx, last_used_idx, etc. 
> Could you consider inserting these tracepoints in some key functions? This would facilitate direct tracking by perf. 
> For example:
> start_xmit
> receive_buf
> skb_xmit_done
> skb_recv_done

sure, that's a reasonable thing to add.

> >
> >Thanks
> >
> >>
> >> Thanks
> >> >
> >> >Thanks
> >> >
> >> >>
> >> >> The above are some of my personal thoughts, and I would love to hear your opinion.
> >> >> Best regard.
> >> >>
> >> >> >
> >> >> >>
> >> >> >> As for the duplicate code, it is only to obtain the layout of the relevant structure, and I have not yet thought of a way to avoid duplication. I would love to hear your suggestions.
> >> >> >
> >> >> >Thanks
> >> >> >
> >> >> >>
> >> >> >> >
> >> >> >> >> Tang Longjun (7):
> >> >> >> >>   tools/virtio/virtnet_mon: create misc driver for virtnet_mon
> >> >> >> >>   tools/virtio/virtnet_mon: add kfifo to virtnet_mon
> >> >> >> >>   tools/virtio/virtnet_mon: add kprobe start_xmit
> >> >> >> >>   tools/virtio/virtnet_mon: add kprobe gro_receive_skb
> >> >> >> >>   tools/virtio/virtnet_mon: add kprobe ip_local_deliver
> >> >> >> >>   tools/virtio/virtnet_mon: add kprobe skb_xmit_done and skb_recv_done
> >> >> >> >>   tools/virtio/virtnet_mon: add README file for virtnet_moin
> >> >> >> >>
> >> >> >> >>  tools/virtio/virtnet_mon/Makefile      |   10 +
> >> >> >> >>  tools/virtio/virtnet_mon/README        |   35 +
> >> >> >> >>  tools/virtio/virtnet_mon/virtnet_mon.c | 1048 ++++++++++++++++++++++++
> >> >> >> >>  3 files changed, 1093 insertions(+)
> >> >> >> >>  create mode 100644 tools/virtio/virtnet_mon/Makefile
> >> >> >> >>  create mode 100644 tools/virtio/virtnet_mon/README
> >> >> >> >>  create mode 100644 tools/virtio/virtnet_mon/virtnet_mon.c
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> 2.43.0