Re: [RFC v4 0/5] Add packed virtqueue to shadow virtqueue

[Sahil Siddiq <icegambit91@gmail.com>]

Hi,

On 2/6/25 12:42 PM, Eugenio Perez Martin wrote:
> On Thu, Feb 6, 2025 at 6:26 AM Sahil Siddiq <icegambit91@gmail.com> wrote:
>>
>> Hi,
>>
>> On 2/4/25 11:40 PM, Eugenio Perez Martin wrote:
>>> On Tue, Feb 4, 2025 at 1:49 PM Sahil Siddiq <icegambit91@gmail.com> wrote:
>>>> On 1/31/25 12:27 PM, Eugenio Perez Martin wrote:
>>>>> On Fri, Jan 31, 2025 at 6:04 AM Sahil Siddiq <icegambit91@gmail.com> wrote:
>>>>>> On 1/24/25 1:04 PM, Eugenio Perez Martin wrote:
>>>>>>> On Fri, Jan 24, 2025 at 6:47 AM Sahil Siddiq <icegambit91@gmail.com> wrote:
>>>>>>>> On 1/21/25 10:07 PM, Eugenio Perez Martin wrote:
>>>>>>>>> On Sun, Jan 19, 2025 at 7:37 AM Sahil Siddiq <icegambit91@gmail.com> wrote:
>>>>>>>>>> On 1/7/25 1:35 PM, Eugenio Perez Martin wrote:
>>>>>>>>>> [...]
>>>>>>>>>> Apologies for the delay in replying. It took me a while to figure
>>>>>>>>>> this out, but I have now understood why this doesn't work. L1 is
>>>>>>>>>> unable to receive messages from L0 because they get filtered out
>>>>>>>>>> by hw/net/virtio-net.c:receive_filter [1]. There's an issue with
>>>>>>>>>> the MAC addresses.
>>>>>>>>>>
>>>>>>>>>> In L0, I have:
>>>>>>>>>>
>>>>>>>>>> $ ip a show tap0
>>>>>>>>>> 6: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
>>>>>>>>>>           link/ether d2:6d:b9:61:e1:9a brd ff:ff:ff:ff:ff:ff
>>>>>>>>>>           inet 111.1.1.1/24 scope global tap0
>>>>>>>>>>              valid_lft forever preferred_lft forever
>>>>>>>>>>           inet6 fe80::d06d:b9ff:fe61:e19a/64 scope link proto kernel_ll
>>>>>>>>>>              valid_lft forever preferred_lft forever
>>>>>>>>>>
>>>>>>>>>> In L1:
>>>>>>>>>>
>>>>>>>>>> # ip a show eth0
>>>>>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
>>>>>>>>>>           link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
>>>>>>>>>>           altname enp0s2
>>>>>>>>>>           inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
>>>>>>>>>>              valid_lft 83455sec preferred_lft 83455sec
>>>>>>>>>>           inet6 fec0::7bd2:265e:3b8e:5acc/64 scope site dynamic noprefixroute
>>>>>>>>>>              valid_lft 86064sec preferred_lft 14064sec
>>>>>>>>>>           inet6 fe80::50e7:5bf6:fff8:a7b0/64 scope link noprefixroute
>>>>>>>>>>              valid_lft forever preferred_lft forever
>>>>>>>>>>
>>>>>>>>>> I'll call this L1-eth0.
>>>>>>>>>>
>>>>>>>>>> In L2:
>>>>>>>>>> # ip a show eth0
>>>>>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP gro0
>>>>>>>>>>           link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
>>>>>>>>>>           altname enp0s7
>>>>>>>>>>           inet 111.1.1.2/24 scope global eth0
>>>>>>>>>>              valid_lft forever preferred_lft forever
>>>>>>>>>>
>>>>>>>>>> I'll call this L2-eth0.
>>>>>>>>>>
>>>>>>>>>> Apart from eth0, lo is the only other device in both L1 and L2.
>>>>>>>>>>
>>>>>>>>>> A frame that L1 receives from L0 has L2-eth0's MAC address (LSB = 57)
>>>>>>>>>> as its destination address. When booting L2 with x-svq=false, the
>>>>>>>>>> value of n->mac in VirtIONet is also L2-eth0. So, L1 accepts
>>>>>>>>>> the frames and passes them on to L2 and pinging works [2].
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So this behavior is interesting by itself. But L1's kernel net system
>>>>>>>>> should not receive anything. As I read it, even if it receives it, it
>>>>>>>>> should not forward the frame to L2 as it is in a different subnet. Are
>>>>>>>>> you able to read it using tcpdump on L1?
>>>>>>>>
>>>>>>>> I ran "tcpdump -i eth0" in L1. It didn't capture any of the packets
>>>>>>>> that were directed at L2 even though L2 was able to receive them.
>>>>>>>> Similarly, it didn't capture any packets that were sent from L2 to
>>>>>>>> L0. This is when L2 is launched with x-svq=false.
>>>>>>>> [...]
>>>>>>>> With x-svq=true, forcibly setting the LSB of n->mac to 0x57 in
>>>>>>>> receive_filter allows L2 to receive packets from L0. I added
>>>>>>>> the following line just before line 1771 [1] to check this out.
>>>>>>>>
>>>>>>>> n->mac[5] = 0x57;
>>>>>>>>
>>>>>>>
>>>>>>> That's very interesting. Let me answer all the gdb questions below and
>>>>>>> we can debug it deeper :).
>>>>>>>
>>>>>>
>>>>>> Thank you for the primer on using gdb with QEMU. I am able to debug
>>>>>> QEMU now.
>>>>>>
>>>>>>>>> Maybe we can make the scenario clearer by telling which virtio-net
>>>>>>>>> device is which with virtio_net_pci,mac=XX:... ?
>>>>>>>>>
>>>>>>>>>> However, when booting L2 with x-svq=true, n->mac is set to L1-eth0
>>>>>>>>>> (LSB = 56) in virtio_net_handle_mac() [3].
>>>>>>>>>
>>>>>>>>> Can you tell with gdb bt if this function is called from net or the
>>>>>>>>> SVQ subsystem?
>>>>>>>>
>>>>>>
>>>>>> It looks like the function is being called from net.
>>>>>>
>>>>>> (gdb) bt
>>>>>> #0  virtio_net_handle_mac (n=0x15622425e, cmd=85 'U', iov=0x555558865980, iov_cnt=1476792840) at ../hw/net/virtio-net.c:1098
>>>>>> #1  0x0000555555e5920b in virtio_net_handle_ctrl_iov (vdev=0x555558fdacd0, in_sg=0x5555580611f8, in_num=1, out_sg=0x555558061208,
>>>>>>          out_num=1) at ../hw/net/virtio-net.c:1581
>>>>>> #2  0x0000555555e593a0 in virtio_net_handle_ctrl (vdev=0x555558fdacd0, vq=0x555558fe7730) at ../hw/net/virtio-net.c:1610
>>>>>> #3  0x0000555555e9a7d8 in virtio_queue_notify_vq (vq=0x555558fe7730) at ../hw/virtio/virtio.c:2484
>>>>>> #4  0x0000555555e9dffb in virtio_queue_host_notifier_read (n=0x555558fe77a4) at ../hw/virtio/virtio.c:3869
>>>>>> #5  0x000055555620329f in aio_dispatch_handler (ctx=0x555557d9f840, node=0x7fffdca7ba80) at ../util/aio-posix.c:373
>>>>>> #6  0x000055555620346f in aio_dispatch_handlers (ctx=0x555557d9f840) at ../util/aio-posix.c:415
>>>>>> #7  0x00005555562034cb in aio_dispatch (ctx=0x555557d9f840) at ../util/aio-posix.c:425
>>>>>> #8  0x00005555562242b5 in aio_ctx_dispatch (source=0x555557d9f840, callback=0x0, user_data=0x0) at ../util/async.c:361
>>>>>> #9  0x00007ffff6d86559 in ?? () from /usr/lib/libglib-2.0.so.0
>>>>>> #10 0x00007ffff6d86858 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
>>>>>> #11 0x0000555556225bf9 in glib_pollfds_poll () at ../util/main-loop.c:287
>>>>>> #12 0x0000555556225c87 in os_host_main_loop_wait (timeout=294672) at ../util/main-loop.c:310
>>>>>> #13 0x0000555556225db6 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:589
>>>>>> #14 0x0000555555c0c1a3 in qemu_main_loop () at ../system/runstate.c:835
>>>>>> #15 0x000055555612bd8d in qemu_default_main (opaque=0x0) at ../system/main.c:48
>>>>>> #16 0x000055555612be3d in main (argc=23, argv=0x7fffffffe508) at ../system/main.c:76
>>>>>>
>>>>>> virtio_queue_notify_vq at hw/virtio/virtio.c:2484 [2] calls
>>>>>> vq->handle_output(vdev, vq). I see "handle_output" is a function
>>>>>> pointer and in this case it seems to be pointing to
>>>>>> virtio_net_handle_ctrl.
>>>>>>
>>>>>>>>>> [...]
>>>>>>>>>> With x-svq=true, I see that n->mac is set by virtio_net_handle_mac()
>>>>>>>>>> [3] when L1 receives VIRTIO_NET_CTRL_MAC_ADDR_SET. With x-svq=false,
>>>>>>>>>> virtio_net_handle_mac() doesn't seem to be getting called. I haven't
>>>>>>>>>> understood how the MAC address is set in VirtIONet when x-svq=false.
>>>>>>>>>> Understanding this might help see why n->mac has different values
>>>>>>>>>> when x-svq is false vs when it is true.
>>>>>>>>>
>>>>>>>>> Ok this makes sense, as x-svq=true is the one that receives the set
>>>>>>>>> mac message. You should see it in L0's QEMU though, both in x-svq=on
>>>>>>>>> and x-svq=off scenarios. Can you check it?
>>>>>>>>
>>>>>>>> L0's QEMU seems to be receiving the "set mac" message only when L1
>>>>>>>> is launched with x-svq=true. With x-svq=off, I don't see any call
>>>>>>>> to virtio_net_handle_mac with cmd == VIRTIO_NET_CTRL_MAC_ADDR_SET
>>>>>>>> in L0.
>>>>>>>>
>>>>>>>
>>>>>>> Ok this is interesting. Let's disable control virtqueue to start with
>>>>>>> something simpler:
>>>>>>> device virtio-net-pci,netdev=net0,ctrl_vq=off,...
>>>>>>>
>>>>>>> QEMU will start complaining about features that depend on ctrl_vq,
>>>>>>> like ctrl_rx. Let's disable all of them and check this new scenario.
>>>>>>>
>>>>>>
>>>>>> I am still investigating this part. I set ctrl_vq=off and ctrl_rx=off.
>>>>>> I didn't get any errors as such about features that depend on ctrl_vq.
>>>>>> However, I did notice that after booting L2 (x-svq=true as well as
>>>>>> x-svq=false), no eth0 device was created. There was only a "lo" interface
>>>>>> in L2. An eth0 interface is present only when L1 (L0 QEMU) is booted
>>>>>> with ctrl_vq=on and ctrl_rx=on.
>>>>>>
>>>>>
>>>>> Any error messages on the nested guest's dmesg?
>>>>
>>>> Oh, yes, there were error messages in the output of dmesg related to
>>>> ctrl_vq. After adding the following args, there were no error messages
>>>> in dmesg.
>>>>
>>>> -device virtio-net-pci,ctrl_vq=off,ctrl_rx=off,ctrl_vlan=off,ctrl_mac_addr=off
>>>>
>>>> I see that the eth0 interface is also created. I am able to ping L0
>>>> from L2 and vice versa as well (even with x-svq=true). This is because
>>>> n->promisc is set when these features are disabled and receive_filter() [1]
>>>> always returns 1.
>>>>
>>>>> Is it fixed when you set the same mac address on L0
>>>>> virtio-net-pci and L1's?
>>>>>
>>>>
>>>> I didn't have to set the same mac address in this case since promiscuous
>>>> mode seems to be getting enabled which allows pinging to work.
>>>>
>>>> There is another concept that I am a little confused about. In the case
>>>> where L2 is booted with x-svq=false (and all ctrl features such as ctrl_vq,
>>>> ctrl_rx, etc. are on), I am able to ping L0 from L2. When tracing
>>>> receive_filter() in L0-QEMU, I see the values of n->mac and the destination
>>>> mac address in the ICMP packet match [2].
>>>>
>>>
>>> SVQ makes an effort to set the mac address at the beginning of
>>> operation. The L0 interpret it as "filter out all MACs except this
>>> one". But SVQ cannot set the mac if ctrl_mac_addr=off, so the nic
>>> receives all packets and the guest kernel needs to filter out by
>>> itself.
>>>
>>>> I haven't understood what n->mac refers to over here. MAC addresses are
>>>> globally unique and so the mac address of the device in L1 should be
>>>> different from that in L2.
>>>
>>> With vDPA, they should be the same device even if they are declared in
>>> different cmdlines or layers of virtualizations. If it were a physical
>>> NIC, QEMU should declare the MAC of the physical NIC too.
>>
>> Understood. I guess the issue with x-svq=true is that the MAC address
>> set in L0-QEMU's n->mac is different from the device in L2. That's why
>> the packets get filtered out with x-svq=true but pinging works with
>> x-svq=false.
>>
> 
> Right!
> 
> 
>>> There is a thread in QEMU maul list where how QEMU should influence
>>> the control plane is discussed, and maybe it would be easier if QEMU
>>> just checks the device's MAC and ignores cmdline. But then, that
>>> behavior would be surprising for the rest of vhosts like vhost-kernel.
>>> Or just emit a warning if the MAC is different than the one that the
>>> device reports.
>>>
>>
>> Got it.
>>
>>>> But I see L0-QEMU's n->mac is set to the mac
>>>> address of the device in L2 (allowing receive_filter to accept the packet).
>>>>
>>>
>>> That's interesting, can you check further what does receive_filter and
>>> virtio_net_receive_rcu do with gdb? As long as virtio_net_receive_rcu
>>> flushes the packet on the receive queue, SVQ should receive it.
>>>
>> The control flow irrespective of the value of x-svq is the same up till
>> the MAC address comparison in receive_filter() [1]. For x-svq=true,
>> the equality check between n->mac and the packet's destination MAC address
>> fails and the packet is filtered out. It is not flushed to the receive
>> queue. With x-svq=false, this is not the case.
>>
>> On 2/4/25 11:45 PM, Eugenio Perez Martin wrote:
>>> PS: Please note that you can check packed_vq SVQ implementation
>>> already without CVQ, as these features are totally orthogonal :).
>>>
>>
>> Right. Now that I can ping with the ctrl features turned off, I think
>> this should take precedence. There's another issue specific to the
>> packed virtqueue case. It causes the kernel to crash. I have been
>> investigating this and the situation here looks very similar to what's
>> explained in Jason Wang's mail [2]. My plan of action is to apply his
>> changes in L2's kernel and check if that resolves the problem.
>>
>> The details of the crash can be found in this mail [3].
>>
> 
> If you're testing this series without changes, I think that is caused
> by not implementing the packed version of vhost_svq_get_buf.
> 
> https://lists.nongnu.org/archive/html/qemu-devel/2024-12/msg01902.html
> 

Oh, apologies, I think I had misunderstood your response in the linked mail.
Until now, I thought they were unrelated. In that case, I'll implement the
packed version of vhost_svq_get_buf. Hopefully that fixes it :).

Thanks,
Sahil