[RFC v4 0/5] Add packed virtqueue to shadow virtqueue

[Sahil Siddiq <icegambit91@gmail.com>]

Hi,

There are two issues that I found while trying to test
my changes. I thought I would send the patch series
as well in case that helps in troubleshooting. I haven't
been able to find an issue in the implementation yet.
Maybe I am missing something.

I have been following the "Hands on vDPA: what do you do
when you ain't got the hardware v2 (Part 2)" [1] blog to
test my changes. To boot the L1 VM, I ran:

sudo ./qemu/build/qemu-system-x86_64 \
-enable-kvm \
-drive file=//home/valdaarhun/valdaarhun/qcow2_img/L1.qcow2,media=disk,if=virtio \
-net nic,model=virtio \
-net user,hostfwd=tcp::2222-:22 \
-device intel-iommu,snoop-control=on \
-device virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,guest_uso4=off,guest_uso6=off,host_uso=off,guest_announce=off,ctrl_vq=on,ctrl_rx=on,packed=on,event_idx=off,bus=pcie.0,addr=0x4 \
-netdev tap,id=net0,script=no,downscript=no \
-nographic \
-m 8G \
-smp 4 \
-M q35 \
-cpu host 2>&1 | tee vm.log

Without "guest_uso4=off,guest_uso6=off,host_uso=off,
guest_announce=off" in "-device virtio-net-pci", QEMU
throws "vdpa svq does not work with features" [2] when
trying to boot L2.

The enums added in commit #2 in this series is new and
wasn't in the earlier versions of the series. Without
this change, x-svq=true throws "SVQ invalid device feature
flags" [3] and x-svq is consequently disabled.

The first issue is related to running traffic in L2
with vhost-vdpa.

In L0:

$ ip addr add 111.1.1.1/24 dev tap0
$ ip link set tap0 up
$ ip addr show tap0
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether d2:6d:b9:61:e1:9a brd ff:ff:ff:ff:ff:ff
    inet 111.1.1.1/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::d06d:b9ff:fe61:e19a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

I am able to run traffic in L2 when booting without
x-svq.

In L1:

$ ./qemu/build/qemu-system-x86_64 \
-nographic \
-m 4G \
-enable-kvm \
-M q35 \
-drive file=//root/L2.qcow2,media=disk,if=virtio \
-netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0 \
-device virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,ctrl_vq=on,ctrl_rx=on,event_idx=off,bus=pcie.0,addr=0x7 \
-smp 4 \
-cpu host \
2>&1 | tee vm.log

In L2:

# ip addr add 111.1.1.2/24 dev eth0
# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 111.1.1.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9877:de30:5f17:35f9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

# ip route
111.1.1.0/24 dev eth0 proto kernel scope link src 111.1.1.2

# ping 111.1.1.1 -w3
PING 111.1.1.1 (111.1.1.1) 56(84) bytes of data.
64 bytes from 111.1.1.1: icmp_seq=1 ttl=64 time=0.407 ms
64 bytes from 111.1.1.1: icmp_seq=2 ttl=64 time=0.671 ms
64 bytes from 111.1.1.1: icmp_seq=3 ttl=64 time=0.291 ms

--- 111.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms
rtt min/avg/max/mdev = 0.291/0.456/0.671/0.159 ms


But if I boot L2 with x-svq=true as shown below, I am unable
to ping the host machine.

$ ./qemu/build/qemu-system-x86_64 \
-nographic \
-m 4G \
-enable-kvm \
-M q35 \
-drive file=//root/L2.qcow2,media=disk,if=virtio \
-netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,x-svq=true,id=vhost-vdpa0 \
-device virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,ctrl_vq=on,ctrl_rx=on,event_idx=off,bus=pcie.0,addr=0x7 \
-smp 4 \
-cpu host \
2>&1 | tee vm.log

In L2:

# ip addr add 111.1.1.2/24 dev eth0
# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
    altname enp0s7
    inet 111.1.1.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9877:de30:5f17:35f9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

# ip route
111.1.1.0/24 dev eth0 proto kernel scope link src 111.1.1.2

# ping 111.1.1.1 -w10
PING 111.1.1.1 (111.1.1.1) 56(84) bytes of data.
From 111.1.1.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: No route to host
From 111.1.1.2 icmp_seq=2 Destination Host Unreachable
From 111.1.1.2 icmp_seq=3 Destination Host Unreachable

--- 111.1.1.1 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2076ms
pipe 3

The other issue is related to booting L2 with "x-svq=true"
and "packed=on".

In L1:

$ ./qemu/build/qemu-system-x86_64 \
-nographic \
-m 4G \
-enable-kvm \
-M q35 \
-drive file=//root/L2.qcow2,media=disk,if=virtio \
-netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0,x-svq=true \
-device virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,guest_uso4=off,guest_uso6=off,host_uso=off,guest_announce=off,ctrl_vq=on,ctrl_rx=on,event_idx=off,packed=on,bus=pcie.0,addr=0x7 \
-smp 4 \
-cpu host \
2>&1 | tee vm.log

The kernel throws "virtio_net virtio1: output.0:id 0 is not
a head!" [4].

Here's part of the trace:

[...]
[  945.370085] watchdog: BUG: soft lockup - CPU#2 stuck for 863s! [NetworkManager:795]
[  945.372467] Modules linked in: rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core intel_vsec pmt_g
[  945.387413] CPU: 2 PID: 795 Comm: NetworkManager Tainted: G             L     6.8.7-200.fc39.x86_64 #1
[  945.390685] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[  945.394256] RIP: 0010:virtnet_poll+0xd8/0x5c0 [virtio_net]
[  945.395998] Code: c0 74 5c 65 8b 05 24 37 8b 3f 41 89 86 c4 00 00 00 80 bb 40 04 00 00 00 75 32 48 8b 3b e8 00 00 28 c7 48 89 df be8
[  945.401465] RSP: 0018:ffffabaec0134e48 EFLAGS: 00000246
[  945.403362] RAX: ffff9bf904432000 RBX: ffff9bf9085b1800 RCX: 00000000ffff0001
[  945.405447] RDX: 0000000000008080 RSI: 0000000000000001 RDI: ffff9bf9085b1800
[  945.408361] RBP: ffff9bf9085b0808 R08: 0000000000000001 R09: ffffabaec0134ba8
[  945.410828] R10: ffffabaec0134ba0 R11: 0000000000000003 R12: ffff9bf905a34ac0
[  945.413272] R13: 0000000000000040 R14: ffff9bf905a34a00 R15: ffff9bf9085b0800
[  945.415180] FS:  00007fa81f0f1540(0000) GS:ffff9bf97bd00000(0000) knlGS:0000000000000000
[  945.418177] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  945.419415] CR2: 000055614ba8dc48 CR3: 0000000102b42006 CR4: 0000000000770ef0
[  945.423312] PKRU: 55555554
[  945.424238] Call Trace:
[  945.424238]  <IRQ>
[  945.426236]  ? watchdog_timer_fn+0x1e6/0x270
[  945.427304]  ? __pfx_watchdog_timer_fn+0x10/0x10
[  945.428239]  ? __hrtimer_run_queues+0x10f/0x2b0
[  945.431304]  ? hrtimer_interrupt+0xf8/0x230
[  945.432236]  ? __sysvec_apic_timer_interrupt+0x4d/0x140
[  945.434187]  ? sysvec_apic_timer_interrupt+0x39/0x90
[  945.436306]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  945.438199]  ? virtnet_poll+0xd8/0x5c0 [virtio_net]
[  945.438199]  ? virtnet_poll+0xd0/0x5c0 [virtio_net]
[  945.440197]  ? handle_irq_event+0x50/0x80
[  945.442415]  ? sched_clock_cpu+0x5e/0x190
[  945.444563]  ? irqtime_account_irq+0x40/0xc0
[  945.446191]  __napi_poll+0x28/0x1c0
[  945.446191]  net_rx_action+0x2a4/0x380
[  945.448851]  ? _raw_spin_unlock_irqrestore+0xe/0x40
[  945.450209]  ? note_gp_changes+0x6c/0x80
[  945.452252]  __do_softirq+0xc9/0x2c8
[  945.453579]  do_softirq.part.0+0x3d/0x60
[  945.454188]  </IRQ>
[  945.454188]  <TASK>
[  945.456175]  __local_bh_enable_ip+0x68/0x70
[  945.458373]  virtnet_open+0xdc/0x310 [virtio_net]
[  945.460005]  __dev_open+0xfa/0x1b0
[  945.461310]  __dev_change_flags+0x1dc/0x250
[  945.462800]  dev_change_flags+0x26/0x70
[  945.464190]  do_setlink+0x375/0x12d0
[...]

I am not sure if this issue is similar to the one
described in this patch (race between channels
setting and refill) [5]. As described in the patch,
I see drivers/net/virtio_net:virtnet_open invoke
try_fill_recv() and schedule_delayed_work() [6]. I
am unfamiliar with this and so I am not sure how to
progress.

Maybe I can try disabling napi and checking it out
if that is possible. Would this be a good next step
to troubleshoot the kernel crash?

Thanks,
Sahil

Changes v3 -> v4:
- Split commit #1 of v3 into commit #1 and #2 in
  this series [7].
- Commit #3 is commit #2 of v3.
- Commit #4 is based on commit #3 of v3.
- Commit #5 was sent as an individual patch [8].
- vhost-shadow-virtqueue.c
  (vhost_svq_valid_features): Add enums.
  (vhost_svq_memory_packed): Remove function.
  (vhost_svq_driver_area_size,vhost_svq_descriptor_area_size): Decouple functions.
  (vhost_svq_device_area_size): Rewrite function.
  (vhost_svq_start): Simplify implementation.
  (vhost_svq_stop): Unconditionally munmap().
- vhost-shadow-virtqueue.h: New function declaration.
- vhost-vdpa.c
  (vhost_vdpa_svq_unmap_rings): Call vhost_vdpa_svq_unmap_ring().
  (vhost_vdpa_svq_map_rings): New mappings.
  (vhost_vdpa_svq_setup): Add comment.

[1] https://www.redhat.com/en/blog/hands-vdpa-what-do-you-do-when-you-aint-got-hardware-part-2
[2] https://gitlab.com/qemu-project/qemu/-/blob/master/net/vhost-vdpa.c#L167
[3] https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost-shadow-virtqueue.c#L58
[4] https://github.com/torvalds/linux/blob/master/drivers/virtio/virtio_ring.c#L1763
[5] https://lkml.iu.edu/hypermail/linux/kernel/1307.0/01455.html
[6] https://github.com/torvalds/linux/blob/master/drivers/net/virtio_net.c#L3104
[7] https://lists.nongnu.org/archive/html/qemu-devel/2024-08/msg01148.html
[8] https://lists.nongnu.org/archive/html/qemu-devel/2024-11/msg00598.html

Sahil Siddiq (5):
  vhost: Refactor vhost_svq_add_split
  vhost: Write descriptors to packed svq
  vhost: Data structure changes to support packed vqs
  vdpa: Allocate memory for svq and map them to vdpa
  vdpa: Support setting vring_base for packed svq

 hw/virtio/vhost-shadow-virtqueue.c | 222 +++++++++++++++++++----------
 hw/virtio/vhost-shadow-virtqueue.h |  70 ++++++---
 hw/virtio/vhost-vdpa.c             |  47 +++++-
 3 files changed, 237 insertions(+), 102 deletions(-)

-- 
2.47.0