Re: [RFC v3 3/3] vhost: Allocate memory for packed vring

[Sahil <icegambit91@gmail.com>]

Hi,

On Thursday, September 12, 2024 3:24:27 PM GMT+5:30 Eugenio Perez Martin wrote:
> On Wed, Sep 11, 2024 at 9:36 PM Sahil <icegambit91@gmail.com> wrote:
> > Hi,
> > 
> > On Monday, September 9, 2024 6:04:45 PM GMT+5:30 Eugenio Perez Martin wrote:
> > > On Sun, Sep 8, 2024 at 9:47 PM Sahil <icegambit91@gmail.com> wrote:
> > > > On Friday, August 30, 2024 4:18:31 PM GMT+5:30 Eugenio Perez Martin wrote:
> > > > > On Fri, Aug 30, 2024 at 12:20 PM Sahil <icegambit91@gmail.com>
> > > > > wrote:
> > > > > [...]
> > > > > vdpa_sim does not support packed vq at the moment. You need to build
> > > > > the use case #3 of the second part of that blog [1]. It's good that
> > > > > you build the vdpa_sim earlier as it is a simpler setup.
> > > > > 
> > > > > If you have problems with the vp_vdpa environment please let me know
> > > > > so we can find alternative setups.
> > > > 
> > > > Thank you for the clarification. I tried setting up the vp_vdpa
> > > > environment (scenario 3) but I ended up running into a problem
> > > > in the L1 VM.
> > > > 
> > > > I verified that nesting is enabled in KVM (L0):
> > > > 
> > > > $ grep -oE "(vmx|svm)" /proc/cpuinfo | sort | uniq
> > > > vmx
> > > > 
> > > > $ cat /sys/module/kvm_intel/parameters/nested
> > > > Y
> > > > 
> > > > There are no issues when booting L1. I start the VM by running:
> > > > 
> > > > $ sudo ./qemu/build/qemu-system-x86_64 \
> > > > -enable-kvm \
> > > > -drive
> > > > file=//home/ig91/fedora_qemu_test_vm/L1.qcow2,media=disk,if=virtio
> > > > \
> > > > -net nic,model=virtio \
> > > > -net user,hostfwd=tcp::2222-:22 \
> > > > -device intel-iommu,snoop-control=on \
> > > > -device
> > > > virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off,iommu_
> > > > pla
> > > > tform=on,event_idx=off,packed=on,bus=pcie.0,addr=0x4 \ -netdev
> > > > tap,id=net0,script=no,downscript=no \
> > > > -nographic \
> > > > -m 2G \
> > > > -smp 2 \
> > > > -M q35 \
> > > > -cpu host \
> > > > 2>&1 | tee vm.log
> > > > 
> > > > Kernel version in L1:
> > > > 
> > > > # uname -a
> > > > Linux fedora 6.8.5-201.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 11
> > > > 18:25:26 UTC 2024 x86_64 GNU/Linux
> > > 
> > > Did you run the kernels with the arguments "iommu=pt intel_iommu=on"?
> > > You can print them with cat /proc/cmdline.
> > 
> > I missed this while setting up the environment. After setting the kernel
> > params I managed to move past this issue but my environment in virtualbox
> > was very unstable and it kept crashing.
> 
> I've no experience with virtualbox+vdpa, sorry :). Why not use QEMU also for
> L1?

No, I was using virtualbox for L0. I wasn't able to create L1.qcow2 on my
host OS (Arch Linux). "Virt-sysprep" was giving me "resolve.conf is a
dangling symlink" issues. I had a Fedora VM set up on virtualbox and I
managed to create L1.qcow2 and L2.qcow2 there.

So my environment looked something like this:

Host OS (Bare metal) -> L0 (Fedora on virtualbox) -> L1 (QEMU) -> L2 (QEMU)

I learnt that shared folders can be set up between the host and virtualbox
and so I managed to move L1.qcow2 from the Fedora VM to my host OS.

So now my environment looks like this:

L0 (Arch Linux on bare metal) -> L1 (QEMU) -> L2 (QEMU)

> > I managed to get L1 to run on my host OS, so scenario 3 is now up and
> > running. However, the packed bit seems to be disabled in this scenario
> > too.
> > 
> > L0 (host machine) specs:
> > 
> > - kernel version:
> >   6.6.46-1-lts
> > 
> > - QEMU version:
> >   9.0.50 (v8.2.0-5536-g16514611dc)
> > 
> > - vDPA version:
> >   iproute2-6.10.0
> > 
> > L1 specs:
> > 
> > - kernel version:
> >   6.8.5-201.fc39.x86_64
> > 
> > - QEMU version:
> >   9.0.91
> > 
> > - vDPA version:
> >   iproute2-6.10.0
> > 
> > L2 specs:
> > - kernel version
> > 
> >   6.8.7-200.fc39.x86_64
> > 
> > I followed the following steps to set up scenario 3:
> > 
> > ==== In L0 ====
> > 
> > $ grep -oE "(vmx|svm)" /proc/cpuinfo | sort | uniq
> > vmx
> > 
> > $ cat /sys/module/kvm_intel/parameters/nested
> > Y
> > 
> > $ sudo ./qemu/build/qemu-system-x86_64 \
> > -enable-kvm \
> > -drive
> > file=//home/valdaarhun/valdaarhun/qcow2_img/L1.qcow2,media=disk,if=virtio
> > \ -net nic,model=virtio \
> > -net user,hostfwd=tcp::2222-:22 \
> > -device intel-iommu,snoop-control=on \
> > -device
> > virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off,iommu_pla
> > tform=on,event_idx=off,packed=on,bus=pcie.0,addr=0x4 \ -netdev
> > tap,id=net0,script=no,downscript=no \
> > -nographic \
> > -m 8G \
> > -smp 4 \
> > -M q35 \
> > -cpu host \
> > 2>&1 | tee vm.log
> > 
> > ==== In L1 ====
> > 
> > I verified that the following config variables are set as decribed in the
> > blog [1].
> > 
> > CONFIG_VIRTIO_VDPA=m
> > CONFIG_VDPA=m
> > CONFIG_VP_VDPA=m
> > CONFIG_VHOST_VDPA=m
> > 
> > # modprobe vdpa
> > # modprobe vhost_vdpa
> > # modprobe vp_vdpa
> > 
> > # lsmod | grep -i vdpa
> > vp_vdpa                 20480  0
> > vhost_vdpa              32768  0
> > vhost                   65536  1 vhost_vdpa
> > vhost_iotlb             16384  2 vhost_vdpa,vhost
> > vdpa                    36864  2 vp_vdpa,vhost_vdpa
> > irqbypass               12288  2 vhost_vdpa,kvm
> > 
> > # lspci | grep -i ethernet
> > 00:04.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev
> > 01)
> > 
> > # lspci -nn | grep 00:04.0
> > 00:04.0 Ethernet controller [0200]: Red Hat, Inc. Virtio 1.0 network
> > device [1af4:1041] (rev 01)
> > 
> > # echo 0000:00:04.0 > /sys/bus/pci/drivers/virtio-pci/unbind
> > # echo 1af4 1041 > /sys/bus/pci/drivers/vp-vdpa/new_id
> > 
> > # vdpa mgmtdev show
> > 
> > pci/0000:00:04.0:
> >   supported_classes net < unknown class >
> >   max_supported_vqs 3
> >   dev_features  CSUM  GUEST_CSUM  CTRL_GUEST_OFFLOADS  MAC  GUEST_TSO4
> >   GUEST_TSO6  GUEST_ECN  GUEST_UFO  HOST_TSO4  HOST_TSO6  HOST_ECN
> >   HOST_UFO  MRG_RXBUF  STATUS  CTRL_VQ  CTRL_RX  CTRL_VLAN  CTRL_RX_EXTRA
> >   GUEST_ANNOUNCE  CTRL_MAC_ADDR  RING_INDIRECT_DESC  RING_EVENT_IDX
> >   VERSION_1  ACCESS_PLATFORM  bit_40  bit_54  bit_55  bit_56
> > 
> > # vdpa dev add name vdpa0 mgmtdev pci/0000:00:04.0
> > # vdpa dev show -jp
> > {
> > 
> >     "dev": {
> >     
> >         "vdpa0": {
> >         
> >             "type": "network",
> >             "mgmtdev": "pci/0000:00:04.0",
> >             "vendor_id": 6900,
> >             "max_vqs": 3,
> >             "max_vq_size": 256
> >         
> >         }
> >     
> >     }
> > 
> > }
> > 
> > # ls -l /sys/bus/vdpa/devices/vdpa0/driver
> > lrwxrwxrwx. 1 root root 0 Sep 11 17:07 /sys/bus/vdpa/devices/vdpa0/driver
> > -> ../../../../bus/vdpa/drivers/vhost_vdpa
> > 
> > # ls -l /dev/ |grep vdpa
> > crw-------. 1 root root    239,   0 Sep 11 17:07 vhost-vdpa-0
> > 
> > # driverctl -b vdpa
> > vdpa0 vhost_vdpa [*]
> > 
> > Booting L2:
> > 
> > # ./qemu/build/qemu-system-x86_64 \
> > -nographic \
> > -m 4G \
> > -enable-kvm \
> > -M q35 \
> > -drive file=//root/L2.qcow2,media=disk,if=virtio \
> > -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0 \
> > -device
> > virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,ev
> > ent_idx=off,packed=on,bus=pcie.0,addr=0x7 \ -smp 4 \
> > -cpu host \
> > 2>&1 | tee vm.log
> > 
> > ==== In L2 ====
> > 
> > # cut -c 35 /sys/devices/pci0000:00/0000:00:07.0/virtio1/features
> > 0
> > 
> > Based on what I have understood from the kernel's source, vhost-vdpa and
> > vp-vdpa both support packed vqs and v6.6.46 is more recent than the
> > minimum required kernel version. I am not sure where I am going wrong.
> > 
> > > > However, I managed to set up scenario 4 successfully
> > > > and I see that packed vq is enabled in this case.
> > > > 
> > > > # cut -c 35 /sys/devices/pci0000:00/0000:00:04.0/virtio1/features
> > > > 1
> > 
> > So far scenario 4 is the only scenario where the virtio-net device has the
> > packed feature enabled. The difference between scenarios 3 and 4 in terms
> > of the kernel modules loaded is that scenario 4 uses virtio_vdpa while
> > scenario 3 uses vdpa and vhost_vdpa.
> 
> We're pretty close then :). But I don't see anything obviously wrong here.
> 
> I guess it is the same L1 VM in both cases, isn't it?

Right, I am using the same L1.qcow2 image in both scenarios.

> The function that gets the features from vhost-vdpa in QEMU is
> hw/virtio/vhost-vdpa.c:vhost_vdpa_get_features. Can you check that it
> returns bit 34 (offset starts with 0 here)? If it returns it, can you
> keep debugging until you see what clears it?
> 
> If it comes clear, then we need to check the kernel.

Got it. I'll start debugging from here.

> If you don't see anything wrong there, we can keep debugging the
> function that handles getting the features through L2 guest PCI read,
> hw/virtio/virtio-pci.c:virtio_pci_common_read. VIRTIO_PCI_COMMON_DF
> case when proxy->dfselect==1.
> 
> Let me know what you find.

I'll do that. Thank you for the suggestions.

Thanks,
Sahil