[DPDK/vhost/virtio Bug 1394] vq_assert_lock__ fail in vhost_user_set_vring_addr during live migration with HW vDPA

[DPDK/vhost/virtio Bug 1394] vq_assert_lock__ fail in vhost_user_set_vring_addr during live migration with HW vDPA

[bugzilla]

[-- Attachment #1: Type: text/plain, Size: 6735 bytes --]

https://bugs.dpdk.org/show_bug.cgi?id=1394

            Bug ID: 1394
           Summary: vq_assert_lock__ fail in vhost_user_set_vring_addr
                    during live migration with HW vDPA
           Product: DPDK
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: critical
          Priority: Normal
         Component: vhost/virtio
          Assignee: dev@dpdk.org
          Reporter: yajunw@nvidia.com
  Target Milestone: ---

DPDK version: v24.03-rc1

1. boot vDPA and qemu(8.1)
build/examples/dpdk-vdpa -a 5e:00.2,class=vdpa --file-prefix vf0
--log-level=.,info -- --client --iface /tmp/vfe-net 

<qemu:arg value='-chardev'/>
<qemu:arg value='socket,id=charnet1,path=/tmp/vfe-net0,server'/>
<qemu:arg value='-netdev'/>
<qemu:arg value='vhost-user,chardev=charnet1,queues=2,id=hostnet1'/>
<qemu:arg value='-device'/>
<qemu:arg
value='virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=00:00:00:00:33:00,bus=pci.0,addr=0x8,page-per-vq=on'/>

2. live migration VM to another server
sudo virsh migrate --verbose --live --persistent gen-l-vrt-295-005-CentOS-7.4
qemu+ssh://gen-l-vrt-294/system  --unsafe

3. dpdk crash

After device configured, vhost_user_lock_all_queue_pairs won't be called. The 
vq_assert_lock__ failed in vhost_user_set_vring_addr for vDPA case.

related to commit:
commit 741dc052eaf9459cc576b0d87e96a40069485c32 (HEAD)
Author: David Marchand david.marchand@redhat.com
Date:   Tue Dec 5 10:45:34 2023 +0100

    vhost: annotate virtqueue access checks

    Modifying vq->access_ok should be done with a write lock taken.
    Annotate vring_translate() and vring_invalidate().



new port /tmp/vfe-net0, device : 5e:00.2
mlx5_vdpa: MTU cannot be set on device 5e:00.2.
mlx5_vdpa: Region 0: HVA 0x7fff00000000, GPA 0x0, size 0xc0000000.
mlx5_vdpa: Region 1: HVA 0x7ffdc0000000, GPA 0x100000000, size 0x140000000.
mlx5_vdpa: Indirect mkey mode is KLM Fixed Buffer Size.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 0.
mlx5_vdpa: Virtq 0 notifier state is enabled.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 1.
mlx5_vdpa: Virtq 1 notifier state is enabled.
[New Thread 0x7fffe7b9b400 (LWP 962699)]
mlx5_vdpa: vDPA device 0 was configured.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 0
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 1
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 2
mlx5_vdpa: Update virtq 2 status disable -> enable.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 2.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 3
mlx5_vdpa: Update virtq 3 status disable -> enable.
mlx5_vdpa: Virtq 2 notifier state is enabled.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 3.
mlx5_vdpa: Virtq 3 notifier state is enabled.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_LOG_BASE
VHOST_CONFIG: (/tmp/vfe-net0) log mmap size: 294912, offset: 0
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: (/tmp/vfe-net0) negotiated Virtio features: 0x144601803
mlx5_vdpa: mlx5 vdpa: enabling dirty logging...
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_GET_STATUS
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ADDR
EAL: PANIC in vq_assert_lock__():
VHOST_CONFIG: (/tmp/vfe-net0) vhost_user_set_vring_addr() called without access
lock taken.
0: /images/vdpa/dpdk/build/examples/dpdk-vdpa (rte_dump_stack+0x1f) [aadfca]
1: /images/vdpa/dpdk/build/examples/dpdk-vdpa (__rte_panic+0xe2) [a8032b]
2: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x3bac83) [7bac83]
3: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x3bd0ef) [7bd0ef]
4: /images/vdpa/dpdk/build/examples/dpdk-vdpa (vhost_user_msg_handler+0x508)
[7c264f]
5: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x36b797) [76b797]
6: /images/vdpa/dpdk/build/examples/dpdk-vdpa (fdset_event_dispatch+0x1cd)
[769793]
7: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x6970ca) [a970ca]
8: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x6af41c) [aaf41c]
9: /lib64/libpthread.so.0 (7ffff6426000+0x814a) [7ffff642e14a]
10: /lib64/libc.so.6 (clone+0x43) [7ffff615ddc3]

Thread 29 "dpdk-vhost-evt" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe839c400 (LWP 962487)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50        return ret;
Missing separate debuginfos, use: yum debuginfo-install
elfutils-libelf-0.182-3.el8.x86_64 libgcc-8.4.1-1.el8.x86_64
libibverbs-2307mlnx47-1.2310007.x86_64 libnl3-3.5.0-1.el8.x86_64
libpcap-1.9.1-5.el8.x86_64 numactl-libs-2.0.12-11.el8.x86_64
openssl-libs-1.1.1k-5.el8_5.x86_64 zlib-1.2.11-17.el8.x86_64
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff6082db5 in __GI_abort () at abort.c:79
#2  0x0000000000a80330 in __rte_panic (funcname=0x3461330 <__func__.33032>
"vq_assert_lock__", 
    format=0x345eb90 "VHOST_CONFIG: (%s) %s() called without access lock
taken.\n%.0s") at ../lib/eal/common/eal_common_debug.c:26
#3  0x00000000007bac83 in vq_assert_lock__ (dev=0x17ffc2500, vq=0x17ffa2280,
func=0x3461370 <__func__.33676> "vhost_user_set_vring_addr")
    at ../lib/vhost/vhost.h:547
#4  0x00000000007bd0ef in vhost_user_set_vring_addr (pdev=0x7fffe8399270,
ctx=0x7fffe8398fc0, main_fd=115) at ../lib/vhost/vhost_user.c:930
#5  0x00000000007c264f in vhost_user_msg_handler (vid=0, fd=115) at
../lib/vhost/vhost_user.c:3197
#6  0x000000000076b797 in vhost_user_read_cb (connfd=115, dat=0x7fffe0000dd0,
remove=0x7fffe8399354) at ../lib/vhost/socket.c:318
#7  0x0000000000769793 in fdset_event_dispatch (arg=0x3cb3940
<vhost_user+8192>) at ../lib/vhost/fd_man.c:282
#8  0x0000000000a970ca in control_thread_start (arg=0x9a5ed30) at
../lib/eal/common/eal_common_thread.c:282
#9  0x0000000000aaf41c in thread_start_wrapper (arg=0x7fffffffe090) at
../lib/eal/unix/rte_thread.c:114
#10 0x00007ffff642e14a in start_thread (arg=<optimized out>) at
pthread_create.c:479
#11 0x00007ffff615ddc3 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #2: Type: text/html, Size: 8799 bytes --]