[Qemu-devel] netfilter crash with device-add e1000e

[Qemu-devel] netfilter crash with device-add e1000e

[Dr. David Alan Gilbert]

Hi,
  I've got a 25% repeatable crash doing a 'device-add e1000e'
in the netfilter code:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  qemu_netfilter_receive (nf=0x76656474656e, direction=NET_FILTER_DIRECTION_TX, sender=0x563b5c78e130, flags=0, iov=0x563b5c78e7a0, iovcnt=4, sent_cb=0x0) at /home/dgilbert/git/hmp/net/filter.c:34
34	    if (qemu_can_skip_netfilter(nf)) {
[Current thread is 1 (Thread 0x7f9657cfc700 (LWP 21410))]
Missing separate debuginfos, use: dnf debuginfo-install SDL-1.2.15-29.fc27.x86_64 at-spi2-atk-2.26.1-1.fc27.x86_64 at-spi2-core-2.26.2-1.fc27.x86_64 atk-2.26.1-1.fc27.x86_64 bluez-libs-5.47-2.fc27.x86_64 brlapi-0.6.6-8.fc27.x86_64 bzip2-libs-1.0.6-24.fc27.x86_64 cairo-1.15.8-1.fc27.x86_64 cairo-gobject-1.15.8-1.fc27.x86_64 celt051-0.5.1.3-14.fc27.x86_64 cyrus-sasl-lib-2.1.26-34.fc27.x86_64 dbus-libs-1.12.0-1.fc27.x86_64 expat-2.2.5-1.fc27.x86_64 fontconfig-2.12.6-4.fc27.x86_64 freetype-2.8-6.fc27.x86_64 gdk-pixbuf2-2.36.11-1.fc27.x86_64 glib2-2.54.2-1.fc27.x86_64 glibc-2.26-16.fc27.x86_64 glusterfs-api-3.12.3-1.fc27.x86_64 glusterfs-libs-3.12.3-1.fc27.x86_64 gmp-6.1.2-6.fc27.x86_64 gnutls-3.5.16-3.fc27.x86_64 graphite2-1.3.10-3.fc27.x86_64 gstreamer1-1.12.3-1.fc27.x86_64 gstreamer1-plugins-base-1.12.3-1.fc27.x86_64 gtk3-3.22.26-1.fc27.x86_64 gvfs-client-1.34.1-1.fc27.x86_64 harfbuzz-1.4.8-1.fc27.x86_64 keyutils-libs-1.5.10-3.fc27.x86_64 krb5-libs-1.15.2-4.fc27.x86_64 libX11-1.6.5-4.fc27.x86_64 libXau-1.0.8-9.fc27.x86_64 libXcomposite-0.4.4-11.fc27.x86_64 libXcursor-1.1.14-10.fc27.x86_64 libXdamage-1.1.4-11.fc27.x86_64 libXext-1.3.3-7.fc27.x86_64 libXfixes-5.0.3-4.fc27.x86_64 libXi-1.7.9-4.fc27.x86_64 libXinerama-1.1.3-9.fc27.x86_64 libXrandr-1.5.1-4.fc27.x86_64 libXrender-0.9.10-4.fc27.x86_64 libXtst-1.2.3-4.fc27.x86_64 libacl-2.2.52-18.fc27.x86_64 libaio-0.3.110-9.fc27.x86_64 libattr-2.4.47-21.fc27.x86_64 libblkid-2.30.2-1.fc27.x86_64 libcacard-2.5.3-3.fc27.x86_64 libcom_err-1.43.5-2.fc27.x86_64 libcrypt-nss-2.26-16.fc27.x86_64 libcurl-7.55.1-7.fc27.x86_64 libdatrie-0.2.9-6.fc27.x86_64 libdrm-2.4.88-1.fc27.x86_64 libepoxy-1.4.3-3.fc27.x86_64 libfdt-1.4.5-1.fc27.x86_64 libffi-3.1-14.fc27.x86_64 libgcc-7.2.1-2.fc27.x86_64 libgcrypt-1.8.1-1.fc27.x86_64 libgpg-error-1.27-3.fc27.x86_64 libibverbs-14-4.fc27.x86_64 libidn2-2.0.4-1.fc27.x86_64 libiscsi-1.15.0-5.fc27.x86_64 libjpeg-turbo-1.5.1-4.fc27.x86_64 libmount-2.30.2-1.fc27.x86_64 libnfs-1.9.8-5.fc27.x86_64 libnghttp2-1.25.0-1.fc27.x86_64 libnl3-3.4.0-1.fc27.x86_64 libpng-1.6.31-1.fc27.x86_64 libpsl-0.18.0-1.fc27.x86_64 librados2-12.2.1-1.fc27.x86_64 librbd1-12.2.1-1.fc27.x86_64 librdmacm-14-4.fc27.x86_64 libseccomp-2.3.2-5.fc27.x86_64 libselinux-2.7-2.fc27.x86_64 libssh2-1.8.0-5.fc27.x86_64 libstdc++-7.2.1-2.fc27.x86_64 libtasn1-4.12-3.fc27.x86_64 libthai-0.1.25-4.fc27.x86_64 libunistring-0.9.7-3.fc27.x86_64 libusbx-1.0.21-4.fc27.x86_64 libuuid-2.30.2-1.fc27.x86_64 libwayland-client-1.14.0-1.fc27.x86_64 libwayland-cursor-1.14.0-1.fc27.x86_64 libwayland-server-1.14.0-1.fc27.x86_64 libxcb-1.12-5.fc27.x86_64 libxkbcommon-0.7.1-5.fc27.x86_64 lttng-ust-2.10.0-2.fc27.x86_64 lz4-libs-1.8.0-1.fc27.x86_64 lzo-2.08-11.fc27.x86_64 mesa-libgbm-17.2.4-2.fc27.x86_64 mesa-libwayland-egl-17.2.4-2.fc27.x86_64 ncurses-libs-6.0-13.20170722.fc27.x86_64 nettle-3.4-1.fc27.x86_64 nspr-4.17.0-1.fc27.x86_64 nss-3.34.0-1.0.fc27.x86_64 nss-softokn-freebl-3.34.0-1.0.fc27.x86_64 nss-util-3.34.0-1.0.fc27.x86_64 numactl-libs-2.0.11-5.fc27.x86_64 openldap-2.4.45-3.fc27.x86_64 openssl-libs-1.1.0g-1.fc27.x86_64 opus-1.2.1-3.fc27.x86_64 orc-0.4.27-3.fc27.x86_64 p11-kit-0.23.9-2.fc27.x86_64 pango-1.40.14-1.fc27.x86_64 pcre-8.41-3.fc27.x86_64 pcre2-10.30-2.fc27.x86_64 pixman-0.34.0-4.fc27.x86_64 spice-server-0.14.0-1.fc27.x86_64 systemd-libs-234-9.fc27.x86_64 usbredir-0.7.1-5.fc27.x86_64 userspace-rcu-0.10.0-3.fc27.x86_64 vte3-0.36.5-5.fc27.x86_64 xen-libs-4.9.1-1.fc27.x86_64 xz-libs-5.2.3-4.fc27.x86_64 zlib-1.2.11-4.fc27.x86_64
(gdb) where
#0  0x0000563b5aa3bac0 in qemu_netfilter_receive (nf=0x76656474656e, direction=NET_FILTER_DIRECTION_TX, sender=0x563b5c78e130, flags=0, iov=0x563b5c78e7a0, iovcnt=4, sent_cb=0x0) at /home/dgilbert/git/hmp/net/filter.c:34
#1  0x0000563b5aa31cef in filter_receive_iov (nc=0x563b5c78e130, nc=0x563b5c78e130, sent_cb=0x0, iovcnt=4, iov=0x563b5c78e7a0, flags=0, sender=0x563b5c78e130, direction=NET_FILTER_DIRECTION_TX) at /home/dgilbert/git/hmp/net/net.c:571
#2  0x0000563b5aa31cef in qemu_sendv_packet_async (sender=0x563b5c78e130, iov=0x563b5c78e7a0, iovcnt=4, sent_cb=0x0) at /home/dgilbert/git/hmp/net/net.c:768
#3  0x0000563b5a97ea18 in net_tx_pkt_sendv (pkt=0x563b5c867620, iov_cnt=<optimized out>, iov=<optimized out>, nc=0x563b5c78e130) at /home/dgilbert/git/hmp/hw/net/net_tx_pkt.c:546
#4  0x0000563b5a97ea18 in net_tx_pkt_send (pkt=0x563b5c867620, nc=nc@entry=0x563b5c78e130) at /home/dgilbert/git/hmp/hw/net/net_tx_pkt.c:620
#5  0x0000563b5a9882c8 in e1000e_tx_pkt_send (queue_index=<optimized out>, tx=0x563b5cbe3108, core=0x563b5cbc2ea0) at /home/dgilbert/git/hmp/hw/net/e1000e_core.c:665
#6  0x0000563b5a9882c8 in e1000e_process_tx_desc (queue_index=<optimized out>, dp=0x7f9657cf9010, tx=0x563b5cbe3108, core=0x563b5cbc2ea0) at /home/dgilbert/git/hmp/hw/net/e1000e_core.c:742
#7  0x0000563b5a9882c8 in e1000e_start_xmit (core=0x563b5cbc2ea0, txr=txr@entry=0x7f9657cf9080) at /home/dgilbert/git/hmp/hw/net/e1000e_core.c:933
#8  0x0000563b5a9884ce in e1000e_set_tdt (core=<optimized out>, index=<optimized out>, val=<optimized out>) at /home/dgilbert/git/hmp/hw/net/e1000e_core.c:2443
#9  0x0000563b5a98b236 in e1000e_core_write (core=0x563b5cbc2ea0, addr=<optimized out>, val=1, size=4) at /home/dgilbert/git/hmp/hw/net/e1000e_core.c:3248
#10 0x0000563b5a7b63d8 in memory_region_write_accessor (mr=0x563b5cbc2ad0, addr=14360, value=<optimized out>, size=4, shift=<optimized out>, mask=<optimized out>, attrs=...) at /home/dgilbert/git/hmp/memory.c:560
#11 0x0000563b5a7b386e in access_with_adjusted_size (addr=addr@entry=14360, value=value@entry=0x7f9657cf9238, size=size@entry=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access_fn=
    0x563b5a7b6360 <memory_region_write_accessor>, mr=0x563b5cbc2ad0, attrs=...) at /home/dgilbert/git/hmp/memory.c:627
#12 0x0000563b5a7b8357 in memory_region_dispatch_write (mr=mr@entry=0x563b5cbc2ad0, addr=14360, data=<optimized out>, size=size@entry=4, attrs=attrs@entry=...) at /home/dgilbert/git/hmp/memory.c:1516
#13 0x0000563b5a773e7e in flatview_write_continue (mr=0x563b5cbc2ad0, l=<optimized out>, addr1=<optimized out>, len=4, buf=0x7f96bdf27028 <error: Cannot access memory at address 0x7f96bdf27028>, attrs=..., addr=1074018328, fv=0x7f96480122e0) at /home/dgilbert/git/hmp/exec.c:2963
#14 0x0000563b5a773e7e in flatview_write (fv=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>) at /home/dgilbert/git/hmp/exec.c:3020
#15 0x0000563b5a778695 in flatview_rw (fv=<optimized out>, addr=<optimized out>, attrs=..., buf=buf@entry=0x7f96bdf27028 <error: Cannot access memory at address 0x7f96bdf27028>, len=len@entry=0, is_write=<optimized out>)
    at /home/dgilbert/git/hmp/exec.c:3129
#16 0x0000563b5a7786df in address_space_rw (as=<optimized out>, addr=<optimized out>, attrs=..., attrs@entry=..., buf=buf@entry=0x7f96bdf27028 <error: Cannot access memory at address 0x7f96bdf27028>, len=0, is_write=<optimized out>)
    at /home/dgilbert/git/hmp/exec.c:3139
#17 0x0000563b5a7c71c8 in kvm_cpu_exec (cpu=cpu@entry=0x563b5be6a680) at /home/dgilbert/git/hmp/accel/kvm/kvm-all.c:1937
#18 0x0000563b5a7a3c74 in qemu_kvm_cpu_thread_fn (arg=0x563b5be6a680) at /home/dgilbert/git/hmp/cpus.c:1128
#19 0x00007f96bd3be609 in start_thread () at /lib64/libpthread.so.0
#20 0x00007f96b3134e6f in clone () at /lib64/libc.so.6
(gdb) p nf
$1 = (NetFilterState *) 0x76656474656e

that nf value is ASCII 'netdev'.

My test is currently:
QEMU -enable-kvm -m 1G -smp 2 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/shm,share=on -numa node,memdev=mem -mem-prealloc -trace events=vhost-trace-file -chardev socket,id=char0,path=/tmp/vubrsrc.sock -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce -device virtio-net-pci,netdev=mynet1 $IMAGE -net none -monitor stdio

then I've got a vhost-user-bridge running on that socket and doing
routing.
In the guest it's doing a looping curl just fetching a page.
And then at the HMP I do:

device-add e1000e

I'm sometimes seeing the crash on this VM, but also sometimes
seeing it if I then migrate and the destination fails in the same
way.

I don't think it's happening without the device-add.

This is on an unmodified 2994cb2ee244b7d6a from today.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK