Qemu-storage-daemon crash with 8ab8140 or later

Qemu-storage-daemon crash with 8ab8140 or later

[Lukáš Doktor]

[-- Attachment #1.1.1: Type: text/plain, Size: 4221 bytes --]

Hello folks,

I found a functional regression in qemu which is making qemu-storage-daemon to crash when using /dev/ram0 as the host device. I bisected it up to:

commit 8ab8140a04cf771d63e9754d6ba6c1e676bfe507
Author: Kevin Wolf <kwolf@redhat.com>
Date:   Fri Feb 3 16:22:02 2023 +0100

    block: Mark bdrv_co_refresh_total_sectors() and callers GRAPH_RDLOCK
    
    This adds GRAPH_RDLOCK annotations to declare that callers of
    bdrv_co_refresh_total_sectors() need to hold a reader lock for the
    graph.
    
    Signed-off-by: Kevin Wolf <kwolf@redhat.com>
    Message-Id: <20230203152202.49054-24-kwolf@redhat.com>
    Reviewed-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
    Signed-off-by: Kevin Wolf <kwolf@redhat.com>

To reproduce it create the export by:

    mkdir -p /var/lib/runperf/runperf-libblkio/
    modprobe brd rd_nr=1 rd_size=1048576 max_part=0
    qemu-storage-daemon --blockdev driver=host_device,node-name=file,filename=/dev/ram0,cache.direct=on --object iothread,id=iothread0 --export type=vhost-user-blk,iothread=iothread0,id=export,node-name=file,addr.type=unix,addr.path=/var/lib/runperf/runperf-libblkio/vhost-user-blk.sock,writable=on

Then create the following fio job in /var/lib/runperf/runperf-libblkio/libblkio.fio:

[global]
bs=4k
ioengine=libblkio
libblkio_driver=virtio-blk-vhost-user
libblkio_path=/var/lib/runperf/runperf-libblkio/vhost-user-blk.sock
rw=read
iodepth=1
hipri=1        # Can not be set by pbench-fio, use 0 or 1
direct=1
sync=0
time_based=1
clocksource=gettimeofday
ramp_time=5
runtime=10
write_bw_log=fio
write_iops_log=fio
write_lat_log=fio
log_avg_msec=1000
write_hist_log=fio
log_hist_msec=10000
# log_hist_coarseness=4 # 76 bins

[job0]

And run it using:

    timeout -s 9 20 fio /var/lib/runperf/runperf-libblkio/libblkio.fio

Before the 8ab8140a04cf771d63e9754d6ba6c1e676bfe507 this worked well, using every newer commit results in qemu-storage-daemon coredump and fio hang (which is why I added the timeout to this example)

Coredump:

[root@virtlab722 ~]# coredumpctl info
           PID: 40972 (qemu-storage-da)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Mon 2023-03-13 07:51:09 EDT (34s ago)
  Command Line: qemu-storage-daemon --blockdev driver=host_device,node-name=file,filename=/dev/ram0,cache.direct=on --object iothread,id=iothread0 ->
    Executable: /usr/local/bin/qemu-storage-daemon
 Control Group: /user.slice/user-0.slice/session-5.scope
          Unit: session-5.scope
         Slice: user-0.slice
       Session: 5
     Owner UID: 0 (root)
       Boot ID: 7705815f01cb4cce9999fe6ccfe2c0ba
    Machine ID: 617b70389a7d4b4280ec97bdcb203def
      Hostname: virtlab722.virt.lab.eng.bos.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-storage-da.0.7705815f01cb4cce9999fe6ccfe2c0ba.40972.1678708269000000.zst (present)
  Size on Disk: 115.8K
       Message: Process 40972 (qemu-storage-da) of user 0 dumped core.
                
                Stack trace of thread 40974:
                #0  0x00007f89c22a154c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #1  0x00007f89c2254d46 raise (libc.so.6 + 0x54d46)
                #2  0x00007f89c22287f3 abort (libc.so.6 + 0x287f3)
                #3  0x00007f89c222871b __assert_fail_base.cold (libc.so.6 + 0x2871b)
                #4  0x00007f89c224dce6 __assert_fail (libc.so.6 + 0x4dce6)
                #5  0x00005577565ac060 assert_bdrv_graph_readable (qemu-storage-daemon + 0xb9060)
                #6  0x0000557756585552 bdrv_co_nb_sectors (qemu-storage-daemon + 0x92552)
                #7  0x0000557756585639 bdrv_get_geometry (qemu-storage-daemon + 0x92639)
                #8  0x0000557756570a55 virtio_blk_sect_range_ok (qemu-storage-daemon + 0x7da55)
                #9  0x0000557756570387 vu_blk_virtio_process_req (qemu-storage-daemon + 0x7d387)
                #10 0x000055775669cbcb coroutine_trampoline (qemu-storage-daemon + 0x1a9bcb)
                #11 0x00007f89c222a360 n/a (libc.so.6 + 0x2a360)
                ELF object binary architecture: AMD x86-64


Regards,
Lukáš Doktor

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]