Re: Problem with iotest 233

["Daniel P. Berrangé" <berrange@redhat.com>]

On Tue, Feb 25, 2025 at 06:52:43PM +0100, Thomas Huth wrote:
> On 25/02/2025 18.44, Thomas Huth wrote:
> > On 25/02/2025 11.12, Kevin Wolf wrote:
> > > Am 25.02.2025 um 08:20 hat Thomas Huth geschrieben:
> > > > 
> > > >   Hi!
> > > > 
> > > > I'm facing a weird hang in iotest 233 on my Fedora 41 laptop. When running
> > > > 
> > > >   ./check -raw 233
> > > > 
> > > > the test simply hangs. Looking at the log, the last message is "== check
> > > > plain client to TLS server fails ==". I added some debug messages, and it
> > > > seems like the previous NBD server is not correctly terminated here.
> > > > The test works fine again if I apply this patch:
> > > > 
> > > > diff --git a/tests/qemu-iotests/common.nbd b/tests/qemu-iotests/common.nbd
> > > > --- a/tests/qemu-iotests/common.nbd
> > > > +++ b/tests/qemu-iotests/common.nbd
> > > > @@ -35,7 +35,7 @@ nbd_server_stop()
> > > >           read NBD_PID < "$nbd_pid_file"
> > > >           rm -f "$nbd_pid_file"
> > > >           if [ -n "$NBD_PID" ]; then
> > > > -            kill "$NBD_PID"
> > > > +            kill -9 "$NBD_PID"
> > > >           fi
> > > >       fi
> > > >       rm -f "$nbd_unix_socket" "$nbd_stderr_fifo"
> > > > 
> > > > ... but that does not look like the right solution to me. What could prevent
> > > > the qemu-nbd from correctly shutting down when it receives a normal SIGTERM
> > > > signal?
> > > 
> > > Not sure. In theory, qemu_system_killed() should set state = TERMINATE
> > > and make main_loop_wait() return through the notification, which should
> > > then make it shut down. Maybe you can attach gdb and check what 'state'
> > > is when it hangs and if it's still in the main loop?
> > 
> > I attached a gdb and ran "bt", and it looks like it is hanging in an
> > exit() handler:
> > 
> > (gdb) bt
> > #0  0x00007f127f8fff1d in syscall () from /lib64/libc.so.6
> > #1  0x00007f127fd32e1d in g_cond_wait () from /lib64/libglib-2.0.so.0
> > #2  0x00005583df3048b2 in flush_trace_file (wait=true) at
> > ../../devel/qemu/ trace/simple.c:140
> > #3  st_flush_trace_buffer () at ../../devel/qemu/trace/simple.c:383
> > #4  0x00007f127f8296c1 in __run_exit_handlers () from /lib64/libc.so.6
> > #5  0x00007f127f82978e in exit () from /lib64/libc.so.6
> > #6  0x00005583df1ae9e1 in main (argc=<optimized out>, argv=<optimized
> > out>) at ../../devel/qemu/qemu-nbd.c:1242
> 
> Ah, now that I wrote that: I recently ran "configure" with
> --enable-trace-backends=simple ... when I remove that from "config.status"
> again, then the test works fine again 8-)
> 
> Still, I think it should not hang with the simple trace backend here, should it?

IIUC this is waiting on trace_empty_cond.

This condition should be signalled from wait_for_trace_records_available
which is in turn called from writeout_thread.

This thread is started from st_init, which is called from trace_init_backends
which should be called from qemu-nbd. I would expect this thread to still
be running when exit() handlers are run.

Does GDB show any other threads running at the time of this hang ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|