[PATCH 0/1] Add missing function names to symbol list

[PATCH 0/1] Add missing function names to symbol list

[Lukas Jünger]

Hi all,

I have been trying to use the hwprofile and cache plugin on
qemu-system-riscv64. They failed to load with an undefined
symbol error. It looks like some of the plugin API functions
are missing from the symbol list, so I added them. Afterwards
the plugins worked (eventhough the cache plugin is segfaulting
on shutdown, but that is a separate, unrelated issue).

Hope that's okay.

BR,
Lukas

Lukas Jünger (1):
  plugins/: Add missing functions to symbol list

 plugins/qemu-plugins.symbols | 3 +++
 1 file changed, 3 insertions(+)

-- 
2.31.1

[PATCH 1/1] plugins/: Add missing functions to symbol list

[Lukas Jünger]

Some functions of the plugin API were missing in
the symbol list. However, they are all used by
the contributed example plugins. QEMU fails to
load the plugin if the function symbol is not
exported.

Signed-off-by: Lukas Jünger <lukas.junger@greensocs.com>
---
 plugins/qemu-plugins.symbols | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/plugins/qemu-plugins.symbols b/plugins/qemu-plugins.symbols
index 67b309ea2a..4834756ba3 100644
--- a/plugins/qemu-plugins.symbols
+++ b/plugins/qemu-plugins.symbols
@@ -1,11 +1,14 @@
 {
   qemu_plugin_bool_parse;
   qemu_plugin_get_hwaddr;
+  qemu_plugin_hwaddr_device_name;
   qemu_plugin_hwaddr_is_io;
+  qemu_plugin_hwaddr_phys_addr;
   qemu_plugin_insn_data;
   qemu_plugin_insn_disas;
   qemu_plugin_insn_haddr;
   qemu_plugin_insn_size;
+  qemu_plugin_insn_symbol;
   qemu_plugin_insn_vaddr;
   qemu_plugin_mem_is_big_endian;
   qemu_plugin_mem_is_sign_extended;
-- 
2.31.1

Re: [PATCH 0/1] Add missing function names to symbol list

[Alex Bennée]

Lukas Jünger <lukas.junger@greensocs.com> writes:

> Hi all,
>
> I have been trying to use the hwprofile and cache plugin on
> qemu-system-riscv64. They failed to load with an undefined
> symbol error. It looks like some of the plugin API functions
> are missing from the symbol list, so I added them.

Heh I had a similar patch in plugins/next but yours includes more
symbols so queued to plugins/next, thanks. Not sure how I missed it
before - I think we might not have been honouring the symbol list
before.

> Afterwards
> the plugins worked (eventhough the cache plugin is segfaulting
> on shutdown, but that is a separate, unrelated issue).

Has this been tracked down yet? There is a fix f7e68c9c99 (tcg/plugins:
implement a qemu_plugin_user_exit helper) which was for user mode. At
the time I didn't think system mode would be affected due to the atexit
callback only being called once vCPUs had shutdown.

>
> Hope that's okay.
>
> BR,
> Lukas
>
> Lukas Jünger (1):
>   plugins/: Add missing functions to symbol list
>
>  plugins/qemu-plugins.symbols | 3 +++
>  1 file changed, 3 insertions(+)


-- 
Alex Bennée

Re: [PATCH 0/1] Add missing function names to symbol list

[Lukas Jünger]

On 9/7/21 13:43, Alex Bennée wrote:
> Lukas Jünger <lukas.junger@greensocs.com> writes:
>
>> Hi all,
>>
>> I have been trying to use the hwprofile and cache plugin on
>> qemu-system-riscv64. They failed to load with an undefined
>> symbol error. It looks like some of the plugin API functions
>> are missing from the symbol list, so I added them.
> Heh I had a similar patch in plugins/next but yours includes more
> symbols so queued to plugins/next, thanks. Not sure how I missed it
> before - I think we might not have been honouring the symbol list
> before.
>
>> Afterwards
>> the plugins worked (eventhough the cache plugin is segfaulting
>> on shutdown, but that is a separate, unrelated issue).
> Has this been tracked down yet? There is a fix f7e68c9c99 (tcg/plugins:
> implement a qemu_plugin_user_exit helper) which was for user mode. At
> the time I didn't think system mode would be affected due to the atexit
> callback only being called once vCPUs had shutdown.

It seems like there is a race condition with the tcg threads.
The plugin exit handler is run with atexit(). While the exit callback
is freeing memory, tcg is still running and memory callbacks write
to the already freed data structures in the plugin causing the segfault.
I tested this with the current master branch and this cmdline:

bin/qemu-system-riscv64 -machine virt -nographic -bios fw_jump.elf
     -kernel Image -plugin path/to/libcache.so -d plugin -D log.txt

I wonder if we could somehow wait for the tcg to exit before executing
plugin exit cb. Do you have an idea?

>
>> Hope that's okay.
>>
>> BR,
>> Lukas
>>
>> Lukas Jünger (1):
>>    plugins/: Add missing functions to symbol list
>>
>>   plugins/qemu-plugins.symbols | 3 +++
>>   1 file changed, 3 insertions(+)
>

Re: [PATCH 0/1] Add missing function names to symbol list

[Alex Bennée]

Lukas Jünger <lukas.junger@greensocs.com> writes:

> On 9/7/21 13:43, Alex Bennée wrote:
>> Lukas Jünger <lukas.junger@greensocs.com> writes:
>>
>>> Hi all,
>>>
>>> I have been trying to use the hwprofile and cache plugin on
>>> qemu-system-riscv64. They failed to load with an undefined
>>> symbol error. It looks like some of the plugin API functions
>>> are missing from the symbol list, so I added them.
>> Heh I had a similar patch in plugins/next but yours includes more
>> symbols so queued to plugins/next, thanks. Not sure how I missed it
>> before - I think we might not have been honouring the symbol list
>> before.
>>
>>> Afterwards
>>> the plugins worked (eventhough the cache plugin is segfaulting
>>> on shutdown, but that is a separate, unrelated issue).
>> Has this been tracked down yet? There is a fix f7e68c9c99 (tcg/plugins:
>> implement a qemu_plugin_user_exit helper) which was for user mode. At
>> the time I didn't think system mode would be affected due to the atexit
>> callback only being called once vCPUs had shutdown.
>
> It seems like there is a race condition with the tcg threads.
> The plugin exit handler is run with atexit(). While the exit callback
> is freeing memory, tcg is still running and memory callbacks write
> to the already freed data structures in the plugin causing the segfault.
> I tested this with the current master branch and this cmdline:

We fixed a bug in linux-user that was leading to the exit callbacks
being called (see qemu_plugin_user_exit).

> bin/qemu-system-riscv64 -machine virt -nographic -bios fw_jump.elf
>     -kernel Image -plugin path/to/libcache.so -d plugin -D log.txt
>
> I wonder if we could somehow wait for the tcg to exit before executing
> plugin exit cb. Do you have an idea?

It should be because I don't see how TCG would still be running when we
run the atexit handler. It literally shouldn't be called until QEMU
itself calls exit which should be well after the TCG has stopped running
(see pause_all_vcpus).

Any chance you could replicate and generate a backtrace that shows this
happening?

-- 
Alex Bennée

Re: [PATCH 0/1] Add missing function names to symbol list

[Lukas Jünger]

On 10/18/21 13:18, Alex Bennée wrote:
> 
> Lukas Jünger <lukas.junger@greensocs.com> writes:
> 
>> On 9/7/21 13:43, Alex Bennée wrote:
>>> Lukas Jünger <lukas.junger@greensocs.com> writes:
>>>
>>>> Hi all,
>>>>
>>>> I have been trying to use the hwprofile and cache plugin on
>>>> qemu-system-riscv64. They failed to load with an undefined
>>>> symbol error. It looks like some of the plugin API functions
>>>> are missing from the symbol list, so I added them.
>>> Heh I had a similar patch in plugins/next but yours includes more
>>> symbols so queued to plugins/next, thanks. Not sure how I missed it
>>> before - I think we might not have been honouring the symbol list
>>> before.
>>>
>>>> Afterwards
>>>> the plugins worked (eventhough the cache plugin is segfaulting
>>>> on shutdown, but that is a separate, unrelated issue).
>>> Has this been tracked down yet? There is a fix f7e68c9c99 (tcg/plugins:
>>> implement a qemu_plugin_user_exit helper) which was for user mode. At
>>> the time I didn't think system mode would be affected due to the atexit
>>> callback only being called once vCPUs had shutdown.
>>
>> It seems like there is a race condition with the tcg threads.
>> The plugin exit handler is run with atexit(). While the exit callback
>> is freeing memory, tcg is still running and memory callbacks write
>> to the already freed data structures in the plugin causing the segfault.
>> I tested this with the current master branch and this cmdline:
> 
> We fixed a bug in linux-user that was leading to the exit callbacks
> being called (see qemu_plugin_user_exit).
> 
>> bin/qemu-system-riscv64 -machine virt -nographic -bios fw_jump.elf
>>      -kernel Image -plugin path/to/libcache.so -d plugin -D log.txt
>>
>> I wonder if we could somehow wait for the tcg to exit before executing
>> plugin exit cb. Do you have an idea?
> 
> It should be because I don't see how TCG would still be running when we
> run the atexit handler. It literally shouldn't be called until QEMU
> itself calls exit which should be well after the TCG has stopped running
> (see pause_all_vcpus).
> 
> Any chance you could replicate and generate a backtrace that shows this
> happening?
> 

I'm on Fedora 34, running:

gdb --args bin/qemu-system-riscv64 -machine virt -nographic -bios 
/home/lukas/work/greensocs/projects/sifive/buildroot-2021.05.1/BUILD/images/fw_jump.elf 
-kernel 
/home/lukas/work/greensocs/projects/sifive/buildroot-2021.05.1/BUILD/images/Image 
-plugin BUILD/contrib/plugins/libcache.so -d plugin -D foo.log

I get:

Thread 1 "qemu-system-ris" received signal SIGSEGV, Segmentation fault.
0x00007ffff76a9571 in unlink_chunk.constprop () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install 
bzip2-libs-1.0.8-6.fc34.x86_64 glib2-2.68.4-1.fc34.x86_64 
glibc-2.33-20.fc34.x86_64 libblkid-2.36.2-1.fc34.x86_64 
libfdt-1.6.1-1.fc34.x86_64 libffi-3.1-28.fc34.x86_64 
libgcc-11.2.1-1.fc34.x86_64 libgcrypt-1.9.3-3.fc34.x86_64 
libgpg-error-1.42-1.fc34.x86_64 libmount-2.36.2-1.fc34.x86_64 
libpng-1.6.37-10.fc34.x86_64 ncurses-libs-6.2-4.20200222.fc34.x86_64 
pcre-8.44-3.fc34.1.x86_64 pcre2-10.36-4.fc34.x86_64 
pixman-0.40.0-3.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
(gdb) thread apply all bt

Thread 3 (Thread 0x7ffff6a85640 (LWP 669129) "qemu-system-ris"):
#0  0x00007ffff7bdd1ad in g_mutex_lock () at /lib64/libglib-2.0.so.0
#1  0x00007ffff7fc0e19 in vcpu_mem_access (vcpu_index=0, info=131121, 
vaddr=18446743936379926112, userdata=0x7fff6812dfb0) at 
/home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:395
#2  0x00007fff7021377c in code_gen_buffer ()
#3  0x0000555555c4daf1 in cpu_tb_exec (cpu=0x555556880570, 
itb=0x7fffb020bc40, tb_exit=0x7ffff6a84834) at 
../../../accel/tcg/cpu-exec.c:353
#4  0x0000555555c4e8f2 in cpu_loop_exec_tb (cpu=0x555556880570, 
tb=0x7fffb020bc40, last_tb=0x7ffff6a84840, tb_exit=0x7ffff6a84834) at 
../../../accel/tcg/cpu-exec.c:829
#5  0x0000555555c4ecd7 in cpu_exec (cpu=0x555556880570) at 
../../../accel/tcg/cpu-exec.c:987
#6  0x0000555555c703ca in tcg_cpus_exec (cpu=0x555556880570) at 
../../../accel/tcg/tcg-accel-ops.c:67
#7  0x0000555555c706dc in mttcg_cpu_thread_fn (arg=0x555556880570) at 
../../../accel/tcg/tcg-accel-ops-mttcg.c:70
#8  0x0000555555e2b806 in qemu_thread_start (args=0x5555568a60b0) at 
../../../util/qemu-thread-posix.c:556
#9  0x00007ffff77f9299 in start_thread () at /lib64/libpthread.so.0
#10 0x00007ffff7721353 in clone () at /lib64/libc.so.6

Thread 2 (Thread 0x7ffff7408640 (LWP 669128) "qemu-system-ris"):
#0  0x00007ffff771be0d in syscall () at /lib64/libc.so.6
#1  0x0000555555e2b468 in qemu_futex_wait (f=0x55555663da00 
<rcu_gp_event>, val=4294967295) at 
/home/lukas/work/greensocs/projects/sifive/upstream_qemu/include/qemu/futex.h:29
#2  0x0000555555e2b653 in qemu_event_wait (ev=0x55555663da00 
<rcu_gp_event>) at ../../../util/qemu-thread-posix.c:481
#3  0x0000555555e364a4 in wait_for_readers () at ../../../util/rcu.c:135
#4  0x0000555555e36620 in synchronize_rcu () at ../../../util/rcu.c:171
#5  0x0000555555e367d3 in call_rcu_thread (opaque=0x0) at 
../../../util/rcu.c:265
#6  0x0000555555e2b806 in qemu_thread_start (args=0x555556648860) at 
../../../util/qemu-thread-posix.c:556
#7  0x00007ffff77f9299 in start_thread () at /lib64/libpthread.so.0
#8  0x00007ffff7721353 in clone () at /lib64/libc.so.6

Thread 1 (Thread 0x7ffff740a0c0 (LWP 669123) "qemu-system-ris"):
#0  0x00007ffff76a9571 in unlink_chunk.constprop () at /lib64/libc.so.6
#1  0x00007ffff76a99e1 in _int_free () at /lib64/libc.so.6
#2  0x00007ffff76ad7c8 in free () at /lib64/libc.so.6
#3  0x00007ffff7b9424d in g_free () at /lib64/libglib-2.0.so.0
#4  0x00007ffff7fc11ad in cache_free (cache=0x5555566a1c60) at 
/home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:478
#5  0x00007ffff7fc1231 in caches_free (caches=0x5555568517e0) at 
/home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:494
#6  0x00007ffff7fc18f3 in plugin_exit (id=14580660273623469927, p=0x0) 
at 
/home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:616
#7  0x0000555555c6eb0b in plugin_cb__udata (ev=QEMU_PLUGIN_EV_ATEXIT) at 
../../../plugins/core.c:156
#8  0x0000555555c6f7b7 in qemu_plugin_atexit_cb () at 
../../../plugins/core.c:480
#9  0x00007ffff7660af7 in __run_exit_handlers () at /lib64/libc.so.6
#10 0x00007ffff7660ca0 in on_exit () at /lib64/libc.so.6
#11 0x0000555555d78166 in mux_proc_byte (chr=0x555556823400, 
d=0x555556823400, ch=120) at ../../../chardev/char-mux.c:160
#12 0x0000555555d7848b in mux_chr_read (opaque=0x555556823400, 
buf=0x7fffffffd240 "x\353\364UUU", size=1) at 
../../../chardev/char-mux.c:230
#13 0x0000555555d7f882 in qemu_chr_be_write_impl (s=0x555556750540, 
buf=0x7fffffffd240 "x\353\364UUU", len=1) at ../../../chardev/char.c:201
#14 0x0000555555d7f8e6 in qemu_chr_be_write (s=0x555556750540, 
buf=0x7fffffffd240 "x\353\364UUU", len=1) at ../../../chardev/char.c:213
#15 0x0000555555d82476 in fd_chr_read (chan=0x555556649e20, 
cond=G_IO_IN, opaque=0x555556750540) at ../../../chardev/char-fd.c:73
#16 0x0000555555c8abcf in qio_channel_fd_source_dispatch 
(source=0x555556d8c2a0, callback=0x555555d8234c <fd_chr_read>, 
user_data=0x555556750540) at ../../../io/channel-watch.c:84
#17 0x00007ffff7b8f4cf in g_main_context_dispatch () at 
/lib64/libglib-2.0.so.0
#18 0x0000555555e50adb in glib_pollfds_poll () at 
../../../util/main-loop.c:232
#19 0x0000555555e50b58 in os_host_main_loop_wait (timeout=1000000000) at 
../../../util/main-loop.c:255
#20 0x0000555555e50c66 in main_loop_wait (nonblocking=0) at 
../../../util/main-loop.c:531
#21 0x0000555555b5216f in qemu_main_loop () at 
../../../softmmu/runstate.c:726
#22 0x000055555580bcf6 in main (argc=14, argv=0x7fffffffe508, 
envp=0x7fffffffe580) at ../../../softmmu/main.c:50

Hope that helps?

Re: [PATCH 0/1] Add missing function names to symbol list

[Alex Bennée]

Lukas Jünger <lukas.junger@greensocs.com> writes:

> On 10/18/21 13:18, Alex Bennée wrote:
>> Lukas Jünger <lukas.junger@greensocs.com> writes:
>> 
>>> On 9/7/21 13:43, Alex Bennée wrote:
>>>> Lukas Jünger <lukas.junger@greensocs.com> writes:
>>>>
>>>>> Hi all,
>>>>>
<snip>
>>> It seems like there is a race condition with the tcg threads.
>>> The plugin exit handler is run with atexit(). While the exit callback
>>> is freeing memory, tcg is still running and memory callbacks write
>>> to the already freed data structures in the plugin causing the segfault.
>>> I tested this with the current master branch and this cmdline:
>> We fixed a bug in linux-user that was leading to the exit callbacks
>> being called (see qemu_plugin_user_exit).
>> 
>>> bin/qemu-system-riscv64 -machine virt -nographic -bios fw_jump.elf
>>>      -kernel Image -plugin path/to/libcache.so -d plugin -D log.txt
>>>
>>> I wonder if we could somehow wait for the tcg to exit before executing
>>> plugin exit cb. Do you have an idea?
>> It should be because I don't see how TCG would still be running when
>> we
>> run the atexit handler. It literally shouldn't be called until QEMU
>> itself calls exit which should be well after the TCG has stopped running
>> (see pause_all_vcpus).
>> Any chance you could replicate and generate a backtrace that shows
>> this
>> happening?
>> 
>
> I'm on Fedora 34, running:
>
> gdb --args bin/qemu-system-riscv64 -machine virt -nographic -bios
> /home/lukas/work/greensocs/projects/sifive/buildroot-2021.05.1/BUILD/images/fw_jump.elf
> -kernel
> /home/lukas/work/greensocs/projects/sifive/buildroot-2021.05.1/BUILD/images/Image
> -plugin BUILD/contrib/plugins/libcache.so -d plugin -D foo.log
>
> I get:
>
> Thread 1 "qemu-system-ris" received signal SIGSEGV, Segmentation fault.
> 0x00007ffff76a9571 in unlink_chunk.constprop () from /lib64/libc.so.6
> Missing separate debuginfos, use: dnf debuginfo-install
> bzip2-libs-1.0.8-6.fc34.x86_64 glib2-2.68.4-1.fc34.x86_64
> glibc-2.33-20.fc34.x86_64 libblkid-2.36.2-1.fc34.x86_64
> libfdt-1.6.1-1.fc34.x86_64 libffi-3.1-28.fc34.x86_64
> libgcc-11.2.1-1.fc34.x86_64 libgcrypt-1.9.3-3.fc34.x86_64
> libgpg-error-1.42-1.fc34.x86_64 libmount-2.36.2-1.fc34.x86_64
> libpng-1.6.37-10.fc34.x86_64 ncurses-libs-6.2-4.20200222.fc34.x86_64
> pcre-8.44-3.fc34.1.x86_64 pcre2-10.36-4.fc34.x86_64
> pixman-0.40.0-3.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
> (gdb) thread apply all bt
>
> Thread 3 (Thread 0x7ffff6a85640 (LWP 669129) "qemu-system-ris"):
> #0  0x00007ffff7bdd1ad in g_mutex_lock () at /lib64/libglib-2.0.so.0
> #1  0x00007ffff7fc0e19 in vcpu_mem_access (vcpu_index=0, info=131121,
>  vaddr=18446743936379926112, userdata=0x7fff6812dfb0) at
>  /home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:395
> #2  0x00007fff7021377c in code_gen_buffer ()
> #3  0x0000555555c4daf1 in cpu_tb_exec (cpu=0x555556880570,
>  itb=0x7fffb020bc40, tb_exit=0x7ffff6a84834) at
>  ../../../accel/tcg/cpu-exec.c:353
> #4  0x0000555555c4e8f2 in cpu_loop_exec_tb (cpu=0x555556880570,
>  tb=0x7fffb020bc40, last_tb=0x7ffff6a84840, tb_exit=0x7ffff6a84834) at
>  ../../../accel/tcg/cpu-exec.c:829
> #5  0x0000555555c4ecd7 in cpu_exec (cpu=0x555556880570) at
>  ../../../accel/tcg/cpu-exec.c:987
> #6  0x0000555555c703ca in tcg_cpus_exec (cpu=0x555556880570) at
>  ../../../accel/tcg/tcg-accel-ops.c:67
> #7  0x0000555555c706dc in mttcg_cpu_thread_fn (arg=0x555556880570) at
>  ../../../accel/tcg/tcg-accel-ops-mttcg.c:70
> #8  0x0000555555e2b806 in qemu_thread_start (args=0x5555568a60b0) at
>  ../../../util/qemu-thread-posix.c:556
> #9  0x00007ffff77f9299 in start_thread () at /lib64/libpthread.so.0
> #10 0x00007ffff7721353 in clone () at /lib64/libc.so.6
>
> Thread 2 (Thread 0x7ffff7408640 (LWP 669128) "qemu-system-ris"):
> #0  0x00007ffff771be0d in syscall () at /lib64/libc.so.6
> #1  0x0000555555e2b468 in qemu_futex_wait (f=0x55555663da00
>  <rcu_gp_event>, val=4294967295) at
>  /home/lukas/work/greensocs/projects/sifive/upstream_qemu/include/qemu/futex.h:29
> #2  0x0000555555e2b653 in qemu_event_wait (ev=0x55555663da00
>  <rcu_gp_event>) at ../../../util/qemu-thread-posix.c:481
> #3  0x0000555555e364a4 in wait_for_readers () at ../../../util/rcu.c:135
> #4  0x0000555555e36620 in synchronize_rcu () at ../../../util/rcu.c:171
> #5  0x0000555555e367d3 in call_rcu_thread (opaque=0x0) at
>  ../../../util/rcu.c:265
> #6  0x0000555555e2b806 in qemu_thread_start (args=0x555556648860) at
>  ../../../util/qemu-thread-posix.c:556
> #7  0x00007ffff77f9299 in start_thread () at /lib64/libpthread.so.0
> #8  0x00007ffff7721353 in clone () at /lib64/libc.so.6
>
> Thread 1 (Thread 0x7ffff740a0c0 (LWP 669123) "qemu-system-ris"):
> #0  0x00007ffff76a9571 in unlink_chunk.constprop () at /lib64/libc.so.6
> #1  0x00007ffff76a99e1 in _int_free () at /lib64/libc.so.6
> #2  0x00007ffff76ad7c8 in free () at /lib64/libc.so.6
> #3  0x00007ffff7b9424d in g_free () at /lib64/libglib-2.0.so.0
> #4  0x00007ffff7fc11ad in cache_free (cache=0x5555566a1c60) at
>  /home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:478
> #5  0x00007ffff7fc1231 in caches_free (caches=0x5555568517e0) at
>  /home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:494
> #6  0x00007ffff7fc18f3 in plugin_exit (id=14580660273623469927, p=0x0)
>  at
>  /home/lukas/work/greensocs/projects/sifive/upstream_qemu/contrib/plugins/cache.c:616
> #7  0x0000555555c6eb0b in plugin_cb__udata (ev=QEMU_PLUGIN_EV_ATEXIT)
>  at ../../../plugins/core.c:156
> #8  0x0000555555c6f7b7 in qemu_plugin_atexit_cb () at
>  ../../../plugins/core.c:480
> #9  0x00007ffff7660af7 in __run_exit_handlers () at /lib64/libc.so.6
> #10 0x00007ffff7660ca0 in on_exit () at /lib64/libc.so.6
> #11 0x0000555555d78166 in mux_proc_byte (chr=0x555556823400,
>  d=0x555556823400, ch=120) at ../../../chardev/char-mux.c:160

Hmm it seems the problem is being able to slam out of QEMU in the mux
handler. Ideally it should be the start of triggering a graceful
shutdown probably with something like
qemu_system_shutdown_request(SHUTDOWN_CAUSE_GUEST_SHUTDOWN);

> Hope that helps?


-- 
Alex Bennée