Re: [BUG] [powerpc] KVM guest boot failure - hangs on startup after commit 98884e0c

[Misbah Anjum N <misanjum@linux.ibm.com>]

On 2026-03-10 15:04, Ani Sinha wrote:
>> On 10 Mar 2026, at 2:38 PM, Misbah Anjum N <misanjum@linux.ibm.com> 
>> wrote:
>> 
>> On 2026-03-10 14:24, Ani Sinha wrote:
>>>> On 10 Mar 2026, at 2:09 PM, Misbah Anjum N <misanjum@linux.ibm.com> 
>>>> wrote:
>>>> Hi Ani and Paolo,
>>>> We have tested the code by applying both the original commit 
>>>> (98884e0cc10997a17ce9abfd6ff10be19224ca6a) and your fix patch 
>>>> (commit 9e5a6945181d4c1fce7f8438e1b6213f1eb79c14) on ppc64le.
>>>> However, the issue persists. We've conducted GDB debugging that 
>>>> shows the hang is occurring in a different location than what the 
>>>> fix addresses.
>>>> Since the original patch is breaking KVM guest bringup completely on 
>>>> ppc64le, and the fix patch does not resolve the issue, given the 
>>>> severity of this regression (complete KVM breakage on ppc64le), we 
>>>> should either find a quick fix or consider reverting the patch until 
>>>> a proper solution can be identified.
>>> Based on what you just described, it does not seem like the issue is
>>> related to 98884e0cc10997a17ce9abfd6ff10be19224ca6a at all. If you
>>> revert this patch in your local tree, can you confirm that your issue
>>> gets fixed?
>> 
>> Yes, the issue is not seen with the immediate previous commit:
>> 
>> commit df8df3cb6b743372ebb335bd8404bc3d748da350 (ani-df8df3cb)
>> Author: Ani Sinha <anisinha@redhat.com>
>> Date:   Wed Feb 25 09:19:09 2026 +0530
>> 
>>    system/physmem: add helper to reattach existing memory after KVM VM 
>> fd change
>> 
>>    After the guest KVM file descriptor has changed as a part of the 
>> process of
>>    confidential guest reset mechanism, existing memory needs to be 
>> reattached to
>>    the new file descriptor. This change adds a helper function 
>> ram_block_rebind()
>>    for this purpose. The next patch will make use of this function.
>> 
>>    Signed-off-by: Ani Sinha <anisinha@redhat.com>
>>    Link: 
>> https://lore.kernel.org/r/20260225035000.385950-5-anisinha@redhat.com
>>    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> 
>> Looks like the next patch is enabling the functionality of the 
>> previous patches in such a way which causes bql_lock() to get stuck on 
>> architectures (ppc64le in this case) which does not support this 
>> feature yet.
> 
> This theory is not substantiated by code or evidence.
> 98884e0cc10997a17ce9abfd6ff10be19224ca6a introduces kvm_reset_vmfd()
> which is called by this block of code with the tip at
> 98884e0cc10997a17ce9abfd6ff10be19224ca6a :
> 
>    if (!cpus_are_resettable() &&
>         (reason == SHUTDOWN_CAUSE_GUEST_RESET ||
>          reason == SHUTDOWN_CAUSE_HOST_QMP_SYSTEM_RESET)) {
>         if (ac->rebuild_guest) {
>             ret = ac->rebuild_guest(current_machine);
>             if (ret < 0) {
>                 error_report("unable to rebuild guest: %s(%d)",
>                              strerror(-ret), ret);
>                 vm_stop(RUN_STATE_INTERNAL_ERROR);
>             } else {
>                 info_report("virtual machine state has been rebuilt 
> with new "
>                             "guest file handle.");
>                 guest_state_rebuilt = true;
>             }
>         } else if (!cpus_are_resettable())  {
>             error_report("accelerator does not support reset!");
>         } else {
>             error_report("accelerator does not support rebuilding guest 
> state,"
>                          " proceeding with normal reset!");
>         }
>     }
> 
> If cpus are resettable, this block will not be called and nothing that
> the patch introduces will have been executed.
> So I think you guys need to explain a bit more why you so strongly
> feel this patch broke it. I am confused and unable to reason this.
> 
>> 
>> Did you validate your patches on other architectures which does not 
>> support this feature yet?
> 
> As you have already seen, on other architectures, the entire block of
> code is not executed at all. Only SEV-ES, SEV-SNP and TDX currently
> exercises this.
> 

I understand your concern about the code path analysis. Let me clarify 
our findings with concrete evidence.

Reproducibility Evidence:
With commit 98884e0cc10997a17ce9abfd6ff10be19224ca6a applied, we are 
able to reproduce the hang issue 100% of the time across multiple test 
runs. When we revert to the previous commit 
df8df3cb6b743372ebb335bd8404bc3d748da350, the same KVM guest boots 
successfully 100% of the time.

This consistent reproducibility strongly indicates that commit 
98884e0cc10997a17ce9abfd6ff10be19224ca6a is introducing the regression, 
even if the code path analysis suggests otherwise. This suggests the 
issue may not be in the code path, but rather in the changes introduced 
by the patch series.

As the author who led the development of this patch series, we would 
appreciate your help in figuring out this issue.

>> 
>>>> Analysis:
>>>> 1. This is not a confidential guest. This is a regular KVM guest 
>>>> running on ppc64le.
>>>> 2. The execution flow shows that qemu_system_reset() completes 
>>>> successfully and never enters the code path at line 529-543
>>> This is what I expected and therefore, no code related to coco guest
>>> rebuilding is getting executed. Your issue seems to be somewhere 
>>> else.
>> 
>> The issue occurs only with the introduction of this patch and not with 
>> the previous upstream commit as explained above.
>> 
>>>> 3. The hang occurs later in qemu_default_main() at system/main.c:49, 
>>>> after calling bql_lock()
>>>> 4. The ppc KVM guest boots fine with the previous commit - 
>>>> df8df3cb6b743372ebb335bd8404bc3d748da350
>>>> 5. This suggests the issue is not with error handling of -EOPNOTSUPP 
>>>> during reset, but bql_lock() getting stuck in qemu_default_main()
>>>> GDB Trace Analysis:
>>>> We set breakpoints at qemu_system_reset() and qemu_default_main() to 
>>>> trace the execution flow. The system successfully completes 
>>>> qemu_system_reset() without entering the problematic code path where 
>>>> the fix provided by you applies (system/runstate.c:529-543).
>>>> # gdb --args /usr/bin/qemu-system-ppc64 -name avocado-vt-vm1 
>>>> -machine pseries,accel=kvm -enable-kvm -m 32768 -smp 
>>>> 32,sockets=1,cores=32,threads=1 -nographic -serial pty -device 
>>>> virtio-balloon -device virtio-scsi-pci,id=scsi0 -drive 
>>>> file=/home/kvmci/tests/data/avocado-vt/images/rhel8.0devel-ppc64le.qcow2,if=none,id=drive-scsi0-0-0,format=qcow2 
>>>> -device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0 -netdev 
>>>> bridge,id=net0,br=virbr0 -device virtio-net-pci,netdev=net0
>>>> (gdb) handle SIGUSR1 pass nostop noprint
>>>> Signal        Stop Print Pass to program Description
>>>> SIGUSR1       No No Yes User defined signal 1
>>>> (gdb) b qemu_system_reset
>>>> Breakpoint 1 at 0x69a688: file ../system/runstate.c, line 510.
>>>> (gdb) b qemu_default_main
>>>> Breakpoint 2 at 0xa9aeb8: file ../system/main.c, line 45.
>>>> (gdb) r
>>>> Starting program: /usr/bin/qemu-system-ppc64 -name avocado-vt-vm1 
>>>> -machine pseries,accel=kvm -enable-kvm -m 32768 -smp 
>>>> 32,sockets=1,cores=32,threads=1 -nographic -serial pty -device 
>>>> virtio-balloon -device virtio-scsi-pci,id=scsi0 -drive 
>>>> file=/home/kvmci/tests/data/avocado-vt/images/rhel8.0devel-ppc64le.qcow2,if=none,id=drive-scsi0-0-0,format=qcow2 
>>>> -device scsi-hd,bus=scsi0.0,drive=drive-scsi0-0-0 -netdev 
>>>> bridge,id=net0,br=virbr0 -device virtio-net-pci,netdev=net0
>>>> Thread 1 "qemu-system-ppc" hit Breakpoint 1, qemu_system_reset 
>>>> (reason=reason@entry=SHUTDOWN_CAUSE_NONE) at 
>>>> ../system/runstate.c:513
>>>> 513     AccelClass *ac = ACCEL_GET_CLASS(current_accel());
>>>> (gdb) n
>>>> 517     mc = current_machine ? MACHINE_GET_CLASS(current_machine) : 
>>>> NULL;
>>>> (gdb) n
>>>> 519     cpu_synchronize_all_states();
>>>> (gdb) n
>>>> 521     switch (reason) {
>>>> (gdb) n
>>>> 529     if (!cpus_are_resettable() &&
>>>> (gdb) n
>>>> 553     if (mc && mc->reset) {
>>>> (gdb) n
>>>> 554         mc->reset(current_machine, type);
>>>> (gdb) n
>>>> 558     switch (reason) {
>>>> (gdb) n
>>>> 574     if (cpus_are_resettable()) {
>>>> (gdb) n
>>>> 583             cpu_synchronize_all_post_reset();
>>>> (gdb) n
>>>> 587     vm_set_suspended(false);
>>>> (gdb) n
>>>> qdev_machine_creation_done () at ../hw/core/machine.c:1814
>>>> 1814    register_global_state();
>>>> (gdb) n
>>>> qemu_machine_creation_done (errp=0x10123e028 <error_fatal>) at 
>>>> ../system/vl.c:2785
>>>> 2785    if (machine->cgs && !machine->cgs->ready) {
>>>> (gdb) n
>>>> 2791    foreach_device_config_or_exit(DEV_GDB, gdbserver_start);
>>>> (gdb) n
>>>> 2793    if (!vga_interface_created && !default_vga &&
>>>> (gdb) n
>>>> qmp_x_exit_preconfig (errp=errp@entry=0x10123e028 <error_fatal>) at 
>>>> ../system/vl.c:2815
>>>> 2815    if (loadvm) {
>>>> (gdb) n
>>>> 2820    if (replay_mode != REPLAY_MODE_NONE) {
>>>> (gdb) n
>>>> 2824    if (incoming) {
>>>> (gdb) n
>>>> 2837    } else if (autostart) {
>>>> (gdb) n
>>>> 2838        qmp_cont(NULL);
>>>> (gdb) n
>>>> qemu_init (argc=<optimized out>, argv=<optimized out>) at 
>>>> ../system/vl.c:3849
>>>> 3849    qemu_init_displays();
>>>> (gdb) n
>>>> 3850    accel_setup_post(current_machine);
>>>> (gdb) n
>>>> 3851    if (migrate_mode() != MIG_MODE_CPR_EXEC) {
>>>> (gdb) n
>>>> 3852        os_setup_post();
>>>> (gdb) n
>>>> 3854    resume_mux_open();
>>>> (gdb) n
>>>> main (argc=<optimized out>, argv=<optimized out>) at 
>>>> ../system/main.c:84
>>>> 84      bql_unlock();
>>>> (gdb) n
>>>> 85      replay_mutex_unlock();
>>>> (gdb) n
>>>> 87      if (qemu_main) {
>>>> (gdb) n
>>>> 93          qemu_default_main(NULL);
>>>> (gdb) n
>>>> Thread 1 "qemu-system-ppc" hit Breakpoint 2, qemu_default_main 
>>>> (opaque=opaque@entry=0x0) at ../system/main.c:48
>>>> 48      replay_mutex_lock();
>>>> (gdb) n
>>>> 49      bql_lock();
>>>> (gdb) n
>>>> <hangs>
>>>> <system becomes unresponsive at this point>
>>>> Thanks,
>>>> Misbah Anjum N <misanjumn@ibm.com>
>>>> On 2026-03-09 18:53, Ani Sinha wrote:
>>>>> Yes seems this is an issue and I will fix it. Not sure if the fix 
>>>>> will
>>>>> address your issue though ...
>>>>> Can you try the following patch?
>>>>> From 9e5a6945181d4c1fce7f8438e1b6213f1eb79c14 Mon Sep 17 00:00:00 
>>>>> 2001
>>>>> From: Ani Sinha <anisinha@redhat.com>
>>>>> Date: Mon, 9 Mar 2026 18:44:40 +0530
>>>>> Subject: [PATCH] Fix reset for non-x86 archs that do not support 
>>>>> reset yet
>>>>> Signed-off-by: Ani Sinha <anisinha@redhat.com>
>>>>> ---
>>>>> system/runstate.c | 4 +++-
>>>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>>>> diff --git a/system/runstate.c b/system/runstate.c
>>>>> index eca722b43c..c1f41284c9 100644
>>>>> --- a/system/runstate.c
>>>>> +++ b/system/runstate.c
>>>>> @@ -531,10 +531,12 @@ void qemu_system_reset(ShutdownCause reason)
>>>>>        (current_machine->new_accel_vmfd_on_reset || 
>>>>> !cpus_are_resettable())) {
>>>>>        if (ac->rebuild_guest) {
>>>>>            ret = ac->rebuild_guest(current_machine);
>>>>> -            if (ret < 0) {
>>>>> +            if (ret < 0 && ret != -EOPNOTSUPP) {
>>>>>                error_report("unable to rebuild guest: %s(%d)",
>>>>>                             strerror(-ret), ret);
>>>>>                vm_stop(RUN_STATE_INTERNAL_ERROR);
>>>>> +            } else if (ret == -EOPNOTSUPP) {
>>>>> +                error_report("accelerator does not support 
>>>>> reset!");
>>>>>            } else {
>>>>>                info_report("virtual machine state has been rebuilt 
>>>>> with new "
>>>>>                            "guest file handle.");
>>>>> --
>>>>> 2.42.0
>>>>>> Is this a confidential guest that cannot be normally reset?