Re: [PATCH RFC V6 05/24] arm/virt,kvm: Pre-create KVM vCPUs for 'disabled' QOM vCPUs at machine init

[Gavin Shan <gshan@redhat.com>]

Hi Salil,

On 10/23/25 10:35 AM, Salil Mehta wrote:
> On Thu, Oct 23, 2025 at 12:14 AM Gavin Shan <gshan@redhat.com> wrote:
>> On 10/23/25 4:50 AM, Salil Mehta wrote:
>>> On Wed, Oct 22, 2025 at 6:18 PM Salil Mehta <salil.mehta@opnsrc.net> wrote:
>>>> On Wed, Oct 22, 2025 at 10:37 AM Gavin Shan <gshan@redhat.com> wrote:
>>>>> On 10/1/25 11:01 AM, salil.mehta@opnsrc.net wrote:
>>>>>> From: Salil Mehta <salil.mehta@huawei.com>

[...]

>>>>>> +void kvm_arm_create_host_vcpu(ARMCPU *cpu)
>>>>>> +{
>>>>>> +    CPUState *cs = CPU(cpu);
>>>>>> +    unsigned long vcpu_id = cs->cpu_index;
>>>>>> +    int ret;
>>>>>> +
>>>>>> +    ret = kvm_create_vcpu(cs);
>>>>>> +    if (ret < 0) {
>>>>>> +        error_report("Failed to create host vcpu %ld", vcpu_id);
>>>>>> +        abort();
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * Initialize the vCPU in the host. This will reset the sys regs
>>>>>> +     * for this vCPU and related registers like MPIDR_EL1 etc. also
>>>>>> +     * get programmed during this call to host. These are referenced
>>>>>> +     * later while setting device attributes of the GICR during GICv3
>>>>>> +     * reset.
>>>>>> +     */
>>>>>> +    ret = kvm_arch_init_vcpu(cs);
>>>>>> +    if (ret < 0) {
>>>>>> +        error_report("Failed to initialize host vcpu %ld", vcpu_id);
>>>>>> +        abort();
>>>>>> +    }
>>>>>> +
>>>>>> +    /*
>>>>>> +     * park the created vCPU. shall be used during kvm_get_vcpu() when
>>>>>> +     * threads are created during realization of ARM vCPUs.
>>>>>> +     */
>>>>>> +    kvm_park_vcpu(cs);
>>>>>> +}
>>>>>> +
>>>>>
>>>>> I don't think we're able to simply call kvm_arch_init_vcpu() in the lazily realized
>>>>> path. Otherwise, it can trigger a crash dump on my Nvidia's grace-hopper machine where
>>>>> SVE is supported by default.
>>>>
>>>> Thanks for reporting this. That is not true. As long as we initialize
>>>> KVM correctly and
>>>> finalize the features like SVE we should be fine. In fact, this is
>>>> precisely what we are
>>>> doing right now.
>>>>
>>>> To understand the crash, I need a bit more info.
>>>>
>>>> 1#  is happening because KVM_ARM_VCPU_INIT is failing. If yes, the can you check
>>>>         within the KVM if it is happening because
>>>>        a.  features specified by QEMU are not matching the defaults within the KVM
>>>>              (HInt: check kvm_vcpu_init_check_features())?
>>>>        b. or complaining about init feate change kvm_vcpu_init_changed()?
>>>> 2#  or it is happening during the setting of vector length or
>>>> finalizing features?
>>>>
>>>> int kvm_arch_init_vcpu(CPUState *cs)
>>>> {
>>>>      [...]
>>>>            /* Do KVM_ARM_VCPU_INIT ioctl */
>>>>           ret = kvm_arm_vcpu_init(cpu);   ---->[1]
>>>>           if (ret) {
>>>>              return ret;
>>>>          }
>>>>             if (cpu_isar_feature(aa64_sve, cpu)) {
>>>>           ret = kvm_arm_sve_set_vls(cpu); ---->[2]
>>>>           if (ret) {
>>>>               return ret;
>>>>           }
>>>>           ret = kvm_arm_vcpu_finalize(cpu, KVM_ARM_VCPU_SVE);--->[3]
>>>>           if (ret) {
>>>>               return ret;
>>>>           }
>>>>       }
>>>> [...]
>>>> }
>>>>
>>>> I think it's happening because vector length is going uninitialized.
>>>> This initialization
>>>> happens in context to  arm_cpu_finalize_features() which I forgot to call before
>>>> calling KVM finalize.
>>>>
>>>>>
>>>>> kvm_arch_init_vcpu() is supposed to be called in the realization path in current
>>>>> implementation (without this series) because the parameters (features) to KVM_ARM_VCPU_INIT
>>>>> is populated at vCPU realization time.
>>>>
>>>> Not necessarily. It is just meant to initialize the KVM. If we take care of the
>>>> KVM requirements in the similar way the realize path does we should be
>>>> fine. Can you try to add the patch below in your code and test if it works?
>>>>
>>>>    diff --git a/target/arm/kvm.c b/target/arm/kvm.c
>>>> index c4b68a0b17..1091593478 100644
>>>> --- a/target/arm/kvm.c
>>>> +++ b/target/arm/kvm.c
>>>> @@ -1068,6 +1068,9 @@ void kvm_arm_create_host_vcpu(ARMCPU *cpu)
>>>>            abort();
>>>>        }
>>>>
>>>> +     /* finalize the features like SVE, SME etc */
>>>> +     arm_cpu_finalize_features(cpu, &error_abort);
>>>> +
>>>>        /*
>>>>         * Initialize the vCPU in the host. This will reset the sys regs
>>>>         * for this vCPU and related registers like MPIDR_EL1 etc. also
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> $ home/gavin/sandbox/qemu.main/build/qemu-system-aarch64           \
>>>>>      --enable-kvm -machine virt,gic-version=3 -cpu host               \
>>>>>      -smp cpus=4,disabledcpus=2 -m 1024M                              \
>>>>>      -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image    \
>>>>>      -initrd /home/gavin/sandbox/images/rootfs.cpio.xz -nographic
>>>>> qemu-system-aarch64: Failed to initialize host vcpu 4
>>>>> Aborted (core dumped)
>>>>>
>>>>> Backtrace
>>>>> =========
>>>>> (gdb) bt
>>>>> #0  0x0000ffff9106bc80 in __pthread_kill_implementation () at /lib64/libc.so.6
>>>>> #1  0x0000ffff9101aa40 [PAC] in raise () at /lib64/libc.so.6
>>>>> #2  0x0000ffff91005988 [PAC] in abort () at /lib64/libc.so.6
>>>>> #3  0x0000aaaab1cc26b8 [PAC] in kvm_arm_create_host_vcpu (cpu=0xaaaab9ab1bc0)
>>>>>        at ../target/arm/kvm.c:1081
>>>>> #4  0x0000aaaab1cd0c94 in virt_setup_lazy_vcpu_realization (cpuobj=0xaaaab9ab1bc0, vms=0xaaaab98870a0)
>>>>>        at ../hw/arm/virt.c:2483
>>>>> #5  0x0000aaaab1cd180c in machvirt_init (machine=0xaaaab98870a0) at ../hw/arm/virt.c:2777
>>>>> #6  0x0000aaaab160f220 in machine_run_board_init
>>>>>        (machine=0xaaaab98870a0, mem_path=0x0, errp=0xfffffa86bdc8) at ../hw/core/machine.c:1722
>>>>> #7  0x0000aaaab1a25ef4 in qemu_init_board () at ../system/vl.c:2723
>>>>> #8  0x0000aaaab1a2635c in qmp_x_exit_preconfig (errp=0xaaaab38a50f0 <error_fatal>)
>>>>>        at ../system/vl.c:2821
>>>>> #9  0x0000aaaab1a28b08 in qemu_init (argc=15, argv=0xfffffa86c1f8) at ../system/vl.c:3882
>>>>> #10 0x0000aaaab221d9e4 in main (argc=15, argv=0xfffffa86c1f8) at ../system/main.c:71
>>>>
>>>>
>>>> Thank you for this. Please let me know if the above fix works and also
>>>> the return values in
>>>> case you encounter errors.
>>>
>>> I've pushed the fix to below branch for your convenience:
>>>
>>> Branch: https://github.com/salil-mehta/qemu/commits/virt-cpuhp-armv8/rfc-v6.2
>>> Fix: https://github.com/salil-mehta/qemu/commit/1f1fbc0998ffb1fe26140df3c336bf2be2aa8669
>>>
>>
>> I guess rfc-v6.2 branch isn't ready for test because it runs into another crash
>> dump with rfc-v6.2 branch, like below.
> 
> 
> rfc-6.2 is not crashing on Kunpeng920 where I tested. But this
> chip does not have some ARM extensions like SVE etc so
> Unfortunately, I can't test SVE/SME/PAuth etc support.
> 
> Can you disable SVE and then try if it comes up just to corner
> the case?
> 

Right, this crash dump shouldn't be encountered if SVE isn't supported. I already
had the workaround "-cpu host,sve=off" to keep my tests moving forwards...

>>
>> host$ /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                     \
>>         -accel kvm -machine virt,gic-version=host,nvdimm=on                         \
>>         -cpu host,sve=on                                                            \
>>         -smp maxcpus=4,cpus=2,disabledcpus=2,sockets=2,clusters=2,cores=1,threads=1 \
>>         -m 4096M,slots=16,maxmem=128G                                               \
>>         -object memory-backend-ram,id=mem0,size=2048M                               \
>>         -object memory-backend-ram,id=mem1,size=2048M                               \
>>         -numa node,nodeid=0,memdev=mem0,cpus=0-1                                    \
>>         -numa node,nodeid=1,memdev=mem1,cpus=2-3                                    \
>>         -L /home/gavin/sandbox/qemu.main/build/pc-bios                              \
>>         -monitor none -serial mon:stdio -nographic -gdb tcp::6666                   \
>>         -qmp tcp:localhost:5555,server,wait=off                                     \
>>         -bios /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd      \
>>         -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image               \
>>         -initrd /home/gavin/sandbox/images/rootfs.cpio.xz                           \
>>         -append memhp_default_state=online_movable
>>           :
>>           :
>> guest$ cd /sys/devices/system/cpu/
>> guest$ cat present enabled online
>> 0-3
>> 0-1
>> 0-1
>> (qemu) device_set host-arm-cpu,socket-id=1,cluster-id=0,core-id=0,thread-id=0,admin-state=enable
>> qemu-system-aarch64: kvm_init_vcpu: kvm_arch_init_vcpu failed (2): Operation not permitted
> 
> 
> Ah, I see. I think I understand the issue. It's complaining
> about calling the  finalize twice. Is it possible to check as
> I do not have a way to test it?
> 
> 
> int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature)
> {
> switch (feature) {
> case KVM_ARM_VCPU_SVE:
> [...]
> if (kvm_arm_vcpu_sve_finalized(vcpu))
> return -EPERM;-----> this where it must be popping?
> [...]
> }
> 

Right, I think that's the case: QEMU tries to finalize SVE capability for twice,
which is the real problem. I'm explaining what I found as below, which would be
helpful to the forthcoming revisions.

machvirt_init
   virt_setup_lazy_vcpu_realization
     arm_cpu_finalize_features
     kvm_arm_create_host_vcpu
       kvm_create_vcpu                       // New fd is created
       kvm_arch_init_vcpu
         kvm_arm_vcpu_init
         kvm_arm_sve_set_vls
         kvm_arm_vcpu_finalize               // (A) SVE capability is finalized

device_set_admin_power_state
   device_pre_poweron
     virt_machine_device_pre_poweron
       virt_cpu_pre_poweron
         qdev_realize
           arm_cpu_realizefn
             cpu_exec_realizefn
             arm_cpu_finalize_features       // Called for the second time
             qemu_init_vcpu
               kvm_start_vcpu_thread
                 kvm_vcpu_thread_fn
                   kvm_init_vcpu
                     kvm_create_vcpu         // Called for the second time
                     kvm_arch_init_vcpu      // Called for the second time
                       kvm_arm_vcpu_init
                       kvm_arm_sve_set_vls   // (B) Failed here
                       kvm_arm_vcpu_finalize

(B) where we try to finalize SVE capability again. It has been finalized at (A)
     Fianlizing SVE capability for twice is disallowed by KVM on the host side.


>>
>> I picked the fix (the last patch in rfc-v6.2 branch) to rfc-v6 branch, same crash dump
>> can be seen.
> 
> Are you getting previously reported abort or above new problem?
> 

Previously, the VM can't be started. After your fix is applied, the VM is able to start.
It's a new problem that qemu crash dump is seens on attempt to hot add a vCPU.

Thanks,
Gavin