Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code

[David Hildenbrand <david@redhat.com>]

On 09.10.23 15:42, Salil Mehta wrote:
> Hi David,
> Thanks for the review.
> 
>> From: David Hildenbrand <david@redhat.com>
>> Sent: Monday, October 9, 2023 1:21 PM
>> To: Salil Mehta <salil.mehta@huawei.com>; qemu-devel@nongnu.org; qemu-arm@nongnu.org
>> Cc: maz@kernel.org; jean-philippe@linaro.org; Jonathan Cameron
>> <jonathan.cameron@huawei.com>; lpieralisi@kernel.org;
>> peter.maydell@linaro.org; richard.henderson@linaro.org;
>> imammedo@redhat.com; andrew.jones@linux.dev; philmd@linaro.org;
>> eric.auger@redhat.com; oliver.upton@linux.dev; pbonzini@redhat.com;
>> mst@redhat.com; will@kernel.org; gshan@redhat.com; rafael@kernel.org;
>> alex.bennee@linaro.org; linux@armlinux.org.uk;
>> darren@os.amperecomputing.com; ilkka@os.amperecomputing.com;
>> vishnu@os.amperecomputing.com; karl.heubaum@oracle.com;
>> miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian
>> <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>;
>> wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com;
>> maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com>
>> Subject: Re: [PATCH V3 01/10] accel/kvm: Extract common KVM vCPU
>> {creation,parking} code
>>
>> On 09.10.23 13:28, Salil Mehta wrote:
>>> KVM vCPU creation is done once during the initialization of the VM when Qemu
>>> thread is spawned. This is common to all the architectures.
>>>
>>> Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the
>>> corresponding KVM vCPU object in the Host KVM is not destroyed and its
>>> representative KVM vCPU object/context in Qemu is parked.
>>>
>>> Refactor common logic so that some APIs could be reused by vCPU Hotplug code.
>>>
>>> Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
>>
>> [...]
>>
>>>
>>>    int kvm_init_vcpu(CPUState *cpu, Error **errp)
>>> @@ -395,19 +434,14 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp)
>>>
>>>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
>>>
>>> -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
>>> +    ret = kvm_create_vcpu(cpu);
>>>        if (ret < 0) {
>>> -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed (%lu)",
>>> +        error_setg_errno(errp, -ret,
>>> +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
>>
>> Unrelated change.
> 
> 
> It is related. I think you missed kvm_get_vcpu -> kvm_create_vcpu change
> in the string.

Indeed, I did :)

> 
> 
>>>                             kvm_arch_vcpu_id(cpu));
>>>            goto err;
>>>        }
>>>
>>> -    cpu->kvm_fd = ret;
>>> -    cpu->kvm_state = s;
>>> -    cpu->vcpu_dirty = true;
>>> -    cpu->dirty_pages = 0;
>>> -    cpu->throttle_us_per_full = 0;
>>> -
>>>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
>>>        if (mmap_size < 0) {
>>>            ret = mmap_size;
>>> diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events
>>> index 399aaeb0ec..08e2dc253f 100644
>>> --- a/accel/kvm/trace-events
>>> +++ b/accel/kvm/trace-events
>>> @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
>>>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
>>>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
>>>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
>>> +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "creating KVM cpu: cpu_index: %d arch vcpu-id: %lu"
>>> +kvm_get_vcpu(unsigned long arch_cpu_id) "unparking KVM vcpu: arch vcpu-id: %lu"
>>> +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "destroy vcpu: cpu_index: %d arch vcpu-id: %lu"
>>> +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "parking KVM vcpu: cpu_index: %d arch vcpu-id: %lu"
>>
>> It's a bit confusing that there is now
>>
>> 1) create (create new or return parked)
>> 2) destroy (cleanup + park)
>> 3) park (park only)
>>
>> Why would one use 2) instead of 3) or the other way around? But I
>> suspect that kvm_destroy_vcpu() is only supposed to be a KVM-internal
>> helper ...
> 
> kvm_destroy_vcpu is more than just parking:
> 
> 1. Arch destroy vcpu
> 2. Unmap cpu->kvm_run
> 3. Parking logic
> 
> To support virtual CPU Hotplug on ARM platforms we pre-create all
> the KVM vCPUs but their corresponding Qemu threads are not spawned
> (and hence cpu->kvm_run is not mapped). Unplugged vCPUs remains
> parked in the list. Hence, only step-3 is required.

IIUC, your current flow is going to be

1) Create
2) Park
3) Create [which ends up reusing the parked VCPU]
4) Destroy [when unplugging the CPU]

If that's the case, that API really is suboptimal.

What speaks against an API that models 1) and 2) in a single step

kvm_precreate_vcpu
kvm_create_vcpu
kvm_destroy_vcpu

One could even make kvm_create_vcpu() fail on ARM if the VCPU hasn't 
been pre-created.

Or did I get it all wrong? :)

-- 
Cheers,

David / dhildenb