Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang <wangyize7@huawei.com>]

Kindly Ping

在 2026/10/8 20:50, Yize Wang 写道:
>
> 在 2026/10/3 22:53, Marc Zyngier 写道:
>> On Sun, 20 Sep 2026 10:15:25 +0100,
>> Yize Wang <wangyize7@huawei.com> wrote:
>>> 在 2026/9/18 20:08, Marc Zyngier 写道:
>>>> On Fri, 18 Sep 2026 09:18:13 +0100,
>>>> Yize Wang<wangyize7@huawei.com> wrote:
>>>>> This series adds batch register access support to KVM/arm64 to reduce
>>>>> syscall overhead during VM live migration.
>>>>>
>>>>> Currently, QEMU issues one ioctl per register when 
>>>>> saving/restoring VGIC
>>>>> state. On large VM configurations this means tens of thousands of 
>>>>> syscalls,
>>>>> where lock acquisition and context switch overhead dominates 
>>>>> migration
>>>>> downtime. Thus, we provide a batch register method to allow userspace
>>>>> read/write multiple distributor and redistributor registers in a 
>>>>> single call.
>>>>> In this way, we can significantly reduce syscalls and migration 
>>>>> downtime.
>>>>>
>>>>> Test the VM migration time under pressure conditions.
>>>>> The VM specifications for migration are as follows:
>>>>> - VM use 4-K page;
>>>>> - the number of VCPU is 160;
>>>>> - the total memory is 320Gigabit;
>>>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>>>
>>>>> Performance results (3-run average, ms):
>>>>>       | Metric              | Without patch | With patch | 
>>>>> Improvement |
>>>>> |---------------------|---------------|------------|-------------|
>>>>>       | Migration downtime  |        536    |     321 |     40%     |
>>>>>       | Source (total)      |        344    |     230 |     33%     |
>>>>>       |   - VGIC put        |        158    |      40 |     75%     |
>>>>>       |   - VGIC get        |        120    |      19 |     84%     |
>>>>>       | Destination (total) |        192    |      91 |     53%     |
>>>>>       |   - VGIC put        |        132    |      27 |     80%     |
>>>>>
>>>>> Yize Wang (2):
>>>>>     KVM: arm64: Add batch group constant and data structure to 
>>>>> UAPI header
>>>>>     KVM: arm64: Add VGIC v3 batch register access implementation
>>>> Questions:
>>>>
>>>> - Why only the MMIO registers?
>>>>
>>>> - Why not the sysregs?
>>>>
>>>> - Why only the GIC?
>>>>
>>>> - Why not all of the state?
>>>>
>>>> - Where is the corresponding userspace code?
>>>>
>>>> More importantly, since this is about batching system calls:
>>>>
>>>> - Why can't this be done with io_uring instead?
>>>>
>>>>     M.
>>>
>>> Hi, Marc! Thank you for the review.
>>>
>>>    These patches focus on optimizing GICv3 register access during live
>>> migration. We found that there are a large number of locks (kvm->lock,
>>> vcpus, config_lock) in the GIC, these lock operations wil cost large
>>> time waste. The batches of sysreg for vcpu optimization will come in
>>> follow as a separate series. And let me address these questions one by
>>> one.
>> All these locks should be non-blocking by the time you save anything
>> related to the GIC, because:
>>
>> - none of the vcpu can be running
>>
>> - this must be a single threaded operation
>>
>> So if you are seeing anything contended, this is either the sign of a
>> bad KVM bug, or an indication that you are violating the above
>> requirements.
>>
>>     M.
>
>
> Hi, Marc! Thanks for your review.
>
> You are right that when saving the GIC state during live migration 
> process, all vCPUs are not running and it is a single threaded 
> operation. Our patches process the packaged register states from the 
> userspace under the same lock. These operations only aim to reduce the 
> lock opreation downtime and do not involve lock contention.
>
> In addition, these patches focus on reducing the syscalls of ioctl. 
> Now, every register requires an ioctl, resulting in (num_cpu * 
> registers_per_cpu) system calls. This will cost lots of time during 
> the large-scale VM  migration. Our patches batch registers by category 
> in qemu and then send them to kernel in batches. This operations 
> significantly reduce the number of ioctl syscalls and save a great 
> amount of downtime.
>
> The comparison is as follow:
> BEFORE:
>
>        QEMU (Userspace)                  Kernel (Kernelspace)
>   +------------------------+        +----------------------------+
>   |                        |        |                            |
>   |  kvm_arm_gicv3_get()   |        |  vgic_v3_attr_regs_access  |
>   |                        |        |                            |
>   |  for ncpu in num_cpu:  |        |                            |
>   |    for each register:  |        |                            |
>   |                        |        |                            |
>   |  GICR_CTLR             | ioctl->| mutex_lock(&kvm->lock)    |
>   |                        |        |  kvm_trylock_all_vcpus     |
>   |                        |        | mutex_lock(&config_lock)  |
>   |                        | <-val  |  read/write register       |
>   |                        |        |  mutex_unlock              |
>   |                        |        |  kvm_unlock_all_vcpus      |
>   |                        |        | mutex_unlock(&kvm->lock)  |
>   |                        |        |                            |
>   |  GICR_STATUSR          | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   |  GICR_WAKER            | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   |  ICC_SRE_EL1           | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   +------------------------+        +----------------------------+
>
>
> AFTER:
>
>        QEMU (Userspace)                  Kernel (Kernelspace)
>   +------------------------+        +----------------------------+
>   |                        |        |                            |
>   |  kvm_arm_gicv3_get()   |        |  vgic_v3_batch_access      |
>   |                        |        |                            |
>   |  Collect Phase:        |        |                            |
>   |    add(GICR_CTLR)      |        |                            |
>   |    add(GICR_STATUSR)   |        |                            |
>   |    add(GICR_WAKER)     |        |                            |
>   |    add(GICR_IGROUPR0)  |        |                            |
>   |    ...                 |        |                            |
>   |    add(ICC_SRE_EL1)    |        |                            |
>   |    ...                 |        |                            |
>   |                        |        |                            |
>   |  Commit Phase:         | BATCH->| copy_from_user            |
>   |    batch_commit()      |        | mutex_lock(&kvm->lock)    |
>   |                        |        |  kvm_trylock_all_vcpus     |
>   |                        |        | mutex_lock(&config_lock)  |
>   |                        |        |                            |
>   |                        |        |  Process Each Entry:       |
>   |                        |        |    entry[0]: GICR_CTLR     |
>   |                        |        |    entry[1]: GICR_STATUSR  |
>   |                        |        |    entry[2]: GICR_WAKER    |
>   |                        |        |    ...                     |
>   |                        |        |    entry[N]: ICC_SRE       |
>   |                        |        |    FLAG_LOCKED: skip       |
>   |                        |        |                            |
>   |  Unpack Phase:         |<-result| mutex_unlock              |
>   |    Write back          |        |  kvm_unlock_all_vcpus      |
>   |    Post-process:       |        | mutex_unlock(&kvm->lock)  |
>   |      unshuffle,split   |        |  copy_to_user              |
>   |                        |        |                            |
>   +------------------------+        +----------------------------+