[PATCH 0/2] Batch register access for live migration optimization

[PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

This series adds batch register access support to KVM/arm64 to reduce
syscall overhead during VM live migration.

Currently, QEMU issues one ioctl per register when saving/restoring VGIC
state. On large VM configurations this means tens of thousands of syscalls,
where lock acquisition and context switch overhead dominates migration
downtime. Thus, we provide a batch register method to allow userspace
read/write multiple distributor and redistributor registers in a single call.
In this way, we can significantly reduce syscalls and migration downtime.

Test the VM migration time under pressure conditions.
The VM specifications for migration are as follows:
- VM use 4-K page;
- the number of VCPU is 160;
- the total memory is 320Gigabit;
- use 'Redis SET-benchmark' to pressurize VM;

Performance results (3-run average, ms):
    | Metric              | Without patch | With patch | Improvement |
    |---------------------|---------------|------------|-------------|
    | Migration downtime  |        536    |     321    |     40%     |
    | Source (total)      |        344    |     230    |     33%     |
    |   - VGIC put        |        158    |      40    |     75%     |
    |   - VGIC get        |        120    |      19    |     84%     |
    | Destination (total) |        192    |      91    |     53%     |
    |   - VGIC put        |        132    |      27    |     80%     |

Yize Wang (2):
  KVM: arm64: Add batch group constant and data structure to UAPI header
  KVM: arm64: Add VGIC v3 batch register access implementation

 arch/arm64/include/uapi/asm/kvm.h     |  10 +++
 arch/arm64/kvm/vgic/vgic-kvm-device.c | 113 +++++++++++++++++++++++++++++++---
 2 files changed, 114 insertions(+), 9 deletions(-)
 

[PATCH 1/2] KVM: arm64: Add batch group constant and data structure to UAPI header

[Yize Wang]

Add KVM_DEV_ARM_VGIC_GRP_BATCH_REGS group ID, KVM_DEV_ARM_VGIC_BATCH_MAX
limit, and struct kvm_dev_arm_vgic_batch_entry for batch VGIC register
access. These UAPI definitions allow userspace to read/write multiple
VGIC registers in a single ioctl call.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Signed-off-by: Yize Wang <wangyize7@huawei.com>
---
 arch/arm64/include/uapi/asm/kvm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 019e5e3d892e..f84530274edd 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -418,12 +418,22 @@ enum {
 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
 #define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ  9
+#define KVM_DEV_ARM_VGIC_GRP_BATCH_REGS 10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
 			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
 #define VGIC_LEVEL_INFO_LINE_LEVEL	0
 
+#define KVM_DEV_ARM_VGIC_BATCH_MAX	4096
+#define KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED 12
+struct kvm_dev_arm_vgic_batch_entry {
+	__u32 group;
+	__u64 attr;
+	__u32 val;
+	__s32 error;
+};
+
 #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
 #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
 #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2
-- 
2.43.0

Re: [PATCH 1/2] KVM: arm64: Add batch group constant and data structure to UAPI header

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] kvm: arm64: implicit padding hole in UAPI struct kvm_dev_arm_vgic_batch_entry
--

commit 54edd6d460b56fe24bcc0c18ebb7ce4904b10a69
Author: Yize Wang <wangyize7@huawei.com>

KVM: arm64: Add batch group constant and data structure to UAPI header

Add KVM_DEV_ARM_VGIC_GRP_BATCH_REGS group ID, KVM_DEV_ARM_VGIC_BATCH_MAX
limit, and struct kvm_dev_arm_vgic_batch_entry for batch VGIC register
access. These UAPI definitions allow userspace to read/write multiple
VGIC registers in a single ioctl call.

> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index 019e5e3d892e6..f84530274edd4 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
[ ... ]
> @@ -418,12 +418,22 @@ enum {
>  #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO  7
>  #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
>  #define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ  9
> +#define KVM_DEV_ARM_VGIC_GRP_BATCH_REGS 10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT	10
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
>  			(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
>  #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK	0x3ff
>  #define VGIC_LEVEL_INFO_LINE_LEVEL	0
>  
> +#define KVM_DEV_ARM_VGIC_BATCH_MAX	4096
> +#define KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED 12
> +struct kvm_dev_arm_vgic_batch_entry {
> +	__u32 group;
> +	__u64 attr;

[Severity: Low]
Does this struct layout introduce a 4-byte implicit padding hole between
the group and attr fields?

This layout appears to violate UAPI design guidelines, which strictly forbid
implicit padding to prevent ABI sizing mismatches and potential information
leaks.

While the kernel currently never copies the structure array back to userspace
(it only writes to val directly via put_user), and 64-bit integers are aligned
to 8 bytes on ARM32 compat (avoiding an immediate sizing mismatch), defining
this structure permanently exposes the flawed layout to userspace.

Could explicit padding be added or the fields reordered to ensure the
layout is perfectly packed without implicit holes?

> +	__u32 val;
> +	__s32 error;
> +};
> +
>  #define   KVM_DEV_ARM_VGIC_CTRL_INIT		0
>  #define   KVM_DEV_ARM_ITS_SAVE_TABLES           1
>  #define   KVM_DEV_ARM_ITS_RESTORE_TABLES        2

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260918081930.4014735-1-wangyize7@huawei.com?part=1

[PATCH 2/2] KVM: arm64: Add VGIC v3 batch register access implementation

[Yize Wang]

Add vgic_v3_batch_access() to allow userspace to read or write
multiple VGIC registers in a single system call.

Introduce KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED to modify the locking
and unlocking logic in vgic_v3_attr_regs_access to prevent repeated
locking and unlocking.

Wire up KVM_DEV_ARM_VGIC_GRP_BATCH_REGS in vgic_v3_set_attr(),
vgic_v3_get_attr(), and vgic_v3_has_attr().

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Signed-off-by: Yize Wang <wangyize7@huawei.com>
---
 arch/arm64/kvm/vgic/vgic-kvm-device.c | 113 ++++++++++++++++++++++++--
 1 file changed, 104 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c
index 90be99443df3..ef0b2e8f68b2 100644
--- a/arch/arm64/kvm/vgic/vgic-kvm-device.c
+++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c
@@ -566,14 +566,21 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev,
 			return -EFAULT;
 	}
 
-	mutex_lock(&dev->kvm->lock);
+	/* If the caller has not indicated that it already holds the necessary locks
+	 * (e.g., via KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED), we must acquire the VM lock
+	 * and pause all VCPUs to ensure consistent access to VGIC state. This prevents
+	 * race conditions with concurrent interrupt injection or VCPU execution.
+	 */
+	if (attr->flags != KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED) {
+		mutex_lock(&dev->kvm->lock);
 
-	if (kvm_trylock_all_vcpus(dev->kvm)) {
-		mutex_unlock(&dev->kvm->lock);
-		return -EBUSY;
-	}
+		if (kvm_trylock_all_vcpus(dev->kvm)) {
+			mutex_unlock(&dev->kvm->lock);
+			return -EBUSY;
+		}
 
-	mutex_lock(&dev->kvm->arch.config_lock);
+		mutex_lock(&dev->kvm->arch.config_lock);
+	}
 
 	if (!(vgic_initialized(dev->kvm) || reg_allowed_pre_init(attr))) {
 		ret = -EBUSY;
@@ -611,9 +618,15 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev,
 	}
 
 out:
-	mutex_unlock(&dev->kvm->arch.config_lock);
-	kvm_unlock_all_vcpus(dev->kvm);
-	mutex_unlock(&dev->kvm->lock);
+	/* Release the locks acquired at the beginning of this function,
+	 * but only if we actually acquired them.
+	 * The order of unlocking must be the reverse of locking.
+	 */
+	if (attr->flags != KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED) {
+		mutex_unlock(&dev->kvm->arch.config_lock);
+		kvm_unlock_all_vcpus(dev->kvm);
+		mutex_unlock(&dev->kvm->lock);
+	}
 
 	if (!ret && uaccess && !is_write) {
 		u32 __user *uaddr = (u32 __user *)(unsigned long)attr->addr;
@@ -623,6 +636,81 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev,
 	return ret;
 }
 
+/* vgic_v3_batch_access - Batch access to VGIC registers
+ *
+ * @dev: The VGIC device
+ * @attr: The device attribute containing the batch entries
+ * @is_write: True for write, false for read
+ *
+ * This function allows userspace to read or write multiple VGIC registers
+ * in a single system call, reducing overhead during live migration.
+ *
+ * Returns: 0 on success, or the first error encountered.
+ * Individual entry errors are stored in entries[i].error.
+ */
+static int vgic_v3_batch_access(struct kvm_device *dev,
+				struct kvm_device_attr *attr, bool is_write)
+{
+	struct kvm_dev_arm_vgic_batch_entry __user *uentries;
+	struct kvm_dev_arm_vgic_batch_entry *entries;
+	struct kvm_device_attr sub_attr;
+	u32 count = attr->attr;
+	int ret = 0, i;
+
+	if (!count || count > KVM_DEV_ARM_VGIC_BATCH_MAX)
+		return -EINVAL;
+
+	entries = kvmalloc_array(count, sizeof(*entries), GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	uentries = (void __user *)attr->addr;
+	if (copy_from_user(entries, uentries, count * sizeof(*entries))) {
+		kvfree(entries);
+		return -EFAULT;
+	}
+
+	mutex_lock(&dev->kvm->lock);
+	if (kvm_trylock_all_vcpus(dev->kvm)) {
+		mutex_unlock(&dev->kvm->lock);
+		kvfree(entries);
+		return -EBUSY;
+	}
+	mutex_lock(&dev->kvm->arch.config_lock);
+
+	/* Process each entry in the batch */
+	for (i = 0; i < count; i++) {
+		int err;
+
+		sub_attr.group = entries[i].group;
+		sub_attr.attr = entries[i].attr;
+		sub_attr.addr = (unsigned long)uentries +
+			i * sizeof(struct kvm_dev_arm_vgic_batch_entry) +
+			offsetof(struct kvm_dev_arm_vgic_batch_entry, val);
+
+		/* Set flag to indicate that locks are already held.
+		 * This prevents vgic_v3_attr_regs_access from acquiring locks again,
+		 * avoiding deadlock and overhead.
+		 */
+		sub_attr.flags = KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED;
+
+		err = vgic_v3_attr_regs_access(dev, &sub_attr, is_write);
+
+		if (err) {
+			pr_err("group %d attr %lld get err %d\n",
+				sub_attr.group, sub_attr.attr, err);
+			ret = err;
+		}
+	}
+
+	mutex_unlock(&dev->kvm->arch.config_lock);
+	kvm_unlock_all_vcpus(dev->kvm);
+	mutex_unlock(&dev->kvm->lock);
+
+	kvfree(entries);
+	return ret;
+}
+
 static int vgic_v3_set_attr(struct kvm_device *dev,
 			    struct kvm_device_attr *attr)
 {
@@ -649,6 +737,8 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
 		dev->kvm->arch.vgic.mi_intid = val;
 		return 0;
 	}
+	case KVM_DEV_ARM_VGIC_GRP_BATCH_REGS:
+		return vgic_v3_batch_access(dev, attr, true);
 	default:
 		return vgic_set_common_attr(dev, attr);
 	}
@@ -669,6 +759,8 @@ static int vgic_v3_get_attr(struct kvm_device *dev,
 		guard(mutex)(&dev->kvm->arch.config_lock);
 		return put_user(dev->kvm->arch.vgic.mi_intid, uaddr);
 	}
+	case KVM_DEV_ARM_VGIC_GRP_BATCH_REGS:
+		return vgic_v3_batch_access(dev, attr, false);
 	default:
 		return vgic_get_common_attr(dev, attr);
 	}
@@ -707,6 +799,9 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
 		case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
 			return 0;
 		}
+		break;
+	case KVM_DEV_ARM_VGIC_GRP_BATCH_REGS:
+		return 0;
 	}
 	return -ENXIO;
 }
-- 
2.43.0

Re: [PATCH 2/2] KVM: arm64: Add VGIC v3 batch register access implementation

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] kvm: arm64: vgic: allow userspace to bypass VGIC locks via KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED
- [High] kvm: arm64: vgic: user memory access under VM locks in batch access enables deadlock
- [Medium] kvm: arm64: vgic: un-ratelimited pr_err in batch access allows userspace log flooding
- [Medium] kvm: arm64: vgic: batch access fails to return first error or copy results to userspace
--

commit 4bb3fc1e279eb19b2c3ca453e47c56bcab743a11
Author: Yize Wang <wangyize7@huawei.com>

KVM: arm64: Add VGIC v3 batch register access implementation

This commit adds vgic_v3_batch_access() to allow userspace to read or write
multiple VGIC registers in a single system call. It introduces a new flag
KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED to prevent repeated locking and unlocking
during these batch accesses.

> diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c
> --- a/arch/arm64/kvm/vgic/vgic-kvm-device.c
> +++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c
> @@ -566,14 +566,21 @@ static int vgic_v3_attr_regs_access(struct kvm_device *dev,
>  			return -EFAULT;
>  	}
>  
> -	mutex_lock(&dev->kvm->lock);
> +	/* If the caller has not indicated that it already holds the necessary locks
> +	 * (e.g., via KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED), we must acquire the VM lock
> +	 * and pause all VCPUs to ensure consistent access to VGIC state. This prevents
> +	 * race conditions with concurrent interrupt injection or VCPU execution.
> +	 */
> +	if (attr->flags != KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED) {
> +		mutex_lock(&dev->kvm->lock);

[Severity: Critical]
Can userspace bypass the VGIC data structure locks by setting this flag
directly?

If a userspace process invokes KVM_SET_DEVICE_ATTR or KVM_GET_DEVICE_ATTR
with attr->flags explicitly set to KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED,
vgic_v3_attr_regs_access() will skip acquiring kvm->lock,
kvm->arch.config_lock, and all VCPU mutexes.

Could this allow concurrent modifications to internal VGIC state, leading
to race conditions or use-after-free regressions?

[ ... ]

> +/* vgic_v3_batch_access - Batch access to VGIC registers
> + *
> + * @dev: The VGIC device
> + * @attr: The device attribute containing the batch entries
> + * @is_write: True for write, false for read
> + *
> + * This function allows userspace to read or write multiple VGIC registers
> + * in a single system call, reducing overhead during live migration.
> + *
> + * Returns: 0 on success, or the first error encountered.
> + * Individual entry errors are stored in entries[i].error.
> + */
> +static int vgic_v3_batch_access(struct kvm_device *dev,
> +				struct kvm_device_attr *attr, bool is_write)
> +{
[ ... ]
> +	mutex_lock(&dev->kvm->lock);
> +	if (kvm_trylock_all_vcpus(dev->kvm)) {
> +		mutex_unlock(&dev->kvm->lock);
> +		kvfree(entries);
> +		return -EBUSY;
> +	}
> +	mutex_lock(&dev->kvm->arch.config_lock);
> +
> +	/* Process each entry in the batch */
> +	for (i = 0; i < count; i++) {
> +		int err;
> +
> +		sub_attr.group = entries[i].group;
> +		sub_attr.attr = entries[i].attr;
> +		sub_attr.addr = (unsigned long)uentries +
> +			i * sizeof(struct kvm_dev_arm_vgic_batch_entry) +
> +			offsetof(struct kvm_dev_arm_vgic_batch_entry, val);
> +
> +		/* Set flag to indicate that locks are already held.
> +		 * This prevents vgic_v3_attr_regs_access from acquiring locks again,
> +		 * avoiding deadlock and overhead.
> +		 */
> +		sub_attr.flags = KVM_DEV_ARM_VGIC_ATTR_FLAG_LOCKED;
> +
> +		err = vgic_v3_attr_regs_access(dev, &sub_attr, is_write);

[Severity: High]
Does executing a user memory access while holding VM locks introduce a
deadlock regression?

Here sub_attr.addr is assigned a userspace address (uentries), which is
then passed into vgic_v3_attr_regs_access(). Inside
vgic_v3_attr_regs_access(), this address is accessed using put_user()
or get_user():

    if (!ret && uaccess && !is_write) {
        u32 __user *uaddr = (u32 __user *)(unsigned long)attr->addr;
        ret = put_user(val, uaddr);
    }

Since the batch caller holds kvm->lock, kvm->arch.config_lock, and all
vcpu->mutexes, could an untrusted userspace process point uentries to
userfaultfd-managed memory and intentionally stall the page fault? This
would keep all global VM and VCPU locks held indefinitely, hanging the VM.

> +
> +		if (err) {
> +			pr_err("group %d attr %lld get err %d\n",
> +				sub_attr.group, sub_attr.attr, err);

[Severity: Medium]
Is there a risk of userspace flooding the host kernel log buffer here?

This is an unconditional pr_err() driven by user-supplied inputs. Since
there is no rate limiting and the loop does not abort on error, a batch of
up to 4096 invalid VGIC attribute entries could spam dmesg and degrade
host performance.

> +			ret = err;
> +		}
> +	}
> +
> +	mutex_unlock(&dev->kvm->arch.config_lock);
> +	kvm_unlock_all_vcpus(dev->kvm);
> +	mutex_unlock(&dev->kvm->lock);
> +
> +	kvfree(entries);
> +	return ret;
> +}

[Severity: Medium]
Does this code fail to return the first error or copy the results back to
userspace?

The function header indicates it should return the first error and store
individual errors in entries[i].error. However, the loop overwrites ret
with the most recent error instead of returning immediately.

Furthermore, the entries array is freed here without being copied back to
user memory, meaning userspace has no way to determine which register
access actually failed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260918081930.4014735-1-wangyize7@huawei.com?part=2

Re: [PATCH 0/2] Batch register access for live migration optimization

[Marc Zyngier]

On Fri, 18 Sep 2026 09:18:13 +0100,
Yize Wang <wangyize7@huawei.com> wrote:
> 
> This series adds batch register access support to KVM/arm64 to reduce
> syscall overhead during VM live migration.
> 
> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
> state. On large VM configurations this means tens of thousands of syscalls,
> where lock acquisition and context switch overhead dominates migration
> downtime. Thus, we provide a batch register method to allow userspace
> read/write multiple distributor and redistributor registers in a single call.
> In this way, we can significantly reduce syscalls and migration downtime.
> 
> Test the VM migration time under pressure conditions.
> The VM specifications for migration are as follows:
> - VM use 4-K page;
> - the number of VCPU is 160;
> - the total memory is 320Gigabit;
> - use 'Redis SET-benchmark' to pressurize VM;
> 
> Performance results (3-run average, ms):
>     | Metric              | Without patch | With patch | Improvement |
>     |---------------------|---------------|------------|-------------|
>     | Migration downtime  |        536    |     321    |     40%     |
>     | Source (total)      |        344    |     230    |     33%     |
>     |   - VGIC put        |        158    |      40    |     75%     |
>     |   - VGIC get        |        120    |      19    |     84%     |
>     | Destination (total) |        192    |      91    |     53%     |
>     |   - VGIC put        |        132    |      27    |     80%     |
> 
> Yize Wang (2):
>   KVM: arm64: Add batch group constant and data structure to UAPI header
>   KVM: arm64: Add VGIC v3 batch register access implementation

Questions:

- Why only the MMIO registers?

- Why not the sysregs?

- Why only the GIC?

- Why not all of the state?

- Where is the corresponding userspace code?

More importantly, since this is about batching system calls:

- Why can't this be done with io_uring instead?

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

在 2026/9/18 20:08, Marc Zyngier 写道:
> On Fri, 18 Sep 2026 09:18:13 +0100,
> Yize Wang <wangyize7@huawei.com> wrote:
>> This series adds batch register access support to KVM/arm64 to reduce
>> syscall overhead during VM live migration.
>>
>> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
>> state. On large VM configurations this means tens of thousands of syscalls,
>> where lock acquisition and context switch overhead dominates migration
>> downtime. Thus, we provide a batch register method to allow userspace
>> read/write multiple distributor and redistributor registers in a single call.
>> In this way, we can significantly reduce syscalls and migration downtime.
>>
>> Test the VM migration time under pressure conditions.
>> The VM specifications for migration are as follows:
>> - VM use 4-K page;
>> - the number of VCPU is 160;
>> - the total memory is 320Gigabit;
>> - use 'Redis SET-benchmark' to pressurize VM;
>>
>> Performance results (3-run average, ms):
>>      | Metric              | Without patch | With patch | Improvement |
>>      |---------------------|---------------|------------|-------------|
>>      | Migration downtime  |        536    |     321    |     40%     |
>>      | Source (total)      |        344    |     230    |     33%     |
>>      |   - VGIC put        |        158    |      40    |     75%     |
>>      |   - VGIC get        |        120    |      19    |     84%     |
>>      | Destination (total) |        192    |      91    |     53%     |
>>      |   - VGIC put        |        132    |      27    |     80%     |
>>
>> Yize Wang (2):
>>    KVM: arm64: Add batch group constant and data structure to UAPI header
>>    KVM: arm64: Add VGIC v3 batch register access implementation
> Questions:
>
> - Why only the MMIO registers?
>
> - Why not the sysregs?
>
> - Why only the GIC?
>
> - Why not all of the state?
>
> - Where is the corresponding userspace code?
>
> More importantly, since this is about batching system calls:
>
> - Why can't this be done with io_uring instead?
>
> 	M.


Hi, Marc! Thank you for the review.


These patches focus on optimizing GICv3 register access during live 
migration. We found that there are a large number of locks (kvm->lock, 
vcpus, config_lock) in the GIC, these lock operations wil cost large 
time waste. The batches of sysreg for vcpu optimization will come in 
follow as a separate series. And let me address these questions one by one.


1. Why only the MMIO registers?

In vgic_v3_batch_access(), we use 'entries' structure to implement batch 
read/write of register status. The structure is 'struct 
kvm_dev_arm_vgic_batch_entry', where the group information can be freely 
specified by userspace. Thus, vgic_v3_batch_access() supports all VGIC 
device attr groups, not only MMIO registers.


2. Why not the sysregs?

The newly added vgic_v3_batch_access() just forwards the groups received 
fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this function 
already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle sysregs. Our 
patch does not modify the existing sysreg handling logic.


3. Why only the GIC?

During live migration, the GIC is the device with the largest number of 
registers, and we found that there are a large number of locks in it. If 
each register is locked and unlocked individually, it would cause large 
time consumption. Thus, we want to optimize the GIC register time 
consumption during live migrate. The experimental results also show that 
batch processing significantly reduces VGIC handling time during migration.

Similarly, vCPU register save/restore also costs significantly time. The 
patch of vCPU sysreg batch access will be submitted separately in the 
future.


4. Why not all of the state?

As different devices have different lock hierarchies and access 
paths(e.g., GIC goes through the device fd, CPU regs go through the vcpu 
fd), it's difficult for us to realize in a single batch handler. 
Additionally, introducing too many changes at once would make review 
harder. These patches focus on GIC-related optimization, and a separate 
series for vCPU batch processing will follow.


5. Where is the corresponding userspace code?

The QEMU-side implementation has been posted to qemu-devel. The link is 
below:

https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t


6. Why can't this be done with io_uring instead?

The core idea of io_uring is to reduce the number of context switches 
between user space and kernel space by utilizing two ring buffers. 
However, each SQE is still processed independently in kernel space. For 
the VGIC registers, each SQE still require lock -> read/write -> unlock, 
so the per-register lock overhead remains unchanged. We hope to 
read/write a set of register states with a single lock operation to 
reduce the time cost. Therefore, io_uring does not meet our needs.

Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

kindly ping

在 2026/9/20 20:12, Yize Wang 写道:
>
> 在 2026/9/18 20:08, Marc Zyngier 写道:
>> On Fri, 18 Sep 2026 09:18:13 +0100,
>> Yize Wang <wangyize7@huawei.com> wrote:
>>> This series adds batch register access support to KVM/arm64 to reduce
>>> syscall overhead during VM live migration.
>>>
>>> Currently, QEMU issues one ioctl per register when saving/restoring 
>>> VGIC
>>> state. On large VM configurations this means tens of thousands of 
>>> syscalls,
>>> where lock acquisition and context switch overhead dominates migration
>>> downtime. Thus, we provide a batch register method to allow userspace
>>> read/write multiple distributor and redistributor registers in a 
>>> single call.
>>> In this way, we can significantly reduce syscalls and migration 
>>> downtime.
>>>
>>> Test the VM migration time under pressure conditions.
>>> The VM specifications for migration are as follows:
>>> - VM use 4-K page;
>>> - the number of VCPU is 160;
>>> - the total memory is 320Gigabit;
>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>
>>> Performance results (3-run average, ms):
>>>      | Metric              | Without patch | With patch | Improvement |
>>> |---------------------|---------------|------------|-------------|
>>>      | Migration downtime  |        536    |     321    | 40%     |
>>>      | Source (total)      |        344    |     230    | 33%     |
>>>      |   - VGIC put        |        158    |      40    | 75%     |
>>>      |   - VGIC get        |        120    |      19    | 84%     |
>>>      | Destination (total) |        192    |      91    | 53%     |
>>>      |   - VGIC put        |        132    |      27    | 80%     |
>>>
>>> Yize Wang (2):
>>>    KVM: arm64: Add batch group constant and data structure to UAPI 
>>> header
>>>    KVM: arm64: Add VGIC v3 batch register access implementation
>> Questions:
>>
>> - Why only the MMIO registers?
>>
>> - Why not the sysregs?
>>
>> - Why only the GIC?
>>
>> - Why not all of the state?
>>
>> - Where is the corresponding userspace code?
>>
>> More importantly, since this is about batching system calls:
>>
>> - Why can't this be done with io_uring instead?
>>
>>     M.
>
>
> Hi, Marc! Thank you for the review.
>
>
> These patches focus on optimizing GICv3 register access during live 
> migration. We found that there are a large number of locks (kvm->lock, 
> vcpus, config_lock) in the GIC, these lock operations wil cost large 
> time waste. The batches of sysreg for vcpu optimization will come in 
> follow as a separate series. And let me address these questions one by 
> one.
>
>
> 1. Why only the MMIO registers?
>
> In vgic_v3_batch_access(), we use 'entries' structure to implement 
> batch read/write of register status. The structure is 'struct 
> kvm_dev_arm_vgic_batch_entry', where the group information can be 
> freely specified by userspace. Thus, vgic_v3_batch_access() supports 
> all VGIC device attr groups, not only MMIO registers.
>
>
> 2. Why not the sysregs?
>
> The newly added vgic_v3_batch_access() just forwards the groups 
> received fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this 
> function already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle 
> sysregs. Our patch does not modify the existing sysreg handling logic.
>
>
> 3. Why only the GIC?
>
> During live migration, the GIC is the device with the largest number 
> of registers, and we found that there are a large number of locks in 
> it. If each register is locked and unlocked individually, it would 
> cause large time consumption. Thus, we want to optimize the GIC 
> register time consumption during live migrate. The experimental 
> results also show that batch processing significantly reduces VGIC 
> handling time during migration.
>
> Similarly, vCPU register save/restore also costs significantly time. 
> The patch of vCPU sysreg batch access will be submitted separately in 
> the future.
>
>
> 4. Why not all of the state?
>
> As different devices have different lock hierarchies and access 
> paths(e.g., GIC goes through the device fd, CPU regs go through the 
> vcpu fd), it's difficult for us to realize in a single batch handler. 
> Additionally, introducing too many changes at once would make review 
> harder. These patches focus on GIC-related optimization, and a 
> separate series for vCPU batch processing will follow.
>
>
> 5. Where is the corresponding userspace code?
>
> The QEMU-side implementation has been posted to qemu-devel. The link 
> is below:
>
> https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t 
>
>
>
> 6. Why can't this be done with io_uring instead?
>
> The core idea of io_uring is to reduce the number of context switches 
> between user space and kernel space by utilizing two ring buffers. 
> However, each SQE is still processed independently in kernel space. 
> For the VGIC registers, each SQE still require lock -> read/write -> 
> unlock, so the per-register lock overhead remains unchanged. We hope 
> to read/write a set of register states with a single lock operation to 
> reduce the time cost. Therefore, io_uring does not meet our needs.

Re: [PATCH 0/2] Batch register access for live migration optimization

[Marc Zyngier]

On Sun, 20 Sep 2026 10:15:25 +0100,
Yize Wang <wangyize7@huawei.com> wrote:
> 
> 在 2026/9/18 20:08, Marc Zyngier 写道:
> > On Fri, 18 Sep 2026 09:18:13 +0100,
> > Yize Wang<wangyize7@huawei.com> wrote:
> >> This series adds batch register access support to KVM/arm64 to reduce
> >> syscall overhead during VM live migration.
> >> 
> >> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
> >> state. On large VM configurations this means tens of thousands of syscalls,
> >> where lock acquisition and context switch overhead dominates migration
> >> downtime. Thus, we provide a batch register method to allow userspace
> >> read/write multiple distributor and redistributor registers in a single call.
> >> In this way, we can significantly reduce syscalls and migration downtime.
> >> 
> >> Test the VM migration time under pressure conditions.
> >> The VM specifications for migration are as follows:
> >> - VM use 4-K page;
> >> - the number of VCPU is 160;
> >> - the total memory is 320Gigabit;
> >> - use 'Redis SET-benchmark' to pressurize VM;
> >> 
> >> Performance results (3-run average, ms):
> >>      | Metric              | Without patch | With patch | Improvement |
> >>      |---------------------|---------------|------------|-------------|
> >>      | Migration downtime  |        536    |     321    |     40%     |
> >>      | Source (total)      |        344    |     230    |     33%     |
> >>      |   - VGIC put        |        158    |      40    |     75%     |
> >>      |   - VGIC get        |        120    |      19    |     84%     |
> >>      | Destination (total) |        192    |      91    |     53%     |
> >>      |   - VGIC put        |        132    |      27    |     80%     |
> >> 
> >> Yize Wang (2):
> >>    KVM: arm64: Add batch group constant and data structure to UAPI header
> >>    KVM: arm64: Add VGIC v3 batch register access implementation
> > Questions:
> > 
> > - Why only the MMIO registers?
> > 
> > - Why not the sysregs?
> > 
> > - Why only the GIC?
> > 
> > - Why not all of the state?
> > 
> > - Where is the corresponding userspace code?
> > 
> > More importantly, since this is about batching system calls:
> > 
> > - Why can't this be done with io_uring instead?
> > 
> > 	M.
> 
> 
> Hi, Marc! Thank you for the review.
> 
>   These patches focus on optimizing GICv3 register access during live
> migration. We found that there are a large number of locks (kvm->lock,
> vcpus, config_lock) in the GIC, these lock operations wil cost large
> time waste. The batches of sysreg for vcpu optimization will come in
> follow as a separate series. And let me address these questions one by
> one.

All these locks should be non-blocking by the time you save anything
related to the GIC, because:

- none of the vcpu can be running

- this must be a single threaded operation

So if you are seeing anything contended, this is either the sign of a
bad KVM bug, or an indication that you are violating the above
requirements.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

在 2026/10/3 22:53, Marc Zyngier 写道:
> On Sun, 20 Sep 2026 10:15:25 +0100,
> Yize Wang <wangyize7@huawei.com> wrote:
>> 在 2026/9/18 20:08, Marc Zyngier 写道:
>>> On Fri, 18 Sep 2026 09:18:13 +0100,
>>> Yize Wang<wangyize7@huawei.com> wrote:
>>>> This series adds batch register access support to KVM/arm64 to reduce
>>>> syscall overhead during VM live migration.
>>>>
>>>> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
>>>> state. On large VM configurations this means tens of thousands of syscalls,
>>>> where lock acquisition and context switch overhead dominates migration
>>>> downtime. Thus, we provide a batch register method to allow userspace
>>>> read/write multiple distributor and redistributor registers in a single call.
>>>> In this way, we can significantly reduce syscalls and migration downtime.
>>>>
>>>> Test the VM migration time under pressure conditions.
>>>> The VM specifications for migration are as follows:
>>>> - VM use 4-K page;
>>>> - the number of VCPU is 160;
>>>> - the total memory is 320Gigabit;
>>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>>
>>>> Performance results (3-run average, ms):
>>>>       | Metric              | Without patch | With patch | Improvement |
>>>>       |---------------------|---------------|------------|-------------|
>>>>       | Migration downtime  |        536    |     321    |     40%     |
>>>>       | Source (total)      |        344    |     230    |     33%     |
>>>>       |   - VGIC put        |        158    |      40    |     75%     |
>>>>       |   - VGIC get        |        120    |      19    |     84%     |
>>>>       | Destination (total) |        192    |      91    |     53%     |
>>>>       |   - VGIC put        |        132    |      27    |     80%     |
>>>>
>>>> Yize Wang (2):
>>>>     KVM: arm64: Add batch group constant and data structure to UAPI header
>>>>     KVM: arm64: Add VGIC v3 batch register access implementation
>>> Questions:
>>>
>>> - Why only the MMIO registers?
>>>
>>> - Why not the sysregs?
>>>
>>> - Why only the GIC?
>>>
>>> - Why not all of the state?
>>>
>>> - Where is the corresponding userspace code?
>>>
>>> More importantly, since this is about batching system calls:
>>>
>>> - Why can't this be done with io_uring instead?
>>>
>>> 	M.
>>
>> Hi, Marc! Thank you for the review.
>>
>>    These patches focus on optimizing GICv3 register access during live
>> migration. We found that there are a large number of locks (kvm->lock,
>> vcpus, config_lock) in the GIC, these lock operations wil cost large
>> time waste. The batches of sysreg for vcpu optimization will come in
>> follow as a separate series. And let me address these questions one by
>> one.
> All these locks should be non-blocking by the time you save anything
> related to the GIC, because:
>
> - none of the vcpu can be running
>
> - this must be a single threaded operation
>
> So if you are seeing anything contended, this is either the sign of a
> bad KVM bug, or an indication that you are violating the above
> requirements.
>
> 	M.


Hi, Marc! Thanks for your review.

You are right that when saving the GIC state during live migration 
process, all vCPUs are not running and it is a single threaded 
operation. Our patches process the packaged register states from the 
userspace under the same lock. These operations only aim to reduce the 
lock opreation downtime and do not involve lock contention.

In addition, these patches focus on reducing the syscalls of ioctl. Now, 
every register requires an ioctl, resulting in (num_cpu * 
registers_per_cpu) system calls. This will cost lots of time during the 
large-scale VM migration. Our patches batch registers by category in 
qemu and then send them to kernel in batches. This operations 
significantly reduce the number of ioctl syscalls and save a great 
amount of downtime.

The comparison is as follow:

BEFORE:

QEMU (Userspace)Kernel (Kernelspace)

+--------------------------++------------------------------+

||||

|kvm_arm_gicv3_get()||vgic_v3_attr_regs_access()|

||||

|for ncpu in num_cpu:|||

|+--------------+|||

|| GICR_CTLR|----|--- ioctl ------> |mutex_lock(&kvm->lock)|

|+--------------+ <--|--- val --------- |kvm_trylock_all_vcpus()|

|+--------------+||mutex_lock(&config_lock)|

|| GICR_STATUSR |----|--- ioctl ------> ||

|+--------------+ <--|--- val --------- |read/write single register|

|+--------------+|||

|| GICR_WAKER|----|--- ioctl ------> |mutex_unlock(&config_lock)|

|+--------------+ <--|--- val --------- |kvm_unlock_all_vcpus()|

|...||mutex_unlock(&kvm->lock)|

|+--------------+|||

|| ICC_SRE_EL1|----|--- ioctl ------> |(repeat lock -> read -> unlock

|+--------------+ <--|--- val --------- |for each register)|

|...|||

||||

+--------------------------++------------------------------+


AFTER:

QEMU (Userspace)Kernel (Kernelspace)

+----------------------------++--------------------------------+

||||

|kvm_arm_gicv3_get()||vgic_v3_batch_access()|

|||        |

|+- Collect Phase ----+|||

||| |||

|| batch_add(GICR_CTLR)| |||

|| batch_add(GICR_STATUSR) |||

|| batch_add(GICR_WAKER)||||

|| batch_add(GICR_IGROUPR0)| ||

|| ...||||

|| batch_add(ICC_SRE_EL1)  |||

|| ...||||

|+---------------------+ |||

||1 ioctl|copy_from_user(entries)|

|+- Commit Phase -----+|||

||| --- ioctl(BATCH)---> |mutex_lock(&kvm->lock)|

||batch_commit()| ||kvm_trylock_all_vcpus()|

|||||mutex_lock(&config_lock)|

||||||

|+---------------------+ ||+-- Process Each Entry --+|

||||||

|+- Unpack Phase -----+|||entry[0]: read GICR_CTLR|  |

||| <---batch result ---||entry[1]: read GICR_STATUSR|

||Write back to QEMU | | ||entry[2]: read GICR_WAKER| |

||state||||...||

||(post-process:| | ||entry[N]: read ICC_SRE | |

||unshuffle, split| | ||| |

||bytes, etc.)| |||* FLAG_LOCKED: skip||

||| | ||locking (already held)|  |

|+---------------------+ ||+-------------------------+ |

|||  |

+----------------------------++--------------------------------+

Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

在 2026/10/3 22:53, Marc Zyngier 写道:
> On Sun, 20 Sep 2026 10:15:25 +0100,
> Yize Wang <wangyize7@huawei.com> wrote:
>> 在 2026/9/18 20:08, Marc Zyngier 写道:
>>> On Fri, 18 Sep 2026 09:18:13 +0100,
>>> Yize Wang<wangyize7@huawei.com> wrote:
>>>> This series adds batch register access support to KVM/arm64 to reduce
>>>> syscall overhead during VM live migration.
>>>>
>>>> Currently, QEMU issues one ioctl per register when saving/restoring VGIC
>>>> state. On large VM configurations this means tens of thousands of syscalls,
>>>> where lock acquisition and context switch overhead dominates migration
>>>> downtime. Thus, we provide a batch register method to allow userspace
>>>> read/write multiple distributor and redistributor registers in a single call.
>>>> In this way, we can significantly reduce syscalls and migration downtime.
>>>>
>>>> Test the VM migration time under pressure conditions.
>>>> The VM specifications for migration are as follows:
>>>> - VM use 4-K page;
>>>> - the number of VCPU is 160;
>>>> - the total memory is 320Gigabit;
>>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>>
>>>> Performance results (3-run average, ms):
>>>>       | Metric              | Without patch | With patch | Improvement |
>>>>       |---------------------|---------------|------------|-------------|
>>>>       | Migration downtime  |        536    |     321    |     40%     |
>>>>       | Source (total)      |        344    |     230    |     33%     |
>>>>       |   - VGIC put        |        158    |      40    |     75%     |
>>>>       |   - VGIC get        |        120    |      19    |     84%     |
>>>>       | Destination (total) |        192    |      91    |     53%     |
>>>>       |   - VGIC put        |        132    |      27    |     80%     |
>>>>
>>>> Yize Wang (2):
>>>>     KVM: arm64: Add batch group constant and data structure to UAPI header
>>>>     KVM: arm64: Add VGIC v3 batch register access implementation
>>> Questions:
>>>
>>> - Why only the MMIO registers?
>>>
>>> - Why not the sysregs?
>>>
>>> - Why only the GIC?
>>>
>>> - Why not all of the state?
>>>
>>> - Where is the corresponding userspace code?
>>>
>>> More importantly, since this is about batching system calls:
>>>
>>> - Why can't this be done with io_uring instead?
>>>
>>> 	M.
>>
>> Hi, Marc! Thank you for the review.
>>
>>    These patches focus on optimizing GICv3 register access during live
>> migration. We found that there are a large number of locks (kvm->lock,
>> vcpus, config_lock) in the GIC, these lock operations wil cost large
>> time waste. The batches of sysreg for vcpu optimization will come in
>> follow as a separate series. And let me address these questions one by
>> one.
> All these locks should be non-blocking by the time you save anything
> related to the GIC, because:
>
> - none of the vcpu can be running
>
> - this must be a single threaded operation
>
> So if you are seeing anything contended, this is either the sign of a
> bad KVM bug, or an indication that you are violating the above
> requirements.
>
> 	M.


Hi, Marc! Thanks for your review.

You are right that when saving the GIC state during live migration process, all vCPUs are not running and it is a single threaded operation. Our patches process the packaged register states from the userspace under the same lock. These operations only aim to reduce the lock opreation downtime and do not involve lock contention.

In addition, these patches focus on reducing the syscalls of ioctl. Now, every register requires an ioctl, resulting in (num_cpu * registers_per_cpu) system calls. This will cost lots of time during the large-scale VM  migration. Our patches batch registers by category in qemu and then send them to kernel in batches. This operations significantly reduce the number of ioctl syscalls and save a great amount of downtime.

The comparison is as follow:
BEFORE:

        QEMU (Userspace)                  Kernel (Kernelspace)
   +------------------------+        +----------------------------+
   |                        |        |                            |
   |  kvm_arm_gicv3_get()   |        |  vgic_v3_attr_regs_access  |
   |                        |        |                            |
   |  for ncpu in num_cpu:  |        |                            |
   |    for each register:  |        |                            |
   |                        |        |                            |
   |  GICR_CTLR             | ioctl->|  mutex_lock(&kvm->lock)    |
   |                        |        |  kvm_trylock_all_vcpus     |
   |                        |        |  mutex_lock(&config_lock)  |
   |                        | <-val  |  read/write register       |
   |                        |        |  mutex_unlock              |
   |                        |        |  kvm_unlock_all_vcpus      |
   |                        |        |  mutex_unlock(&kvm->lock)  |
   |                        |        |                            |
   |  GICR_STATUSR          | ioctl->|  (repeat lock/read         |
   |                        | <-val  |   -> unlock)               |
   |  ...                   |        |                            |
   |                        |        |                            |
   |  GICR_WAKER            | ioctl->|  (repeat lock/read         |
   |                        | <-val  |   -> unlock)               |
   |  ...                   |        |                            |
   |                        |        |                            |
   |  ICC_SRE_EL1           | ioctl->|  (repeat lock/read         |
   |                        | <-val  |   -> unlock)               |
   |  ...                   |        |                            |
   |                        |        |                            |
   +------------------------+        +----------------------------+


AFTER:

        QEMU (Userspace)                  Kernel (Kernelspace)
   +------------------------+        +----------------------------+
   |                        |        |                            |
   |  kvm_arm_gicv3_get()   |        |  vgic_v3_batch_access      |
   |                        |        |                            |
   |  Collect Phase:        |        |                            |
   |    add(GICR_CTLR)      |        |                            |
   |    add(GICR_STATUSR)   |        |                            |
   |    add(GICR_WAKER)     |        |                            |
   |    add(GICR_IGROUPR0)  |        |                            |
   |    ...                 |        |                            |
   |    add(ICC_SRE_EL1)    |        |                            |
   |    ...                 |        |                            |
   |                        |        |                            |
   |  Commit Phase:         | BATCH->|  copy_from_user            |
   |    batch_commit()      |        |  mutex_lock(&kvm->lock)    |
   |                        |        |  kvm_trylock_all_vcpus     |
   |                        |        |  mutex_lock(&config_lock)  |
   |                        |        |                            |
   |                        |        |  Process Each Entry:       |
   |                        |        |    entry[0]: GICR_CTLR     |
   |                        |        |    entry[1]: GICR_STATUSR  |
   |                        |        |    entry[2]: GICR_WAKER    |
   |                        |        |    ...                     |
   |                        |        |    entry[N]: ICC_SRE       |
   |                        |        |    FLAG_LOCKED: skip       |
   |                        |        |                            |
   |  Unpack Phase:         |<-result|  mutex_unlock              |
   |    Write back          |        |  kvm_unlock_all_vcpus      |
   |    Post-process:       |        |  mutex_unlock(&kvm->lock)  |
   |      unshuffle,split   |        |  copy_to_user              |
   |                        |        |                            |
   +------------------------+        +----------------------------+

Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

Kindly Ping

在 2026/10/8 20:50, Yize Wang 写道:
>
> 在 2026/10/3 22:53, Marc Zyngier 写道:
>> On Sun, 20 Sep 2026 10:15:25 +0100,
>> Yize Wang <wangyize7@huawei.com> wrote:
>>> 在 2026/9/18 20:08, Marc Zyngier 写道:
>>>> On Fri, 18 Sep 2026 09:18:13 +0100,
>>>> Yize Wang<wangyize7@huawei.com> wrote:
>>>>> This series adds batch register access support to KVM/arm64 to reduce
>>>>> syscall overhead during VM live migration.
>>>>>
>>>>> Currently, QEMU issues one ioctl per register when 
>>>>> saving/restoring VGIC
>>>>> state. On large VM configurations this means tens of thousands of 
>>>>> syscalls,
>>>>> where lock acquisition and context switch overhead dominates 
>>>>> migration
>>>>> downtime. Thus, we provide a batch register method to allow userspace
>>>>> read/write multiple distributor and redistributor registers in a 
>>>>> single call.
>>>>> In this way, we can significantly reduce syscalls and migration 
>>>>> downtime.
>>>>>
>>>>> Test the VM migration time under pressure conditions.
>>>>> The VM specifications for migration are as follows:
>>>>> - VM use 4-K page;
>>>>> - the number of VCPU is 160;
>>>>> - the total memory is 320Gigabit;
>>>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>>>
>>>>> Performance results (3-run average, ms):
>>>>>       | Metric              | Without patch | With patch | 
>>>>> Improvement |
>>>>> |---------------------|---------------|------------|-------------|
>>>>>       | Migration downtime  |        536    |     321 |     40%     |
>>>>>       | Source (total)      |        344    |     230 |     33%     |
>>>>>       |   - VGIC put        |        158    |      40 |     75%     |
>>>>>       |   - VGIC get        |        120    |      19 |     84%     |
>>>>>       | Destination (total) |        192    |      91 |     53%     |
>>>>>       |   - VGIC put        |        132    |      27 |     80%     |
>>>>>
>>>>> Yize Wang (2):
>>>>>     KVM: arm64: Add batch group constant and data structure to 
>>>>> UAPI header
>>>>>     KVM: arm64: Add VGIC v3 batch register access implementation
>>>> Questions:
>>>>
>>>> - Why only the MMIO registers?
>>>>
>>>> - Why not the sysregs?
>>>>
>>>> - Why only the GIC?
>>>>
>>>> - Why not all of the state?
>>>>
>>>> - Where is the corresponding userspace code?
>>>>
>>>> More importantly, since this is about batching system calls:
>>>>
>>>> - Why can't this be done with io_uring instead?
>>>>
>>>>     M.
>>>
>>> Hi, Marc! Thank you for the review.
>>>
>>>    These patches focus on optimizing GICv3 register access during live
>>> migration. We found that there are a large number of locks (kvm->lock,
>>> vcpus, config_lock) in the GIC, these lock operations wil cost large
>>> time waste. The batches of sysreg for vcpu optimization will come in
>>> follow as a separate series. And let me address these questions one by
>>> one.
>> All these locks should be non-blocking by the time you save anything
>> related to the GIC, because:
>>
>> - none of the vcpu can be running
>>
>> - this must be a single threaded operation
>>
>> So if you are seeing anything contended, this is either the sign of a
>> bad KVM bug, or an indication that you are violating the above
>> requirements.
>>
>>     M.
>
>
> Hi, Marc! Thanks for your review.
>
> You are right that when saving the GIC state during live migration 
> process, all vCPUs are not running and it is a single threaded 
> operation. Our patches process the packaged register states from the 
> userspace under the same lock. These operations only aim to reduce the 
> lock opreation downtime and do not involve lock contention.
>
> In addition, these patches focus on reducing the syscalls of ioctl. 
> Now, every register requires an ioctl, resulting in (num_cpu * 
> registers_per_cpu) system calls. This will cost lots of time during 
> the large-scale VM  migration. Our patches batch registers by category 
> in qemu and then send them to kernel in batches. This operations 
> significantly reduce the number of ioctl syscalls and save a great 
> amount of downtime.
>
> The comparison is as follow:
> BEFORE:
>
>        QEMU (Userspace)                  Kernel (Kernelspace)
>   +------------------------+        +----------------------------+
>   |                        |        |                            |
>   |  kvm_arm_gicv3_get()   |        |  vgic_v3_attr_regs_access  |
>   |                        |        |                            |
>   |  for ncpu in num_cpu:  |        |                            |
>   |    for each register:  |        |                            |
>   |                        |        |                            |
>   |  GICR_CTLR             | ioctl->| mutex_lock(&kvm->lock)    |
>   |                        |        |  kvm_trylock_all_vcpus     |
>   |                        |        | mutex_lock(&config_lock)  |
>   |                        | <-val  |  read/write register       |
>   |                        |        |  mutex_unlock              |
>   |                        |        |  kvm_unlock_all_vcpus      |
>   |                        |        | mutex_unlock(&kvm->lock)  |
>   |                        |        |                            |
>   |  GICR_STATUSR          | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   |  GICR_WAKER            | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   |  ICC_SRE_EL1           | ioctl->|  (repeat lock/read         |
>   |                        | <-val  |   -> unlock)               |
>   |  ...                   |        |                            |
>   |                        |        |                            |
>   +------------------------+        +----------------------------+
>
>
> AFTER:
>
>        QEMU (Userspace)                  Kernel (Kernelspace)
>   +------------------------+        +----------------------------+
>   |                        |        |                            |
>   |  kvm_arm_gicv3_get()   |        |  vgic_v3_batch_access      |
>   |                        |        |                            |
>   |  Collect Phase:        |        |                            |
>   |    add(GICR_CTLR)      |        |                            |
>   |    add(GICR_STATUSR)   |        |                            |
>   |    add(GICR_WAKER)     |        |                            |
>   |    add(GICR_IGROUPR0)  |        |                            |
>   |    ...                 |        |                            |
>   |    add(ICC_SRE_EL1)    |        |                            |
>   |    ...                 |        |                            |
>   |                        |        |                            |
>   |  Commit Phase:         | BATCH->| copy_from_user            |
>   |    batch_commit()      |        | mutex_lock(&kvm->lock)    |
>   |                        |        |  kvm_trylock_all_vcpus     |
>   |                        |        | mutex_lock(&config_lock)  |
>   |                        |        |                            |
>   |                        |        |  Process Each Entry:       |
>   |                        |        |    entry[0]: GICR_CTLR     |
>   |                        |        |    entry[1]: GICR_STATUSR  |
>   |                        |        |    entry[2]: GICR_WAKER    |
>   |                        |        |    ...                     |
>   |                        |        |    entry[N]: ICC_SRE       |
>   |                        |        |    FLAG_LOCKED: skip       |
>   |                        |        |                            |
>   |  Unpack Phase:         |<-result| mutex_unlock              |
>   |    Write back          |        |  kvm_unlock_all_vcpus      |
>   |    Post-process:       |        | mutex_unlock(&kvm->lock)  |
>   |      unshuffle,split   |        |  copy_to_user              |
>   |                        |        |                            |
>   +------------------------+        +----------------------------+

Re: [RESEND][PATCH 0/2] Batch register access for live migration optimization

[Marc Zyngier]

On Thu, 08 Oct 2026 13:50:57 +0100,
Yize Wang <wangyize7@huawei.com> wrote:
> 
> 
> 在 2026/10/3 22:53, Marc Zyngier 写道:
> > On Sun, 20 Sep 2026 10:15:25 +0100,
> > Yize Wang <wangyize7@huawei.com> wrote:
> >> 在 2026/9/18 20:08, Marc Zyngier 写道:
> >>> On Fri, 18 Sep 2026 09:18:13 +0100,
> >>> Yize Wang<wangyize7@huawei.com> wrote:
> >>    These patches focus on optimizing GICv3 register access during live
> >> migration. We found that there are a large number of locks (kvm->lock,
> >> vcpus, config_lock) in the GIC, these lock operations wil cost large
> >> time waste. The batches of sysreg for vcpu optimization will come in
> >> follow as a separate series. And let me address these questions one by
> >> one.
> > All these locks should be non-blocking by the time you save anything
> > related to the GIC, because:
> > 
> > - none of the vcpu can be running
> > 
> > - this must be a single threaded operation
> > 
> > So if you are seeing anything contended, this is either the sign of a
> > bad KVM bug, or an indication that you are violating the above
> > requirements.
> > 
> > 	M.
> 
> 
> Hi, Marc! Thanks for your review.
> 
> You are right that when saving the GIC state during live migration
> process, all vCPUs are not running and it is a single threaded
> operation. Our patches process the packaged register states from the
> userspace under the same lock. These operations only aim to reduce
> the lock opreation downtime and do not involve lock contention.

OK, so your argument was actively misleading, since you were claiming
that locks were the majority of the overhead, and are now
advocating... something else.

> In addition, these patches focus on reducing the syscalls of
> ioctl. Now, every register requires an ioctl, resulting in (num_cpu
> * registers_per_cpu) system calls. This will cost lots of time
> during the large-scale VM migration. Our patches batch registers by
> category in qemu and then send them to kernel in batches. This
> operations significantly reduce the number of ioctl syscalls and
> save a great amount of downtime.

I've suggested you use io_uring, and you completely disregarded my
suggestion based on... locking contention. How do you expect me to
take you seriously?

I've now marked this series as rejected. Please come back once you and
your AI toy have agreed on what you want to do.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.