* [RFC] x86/kvm: Use native qspinlock by default when realtime hinted
@ 2025-07-02 6:42 Liangyan
2025-07-02 8:19 ` Bibo Mao
0 siblings, 1 reply; 6+ messages in thread
From: Liangyan @ 2025-07-02 6:42 UTC (permalink / raw)
To: pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa
Cc: linux-kernel, x86, kvm, Liangyan
When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
currently guest will use virt_spin_lock.
Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
and have better performance than virt_spin_lock.
Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
---
arch/x86/kernel/kvm.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 921c1c783bc1..9080544a4007 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
*/
void __init kvm_spinlock_init(void)
{
+ /*
+ * Disable PV spinlocks and use native qspinlock when dedicated pCPUs
+ * are available.
+ */
+ if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
+ pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n");
+ goto out;
+ }
+
/*
* In case host doesn't support KVM_FEATURE_PV_UNHALT there is still an
* advantage of keeping virt_spin_lock_key enabled: virt_spin_lock() is
@@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
return;
}
- /*
- * Disable PV spinlocks and use native qspinlock when dedicated pCPUs
- * are available.
- */
- if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
- pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n");
- goto out;
- }
-
if (num_possible_cpus() == 1) {
pr_info("PV spinlocks disabled, single CPU\n");
goto out;
--
2.39.3 (Apple Git-145)
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted
2025-07-02 6:42 [RFC] x86/kvm: Use native qspinlock by default when realtime hinted Liangyan
@ 2025-07-02 8:19 ` Bibo Mao
2025-07-02 12:23 ` [External] " Liangyan
0 siblings, 1 reply; 6+ messages in thread
From: Bibo Mao @ 2025-07-02 8:19 UTC (permalink / raw)
To: Liangyan, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa
Cc: linux-kernel, x86, kvm
On 2025/7/2 下午2:42, Liangyan wrote:
> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
> currently guest will use virt_spin_lock.
> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
> and have better performance than virt_spin_lock.
Just be curious, do you have actual data where native qspinlock has
better performance than virt_spin_lock()?
By my understanding, qspinlock is not friendly with VM. When lock is
released, it is acquired with one by one order in contending queue. If
the first vCPU in contending queue is preempted, the other vCPUs can not
get lock. On physical machine it is almost impossible that CPU
contending lock is preempted.
Regards
Bibo Mao
>
> Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
> ---
> arch/x86/kernel/kvm.c | 18 +++++++++---------
> 1 file changed, 9 insertions(+), 9 deletions(-)
>
> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
> index 921c1c783bc1..9080544a4007 100644
> --- a/arch/x86/kernel/kvm.c
> +++ b/arch/x86/kernel/kvm.c
> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
> */
> void __init kvm_spinlock_init(void)
> {
> + /*
> + * Disable PV spinlocks and use native qspinlock when dedicated pCPUs
> + * are available.
> + */
> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n");
> + goto out;
> + }
> +
> /*
> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is still an
> * advantage of keeping virt_spin_lock_key enabled: virt_spin_lock() is
> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
> return;
> }
>
> - /*
> - * Disable PV spinlocks and use native qspinlock when dedicated pCPUs
> - * are available.
> - */
> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n");
> - goto out;
> - }
> -
> if (num_possible_cpus() == 1) {
> pr_info("PV spinlocks disabled, single CPU\n");
> goto out;
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted
2025-07-02 8:19 ` Bibo Mao
@ 2025-07-02 12:23 ` Liangyan
2025-07-02 16:26 ` Konrad Rzeszutek Wilk
2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao
0 siblings, 2 replies; 6+ messages in thread
From: Liangyan @ 2025-07-02 12:23 UTC (permalink / raw)
To: Bibo Mao, Liangyan, pbonzini, vkuznets, tglx, mingo, bp,
dave.hansen, hpa, wanpengli
Cc: linux-kernel, x86, kvm
We test that unixbench spawn has big improvement in Intel 8582c 120-CPU
guest vm if switch to qspinlock.
Command: ./Run -c 120 spawn
Use virt_spin_lock:
System Benchmarks Partial Index BASELINE RESULT INDEX
Process Creation 126.0 71878.4 5704.6
========
System Benchmarks Index Score (Partial Only) 5704.6
Use qspinlock:
System Benchmarks Partial Index BASELINE RESULT INDEX
Process Creation 126.0 173566.6 13775.1
========
System Benchmarks Index Score (Partial Only 13775.1
Regards,
Liangyan
On 2025/7/2 16:19, Bibo Mao wrote:
>
>
> On 2025/7/2 下午2:42, Liangyan wrote:
>> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
>> currently guest will use virt_spin_lock.
>> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
>> and have better performance than virt_spin_lock.
> Just be curious, do you have actual data where native qspinlock has
> better performance than virt_spin_lock()?
>
> By my understanding, qspinlock is not friendly with VM. When lock is
> released, it is acquired with one by one order in contending queue. If
> the first vCPU in contending queue is preempted, the other vCPUs can not
> get lock. On physical machine it is almost impossible that CPU
> contending lock is preempted.
>
> Regards
> Bibo Mao
>>
>> Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
>> ---
>> arch/x86/kernel/kvm.c | 18 +++++++++---------
>> 1 file changed, 9 insertions(+), 9 deletions(-)
>>
>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
>> index 921c1c783bc1..9080544a4007 100644
>> --- a/arch/x86/kernel/kvm.c
>> +++ b/arch/x86/kernel/kvm.c
>> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
>> */
>> void __init kvm_spinlock_init(void)
>> {
>> + /*
>> + * Disable PV spinlocks and use native qspinlock when dedicated
>> pCPUs
>> + * are available.
>> + */
>> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>> hints\n");
>> + goto out;
>> + }
>> +
>> /*
>> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is
>> still an
>> * advantage of keeping virt_spin_lock_key enabled:
>> virt_spin_lock() is
>> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
>> return;
>> }
>> - /*
>> - * Disable PV spinlocks and use native qspinlock when dedicated
>> pCPUs
>> - * are available.
>> - */
>> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>> hints\n");
>> - goto out;
>> - }
>> -
>> if (num_possible_cpus() == 1) {
>> pr_info("PV spinlocks disabled, single CPU\n");
>> goto out;
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted
2025-07-02 12:23 ` [External] " Liangyan
@ 2025-07-02 16:26 ` Konrad Rzeszutek Wilk
2025-07-04 6:12 ` [External] Re: [PATCH RFC] " Liangyan
2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao
1 sibling, 1 reply; 6+ messages in thread
From: Konrad Rzeszutek Wilk @ 2025-07-02 16:26 UTC (permalink / raw)
To: Liangyan
Cc: Bibo Mao, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa,
wanpengli, linux-kernel, x86, kvm
On Wed, Jul 02, 2025 at 08:23:58PM +0800, Liangyan wrote:
> We test that unixbench spawn has big improvement in Intel 8582c 120-CPU
> guest vm if switch to qspinlock.
And ARM or AMD?
>
> Command: ./Run -c 120 spawn
>
> Use virt_spin_lock:
> System Benchmarks Partial Index BASELINE RESULT INDEX
> Process Creation 126.0 71878.4 5704.6
> ========
> System Benchmarks Index Score (Partial Only) 5704.6
>
>
> Use qspinlock:
> System Benchmarks Partial Index BASELINE RESULT INDEX
> Process Creation 126.0 173566.6 13775.1
> ========
> System Benchmarks Index Score (Partial Only 13775.1
>
>
> Regards,
> Liangyan
>
> On 2025/7/2 16:19, Bibo Mao wrote:
> >
> >
> > On 2025/7/2 下午2:42, Liangyan wrote:
> > > When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
> > > currently guest will use virt_spin_lock.
> > > Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
> > > and have better performance than virt_spin_lock.
> > Just be curious, do you have actual data where native qspinlock has
> > better performance than virt_spin_lock()?
> >
> > By my understanding, qspinlock is not friendly with VM. When lock is
> > released, it is acquired with one by one order in contending queue. If
> > the first vCPU in contending queue is preempted, the other vCPUs can not
> > get lock. On physical machine it is almost impossible that CPU
> > contending lock is preempted.
> >
> > Regards
> > Bibo Mao
> > >
> > > Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
> > > ---
> > > arch/x86/kernel/kvm.c | 18 +++++++++---------
> > > 1 file changed, 9 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
> > > index 921c1c783bc1..9080544a4007 100644
> > > --- a/arch/x86/kernel/kvm.c
> > > +++ b/arch/x86/kernel/kvm.c
> > > @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
> > > */
> > > void __init kvm_spinlock_init(void)
> > > {
> > > + /*
> > > + * Disable PV spinlocks and use native qspinlock when dedicated
> > > pCPUs
> > > + * are available.
> > > + */
> > > + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
> > > + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
> > > hints\n");
> > > + goto out;
> > > + }
> > > +
> > > /*
> > > * In case host doesn't support KVM_FEATURE_PV_UNHALT there is
> > > still an
> > > * advantage of keeping virt_spin_lock_key enabled:
> > > virt_spin_lock() is
> > > @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
> > > return;
> > > }
> > > - /*
> > > - * Disable PV spinlocks and use native qspinlock when dedicated
> > > pCPUs
> > > - * are available.
> > > - */
> > > - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
> > > - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
> > > hints\n");
> > > - goto out;
> > > - }
> > > -
> > > if (num_possible_cpus() == 1) {
> > > pr_info("PV spinlocks disabled, single CPU\n");
> > > goto out;
> > >
> >
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [PATCH RFC] x86/kvm: Use native qspinlock by default when realtime hinted
2025-07-02 16:26 ` Konrad Rzeszutek Wilk
@ 2025-07-04 6:12 ` Liangyan
0 siblings, 0 replies; 6+ messages in thread
From: Liangyan @ 2025-07-04 6:12 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Bibo Mao, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa,
wanpengli, linux-kernel, x86, kvm
Find one AMD guest(AMD EPYC 9Y24 128-vCPU) to test, it seems about 9%
improvement.
Command: ./Run -c 128 spawn
With virt spin lock:
System Benchmarks Partial Index BASELINE RESULT INDEX
Process Creation 126.0 120449.8 9559.5
========
System Benchmarks Index Score (Partial Only 9559.5
With qspinlock:
System Benchmarks Partial Index BASELINE RESULT INDEX
Process Creation 126.0 131566.8 10441.8
========
System Benchmarks Index Score (Partial Only) 10441.8
Regards,
Liangyan
On 2025/7/3 00:26, Konrad Rzeszutek Wilk wrote:
> On Wed, Jul 02, 2025 at 08:23:58PM +0800, Liangyan wrote:
>> We test that unixbench spawn has big improvement in Intel 8582c 120-CPU
>> guest vm if switch to qspinlock.
>
> And ARM or AMD?
>
>>
>> Command: ./Run -c 120 spawn
>>
>> Use virt_spin_lock:
>> System Benchmarks Partial Index BASELINE RESULT INDEX
>> Process Creation 126.0 71878.4 5704.6
>> ========
>> System Benchmarks Index Score (Partial Only) 5704.6
>>
>>
>> Use qspinlock:
>> System Benchmarks Partial Index BASELINE RESULT INDEX
>> Process Creation 126.0 173566.6 13775.1
>> ========
>> System Benchmarks Index Score (Partial Only 13775.1
>>
>>
>> Regards,
>> Liangyan
>>
>> On 2025/7/2 16:19, Bibo Mao wrote:
>>>
>>>
>>> On 2025/7/2 下午2:42, Liangyan wrote:
>>>> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
>>>> currently guest will use virt_spin_lock.
>>>> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
>>>> and have better performance than virt_spin_lock.
>>> Just be curious, do you have actual data where native qspinlock has
>>> better performance than virt_spin_lock()?
>>>
>>> By my understanding, qspinlock is not friendly with VM. When lock is
>>> released, it is acquired with one by one order in contending queue. If
>>> the first vCPU in contending queue is preempted, the other vCPUs can not
>>> get lock. On physical machine it is almost impossible that CPU
>>> contending lock is preempted.
>>>
>>> Regards
>>> Bibo Mao
>>>>
>>>> Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
>>>> ---
>>>> arch/x86/kernel/kvm.c | 18 +++++++++---------
>>>> 1 file changed, 9 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
>>>> index 921c1c783bc1..9080544a4007 100644
>>>> --- a/arch/x86/kernel/kvm.c
>>>> +++ b/arch/x86/kernel/kvm.c
>>>> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
>>>> */
>>>> void __init kvm_spinlock_init(void)
>>>> {
>>>> + /*
>>>> + * Disable PV spinlocks and use native qspinlock when dedicated
>>>> pCPUs
>>>> + * are available.
>>>> + */
>>>> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>>>> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>>>> hints\n");
>>>> + goto out;
>>>> + }
>>>> +
>>>> /*
>>>> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is
>>>> still an
>>>> * advantage of keeping virt_spin_lock_key enabled:
>>>> virt_spin_lock() is
>>>> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
>>>> return;
>>>> }
>>>> - /*
>>>> - * Disable PV spinlocks and use native qspinlock when dedicated
>>>> pCPUs
>>>> - * are available.
>>>> - */
>>>> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>>>> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>>>> hints\n");
>>>> - goto out;
>>>> - }
>>>> -
>>>> if (num_possible_cpus() == 1) {
>>>> pr_info("PV spinlocks disabled, single CPU\n");
>>>> goto out;
>>>>
>>>
>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted
2025-07-02 12:23 ` [External] " Liangyan
2025-07-02 16:26 ` Konrad Rzeszutek Wilk
@ 2025-07-05 6:39 ` Bibo Mao
1 sibling, 0 replies; 6+ messages in thread
From: Bibo Mao @ 2025-07-05 6:39 UTC (permalink / raw)
To: Liangyan, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa,
wanpengli
Cc: linux-kernel, x86, kvm
There is big improvement with the test result. spawn test case is a
little tricky, if forked child process is scheduled on the same CPU with
the parent, the benefit is very huge. I doubt it is probably caused by
scheduler rather than by spinlock itself.
1. What is cpu topology and numa information with physical machine and
virtual machine?
2. Could you show reschedule IPI interrupt stat information when running
spawn test case?
3. Could you run this case on CPU over-commit scenary, such as both two
VMs with 120 vCPUs?
Regards
Bibo Mao
On 2025/7/2 下午8:23, Liangyan wrote:
> We test that unixbench spawn has big improvement in Intel 8582c 120-CPU
> guest vm if switch to qspinlock.
>
> Command: ./Run -c 120 spawn
>
> Use virt_spin_lock:
> System Benchmarks Partial Index BASELINE RESULT INDEX
> Process Creation 126.0 71878.4 5704.6
> ========
> System Benchmarks Index Score (Partial Only) 5704.6
>
>
> Use qspinlock:
> System Benchmarks Partial Index BASELINE RESULT INDEX
> Process Creation 126.0 173566.6 13775.1
> ========
> System Benchmarks Index Score (Partial Only 13775.1
>
>
> Regards,
> Liangyan
>
> On 2025/7/2 16:19, Bibo Mao wrote:
>>
>>
>> On 2025/7/2 下午2:42, Liangyan wrote:
>>> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear,
>>> currently guest will use virt_spin_lock.
>>> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe
>>> and have better performance than virt_spin_lock.
>> Just be curious, do you have actual data where native qspinlock has
>> better performance than virt_spin_lock()?
>>
>> By my understanding, qspinlock is not friendly with VM. When lock is
>> released, it is acquired with one by one order in contending queue. If
>> the first vCPU in contending queue is preempted, the other vCPUs can
>> not get lock. On physical machine it is almost impossible that CPU
>> contending lock is preempted.
>>
>> Regards
>> Bibo Mao
>>>
>>> Signed-off-by: Liangyan <liangyan.peng@bytedance.com>
>>> ---
>>> arch/x86/kernel/kvm.c | 18 +++++++++---------
>>> 1 file changed, 9 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
>>> index 921c1c783bc1..9080544a4007 100644
>>> --- a/arch/x86/kernel/kvm.c
>>> +++ b/arch/x86/kernel/kvm.c
>>> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val)
>>> */
>>> void __init kvm_spinlock_init(void)
>>> {
>>> + /*
>>> + * Disable PV spinlocks and use native qspinlock when dedicated
>>> pCPUs
>>> + * are available.
>>> + */
>>> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>>> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>>> hints\n");
>>> + goto out;
>>> + }
>>> +
>>> /*
>>> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is
>>> still an
>>> * advantage of keeping virt_spin_lock_key enabled:
>>> virt_spin_lock() is
>>> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void)
>>> return;
>>> }
>>> - /*
>>> - * Disable PV spinlocks and use native qspinlock when dedicated
>>> pCPUs
>>> - * are available.
>>> - */
>>> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) {
>>> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME
>>> hints\n");
>>> - goto out;
>>> - }
>>> -
>>> if (num_possible_cpus() == 1) {
>>> pr_info("PV spinlocks disabled, single CPU\n");
>>> goto out;
>>>
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-07-05 6:41 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-02 6:42 [RFC] x86/kvm: Use native qspinlock by default when realtime hinted Liangyan
2025-07-02 8:19 ` Bibo Mao
2025-07-02 12:23 ` [External] " Liangyan
2025-07-02 16:26 ` Konrad Rzeszutek Wilk
2025-07-04 6:12 ` [External] Re: [PATCH RFC] " Liangyan
2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).