* [RFC] x86/kvm: Use native qspinlock by default when realtime hinted @ 2025-07-02 6:42 Liangyan 2025-07-02 8:19 ` Bibo Mao 0 siblings, 1 reply; 6+ messages in thread From: Liangyan @ 2025-07-02 6:42 UTC (permalink / raw) To: pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa Cc: linux-kernel, x86, kvm, Liangyan When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, currently guest will use virt_spin_lock. Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe and have better performance than virt_spin_lock. Signed-off-by: Liangyan <liangyan.peng@bytedance.com> --- arch/x86/kernel/kvm.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 921c1c783bc1..9080544a4007 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) */ void __init kvm_spinlock_init(void) { + /* + * Disable PV spinlocks and use native qspinlock when dedicated pCPUs + * are available. + */ + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n"); + goto out; + } + /* * In case host doesn't support KVM_FEATURE_PV_UNHALT there is still an * advantage of keeping virt_spin_lock_key enabled: virt_spin_lock() is @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) return; } - /* - * Disable PV spinlocks and use native qspinlock when dedicated pCPUs - * are available. - */ - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n"); - goto out; - } - if (num_possible_cpus() == 1) { pr_info("PV spinlocks disabled, single CPU\n"); goto out; -- 2.39.3 (Apple Git-145) ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted 2025-07-02 6:42 [RFC] x86/kvm: Use native qspinlock by default when realtime hinted Liangyan @ 2025-07-02 8:19 ` Bibo Mao 2025-07-02 12:23 ` [External] " Liangyan 0 siblings, 1 reply; 6+ messages in thread From: Bibo Mao @ 2025-07-02 8:19 UTC (permalink / raw) To: Liangyan, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa Cc: linux-kernel, x86, kvm On 2025/7/2 下午2:42, Liangyan wrote: > When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, > currently guest will use virt_spin_lock. > Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe > and have better performance than virt_spin_lock. Just be curious, do you have actual data where native qspinlock has better performance than virt_spin_lock()? By my understanding, qspinlock is not friendly with VM. When lock is released, it is acquired with one by one order in contending queue. If the first vCPU in contending queue is preempted, the other vCPUs can not get lock. On physical machine it is almost impossible that CPU contending lock is preempted. Regards Bibo Mao > > Signed-off-by: Liangyan <liangyan.peng@bytedance.com> > --- > arch/x86/kernel/kvm.c | 18 +++++++++--------- > 1 file changed, 9 insertions(+), 9 deletions(-) > > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c > index 921c1c783bc1..9080544a4007 100644 > --- a/arch/x86/kernel/kvm.c > +++ b/arch/x86/kernel/kvm.c > @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) > */ > void __init kvm_spinlock_init(void) > { > + /* > + * Disable PV spinlocks and use native qspinlock when dedicated pCPUs > + * are available. > + */ > + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { > + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n"); > + goto out; > + } > + > /* > * In case host doesn't support KVM_FEATURE_PV_UNHALT there is still an > * advantage of keeping virt_spin_lock_key enabled: virt_spin_lock() is > @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) > return; > } > > - /* > - * Disable PV spinlocks and use native qspinlock when dedicated pCPUs > - * are available. > - */ > - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { > - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME hints\n"); > - goto out; > - } > - > if (num_possible_cpus() == 1) { > pr_info("PV spinlocks disabled, single CPU\n"); > goto out; > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted 2025-07-02 8:19 ` Bibo Mao @ 2025-07-02 12:23 ` Liangyan 2025-07-02 16:26 ` Konrad Rzeszutek Wilk 2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao 0 siblings, 2 replies; 6+ messages in thread From: Liangyan @ 2025-07-02 12:23 UTC (permalink / raw) To: Bibo Mao, Liangyan, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa, wanpengli Cc: linux-kernel, x86, kvm We test that unixbench spawn has big improvement in Intel 8582c 120-CPU guest vm if switch to qspinlock. Command: ./Run -c 120 spawn Use virt_spin_lock: System Benchmarks Partial Index BASELINE RESULT INDEX Process Creation 126.0 71878.4 5704.6 ======== System Benchmarks Index Score (Partial Only) 5704.6 Use qspinlock: System Benchmarks Partial Index BASELINE RESULT INDEX Process Creation 126.0 173566.6 13775.1 ======== System Benchmarks Index Score (Partial Only 13775.1 Regards, Liangyan On 2025/7/2 16:19, Bibo Mao wrote: > > > On 2025/7/2 下午2:42, Liangyan wrote: >> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, >> currently guest will use virt_spin_lock. >> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe >> and have better performance than virt_spin_lock. > Just be curious, do you have actual data where native qspinlock has > better performance than virt_spin_lock()? > > By my understanding, qspinlock is not friendly with VM. When lock is > released, it is acquired with one by one order in contending queue. If > the first vCPU in contending queue is preempted, the other vCPUs can not > get lock. On physical machine it is almost impossible that CPU > contending lock is preempted. > > Regards > Bibo Mao >> >> Signed-off-by: Liangyan <liangyan.peng@bytedance.com> >> --- >> arch/x86/kernel/kvm.c | 18 +++++++++--------- >> 1 file changed, 9 insertions(+), 9 deletions(-) >> >> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c >> index 921c1c783bc1..9080544a4007 100644 >> --- a/arch/x86/kernel/kvm.c >> +++ b/arch/x86/kernel/kvm.c >> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) >> */ >> void __init kvm_spinlock_init(void) >> { >> + /* >> + * Disable PV spinlocks and use native qspinlock when dedicated >> pCPUs >> + * are available. >> + */ >> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >> hints\n"); >> + goto out; >> + } >> + >> /* >> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is >> still an >> * advantage of keeping virt_spin_lock_key enabled: >> virt_spin_lock() is >> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) >> return; >> } >> - /* >> - * Disable PV spinlocks and use native qspinlock when dedicated >> pCPUs >> - * are available. >> - */ >> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >> hints\n"); >> - goto out; >> - } >> - >> if (num_possible_cpus() == 1) { >> pr_info("PV spinlocks disabled, single CPU\n"); >> goto out; >> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted 2025-07-02 12:23 ` [External] " Liangyan @ 2025-07-02 16:26 ` Konrad Rzeszutek Wilk 2025-07-04 6:12 ` [External] Re: [PATCH RFC] " Liangyan 2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao 1 sibling, 1 reply; 6+ messages in thread From: Konrad Rzeszutek Wilk @ 2025-07-02 16:26 UTC (permalink / raw) To: Liangyan Cc: Bibo Mao, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa, wanpengli, linux-kernel, x86, kvm On Wed, Jul 02, 2025 at 08:23:58PM +0800, Liangyan wrote: > We test that unixbench spawn has big improvement in Intel 8582c 120-CPU > guest vm if switch to qspinlock. And ARM or AMD? > > Command: ./Run -c 120 spawn > > Use virt_spin_lock: > System Benchmarks Partial Index BASELINE RESULT INDEX > Process Creation 126.0 71878.4 5704.6 > ======== > System Benchmarks Index Score (Partial Only) 5704.6 > > > Use qspinlock: > System Benchmarks Partial Index BASELINE RESULT INDEX > Process Creation 126.0 173566.6 13775.1 > ======== > System Benchmarks Index Score (Partial Only 13775.1 > > > Regards, > Liangyan > > On 2025/7/2 16:19, Bibo Mao wrote: > > > > > > On 2025/7/2 下午2:42, Liangyan wrote: > > > When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, > > > currently guest will use virt_spin_lock. > > > Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe > > > and have better performance than virt_spin_lock. > > Just be curious, do you have actual data where native qspinlock has > > better performance than virt_spin_lock()? > > > > By my understanding, qspinlock is not friendly with VM. When lock is > > released, it is acquired with one by one order in contending queue. If > > the first vCPU in contending queue is preempted, the other vCPUs can not > > get lock. On physical machine it is almost impossible that CPU > > contending lock is preempted. > > > > Regards > > Bibo Mao > > > > > > Signed-off-by: Liangyan <liangyan.peng@bytedance.com> > > > --- > > > arch/x86/kernel/kvm.c | 18 +++++++++--------- > > > 1 file changed, 9 insertions(+), 9 deletions(-) > > > > > > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c > > > index 921c1c783bc1..9080544a4007 100644 > > > --- a/arch/x86/kernel/kvm.c > > > +++ b/arch/x86/kernel/kvm.c > > > @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) > > > */ > > > void __init kvm_spinlock_init(void) > > > { > > > + /* > > > + * Disable PV spinlocks and use native qspinlock when dedicated > > > pCPUs > > > + * are available. > > > + */ > > > + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { > > > + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME > > > hints\n"); > > > + goto out; > > > + } > > > + > > > /* > > > * In case host doesn't support KVM_FEATURE_PV_UNHALT there is > > > still an > > > * advantage of keeping virt_spin_lock_key enabled: > > > virt_spin_lock() is > > > @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) > > > return; > > > } > > > - /* > > > - * Disable PV spinlocks and use native qspinlock when dedicated > > > pCPUs > > > - * are available. > > > - */ > > > - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { > > > - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME > > > hints\n"); > > > - goto out; > > > - } > > > - > > > if (num_possible_cpus() == 1) { > > > pr_info("PV spinlocks disabled, single CPU\n"); > > > goto out; > > > > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [PATCH RFC] x86/kvm: Use native qspinlock by default when realtime hinted 2025-07-02 16:26 ` Konrad Rzeszutek Wilk @ 2025-07-04 6:12 ` Liangyan 0 siblings, 0 replies; 6+ messages in thread From: Liangyan @ 2025-07-04 6:12 UTC (permalink / raw) To: Konrad Rzeszutek Wilk Cc: Bibo Mao, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa, wanpengli, linux-kernel, x86, kvm Find one AMD guest(AMD EPYC 9Y24 128-vCPU) to test, it seems about 9% improvement. Command: ./Run -c 128 spawn With virt spin lock: System Benchmarks Partial Index BASELINE RESULT INDEX Process Creation 126.0 120449.8 9559.5 ======== System Benchmarks Index Score (Partial Only 9559.5 With qspinlock: System Benchmarks Partial Index BASELINE RESULT INDEX Process Creation 126.0 131566.8 10441.8 ======== System Benchmarks Index Score (Partial Only) 10441.8 Regards, Liangyan On 2025/7/3 00:26, Konrad Rzeszutek Wilk wrote: > On Wed, Jul 02, 2025 at 08:23:58PM +0800, Liangyan wrote: >> We test that unixbench spawn has big improvement in Intel 8582c 120-CPU >> guest vm if switch to qspinlock. > > And ARM or AMD? > >> >> Command: ./Run -c 120 spawn >> >> Use virt_spin_lock: >> System Benchmarks Partial Index BASELINE RESULT INDEX >> Process Creation 126.0 71878.4 5704.6 >> ======== >> System Benchmarks Index Score (Partial Only) 5704.6 >> >> >> Use qspinlock: >> System Benchmarks Partial Index BASELINE RESULT INDEX >> Process Creation 126.0 173566.6 13775.1 >> ======== >> System Benchmarks Index Score (Partial Only 13775.1 >> >> >> Regards, >> Liangyan >> >> On 2025/7/2 16:19, Bibo Mao wrote: >>> >>> >>> On 2025/7/2 下午2:42, Liangyan wrote: >>>> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, >>>> currently guest will use virt_spin_lock. >>>> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe >>>> and have better performance than virt_spin_lock. >>> Just be curious, do you have actual data where native qspinlock has >>> better performance than virt_spin_lock()? >>> >>> By my understanding, qspinlock is not friendly with VM. When lock is >>> released, it is acquired with one by one order in contending queue. If >>> the first vCPU in contending queue is preempted, the other vCPUs can not >>> get lock. On physical machine it is almost impossible that CPU >>> contending lock is preempted. >>> >>> Regards >>> Bibo Mao >>>> >>>> Signed-off-by: Liangyan <liangyan.peng@bytedance.com> >>>> --- >>>> arch/x86/kernel/kvm.c | 18 +++++++++--------- >>>> 1 file changed, 9 insertions(+), 9 deletions(-) >>>> >>>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c >>>> index 921c1c783bc1..9080544a4007 100644 >>>> --- a/arch/x86/kernel/kvm.c >>>> +++ b/arch/x86/kernel/kvm.c >>>> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) >>>> */ >>>> void __init kvm_spinlock_init(void) >>>> { >>>> + /* >>>> + * Disable PV spinlocks and use native qspinlock when dedicated >>>> pCPUs >>>> + * are available. >>>> + */ >>>> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >>>> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >>>> hints\n"); >>>> + goto out; >>>> + } >>>> + >>>> /* >>>> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is >>>> still an >>>> * advantage of keeping virt_spin_lock_key enabled: >>>> virt_spin_lock() is >>>> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) >>>> return; >>>> } >>>> - /* >>>> - * Disable PV spinlocks and use native qspinlock when dedicated >>>> pCPUs >>>> - * are available. >>>> - */ >>>> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >>>> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >>>> hints\n"); >>>> - goto out; >>>> - } >>>> - >>>> if (num_possible_cpus() == 1) { >>>> pr_info("PV spinlocks disabled, single CPU\n"); >>>> goto out; >>>> >>> >> >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [External] Re: [RFC] x86/kvm: Use native qspinlock by default when realtime hinted 2025-07-02 12:23 ` [External] " Liangyan 2025-07-02 16:26 ` Konrad Rzeszutek Wilk @ 2025-07-05 6:39 ` Bibo Mao 1 sibling, 0 replies; 6+ messages in thread From: Bibo Mao @ 2025-07-05 6:39 UTC (permalink / raw) To: Liangyan, pbonzini, vkuznets, tglx, mingo, bp, dave.hansen, hpa, wanpengli Cc: linux-kernel, x86, kvm There is big improvement with the test result. spawn test case is a little tricky, if forked child process is scheduled on the same CPU with the parent, the benefit is very huge. I doubt it is probably caused by scheduler rather than by spinlock itself. 1. What is cpu topology and numa information with physical machine and virtual machine? 2. Could you show reschedule IPI interrupt stat information when running spawn test case? 3. Could you run this case on CPU over-commit scenary, such as both two VMs with 120 vCPUs? Regards Bibo Mao On 2025/7/2 下午8:23, Liangyan wrote: > We test that unixbench spawn has big improvement in Intel 8582c 120-CPU > guest vm if switch to qspinlock. > > Command: ./Run -c 120 spawn > > Use virt_spin_lock: > System Benchmarks Partial Index BASELINE RESULT INDEX > Process Creation 126.0 71878.4 5704.6 > ======== > System Benchmarks Index Score (Partial Only) 5704.6 > > > Use qspinlock: > System Benchmarks Partial Index BASELINE RESULT INDEX > Process Creation 126.0 173566.6 13775.1 > ======== > System Benchmarks Index Score (Partial Only 13775.1 > > > Regards, > Liangyan > > On 2025/7/2 16:19, Bibo Mao wrote: >> >> >> On 2025/7/2 下午2:42, Liangyan wrote: >>> When KVM_HINTS_REALTIME is set and KVM_FEATURE_PV_UNHALT is clear, >>> currently guest will use virt_spin_lock. >>> Since KVM_HINTS_REALTIME is set, use native qspinlock should be safe >>> and have better performance than virt_spin_lock. >> Just be curious, do you have actual data where native qspinlock has >> better performance than virt_spin_lock()? >> >> By my understanding, qspinlock is not friendly with VM. When lock is >> released, it is acquired with one by one order in contending queue. If >> the first vCPU in contending queue is preempted, the other vCPUs can >> not get lock. On physical machine it is almost impossible that CPU >> contending lock is preempted. >> >> Regards >> Bibo Mao >>> >>> Signed-off-by: Liangyan <liangyan.peng@bytedance.com> >>> --- >>> arch/x86/kernel/kvm.c | 18 +++++++++--------- >>> 1 file changed, 9 insertions(+), 9 deletions(-) >>> >>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c >>> index 921c1c783bc1..9080544a4007 100644 >>> --- a/arch/x86/kernel/kvm.c >>> +++ b/arch/x86/kernel/kvm.c >>> @@ -1072,6 +1072,15 @@ static void kvm_wait(u8 *ptr, u8 val) >>> */ >>> void __init kvm_spinlock_init(void) >>> { >>> + /* >>> + * Disable PV spinlocks and use native qspinlock when dedicated >>> pCPUs >>> + * are available. >>> + */ >>> + if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >>> + pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >>> hints\n"); >>> + goto out; >>> + } >>> + >>> /* >>> * In case host doesn't support KVM_FEATURE_PV_UNHALT there is >>> still an >>> * advantage of keeping virt_spin_lock_key enabled: >>> virt_spin_lock() is >>> @@ -1082,15 +1091,6 @@ void __init kvm_spinlock_init(void) >>> return; >>> } >>> - /* >>> - * Disable PV spinlocks and use native qspinlock when dedicated >>> pCPUs >>> - * are available. >>> - */ >>> - if (kvm_para_has_hint(KVM_HINTS_REALTIME)) { >>> - pr_info("PV spinlocks disabled with KVM_HINTS_REALTIME >>> hints\n"); >>> - goto out; >>> - } >>> - >>> if (num_possible_cpus() == 1) { >>> pr_info("PV spinlocks disabled, single CPU\n"); >>> goto out; >>> >> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-07-05 6:41 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-07-02 6:42 [RFC] x86/kvm: Use native qspinlock by default when realtime hinted Liangyan 2025-07-02 8:19 ` Bibo Mao 2025-07-02 12:23 ` [External] " Liangyan 2025-07-02 16:26 ` Konrad Rzeszutek Wilk 2025-07-04 6:12 ` [External] Re: [PATCH RFC] " Liangyan 2025-07-05 6:39 ` [External] Re: [RFC] " Bibo Mao
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).