* A question about HTL VM-Exit handling time
@ 2014-10-16 8:15 Wu, Feng
2014-10-21 18:39 ` Paolo Bonzini
0 siblings, 1 reply; 8+ messages in thread
From: Wu, Feng @ 2014-10-16 8:15 UTC (permalink / raw)
To: kvm@vger.kernel.org; +Cc: Xiao Guangrong
Hi folks,
I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
Analyze events for all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max Time A
MSR_WRITE 3613908 57.53% 18.97% 5us 1362us 9.73
HLT 1399747 22.28% 74.90% 5us 432448us 99.24
CR_ACCESS 961203 15.30% 3.28% 4us 188us 6.33
EXTERNAL_INTERRUPT 213821 3.40% 2.25% 4us 4089us 19.54
EXCEPTION_NMI 25152 0.40% 0.12% 4us 71us 9.05
EPT_MISCONFIG 20104 0.32% 0.15% 8us 5628us 13.74
CPUID 19904 0.32% 0.07% 4us 220us 6.90
IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us 22.08
PAUSE_INSTRUCTION 10737 0.17% 0.05% 4us 53us 8.33
MSR_READ 48 0.00% 0.00% 4us 8us 5.62
Total Samples:6281721, Total events handled time:185457820.41us.
I also do some other experiments with different workload in the guest, I got the same results in terms of
HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
You help!
Thanks,
Feng
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: A question about HTL VM-Exit handling time
2014-10-16 8:15 A question about HTL VM-Exit handling time Wu, Feng
@ 2014-10-21 18:39 ` Paolo Bonzini
2014-10-23 4:56 ` Wu, Feng
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-21 18:39 UTC (permalink / raw)
To: Wu, Feng, kvm@vger.kernel.org; +Cc: Xiao Guangrong
On 10/16/2014 10:15 AM, Wu, Feng wrote:
> Hi folks,
>
> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
>
> Analyze events for all VCPUs:
>
> VM-EXIT Samples Samples% Time% Min Time Max Time A
>
> MSR_WRITE 3613908 57.53% 18.97% 5us 1362us 9.73
> HLT 1399747 22.28% 74.90% 5us 432448us 99.24
> CR_ACCESS 961203 15.30% 3.28% 4us 188us 6.33
> EXTERNAL_INTERRUPT 213821 3.40% 2.25% 4us 4089us 19.54
> EXCEPTION_NMI 25152 0.40% 0.12% 4us 71us 9.05
> EPT_MISCONFIG 20104 0.32% 0.15% 8us 5628us 13.74
> CPUID 19904 0.32% 0.07% 4us 220us 6.90
> IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us 22.08
> PAUSE_INSTRUCTION 10737 0.17% 0.05% 4us 53us 8.33
> MSR_READ 48 0.00% 0.00% 4us 8us 5.62
>
> Total Samples:6281721, Total events handled time:185457820.41us.
>
> I also do some other experiments with different workload in the guest, I got the same results in terms of
> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
> You help!
432 ms sounds like a lot, but in general it is expected that HLT vmexits
take a long time. After an HLT vmexit, the VCPU will not be reentered
until the next interrupt comes. On hardware, the HLT instruction can
also take many milliseconds.
If this is an SMP guest, it's possible that the maximum time is
registered on the APs before Linux boots. With a UP guest I would
expect a shorter maximum time, but still longer than other vmexits.
Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: A question about HTL VM-Exit handling time
2014-10-21 18:39 ` Paolo Bonzini
@ 2014-10-23 4:56 ` Wu, Feng
2014-10-23 12:23 ` Paolo Bonzini
2014-10-27 23:52 ` Wanpeng Li
2014-10-28 7:46 ` Christian Borntraeger
2 siblings, 1 reply; 8+ messages in thread
From: Wu, Feng @ 2014-10-23 4:56 UTC (permalink / raw)
To: Paolo Bonzini, kvm@vger.kernel.org; +Cc: Xiao Guangrong
> -----Original Message-----
> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Wednesday, October 22, 2014 2:40 AM
> To: Wu, Feng; kvm@vger.kernel.org
> Cc: Xiao Guangrong
> Subject: Re: A question about HTL VM-Exit handling time
>
>
>
> On 10/16/2014 10:15 AM, Wu, Feng wrote:
> > Hi folks,
> >
> > I run kernel build in the guest and use perf kvm to get some VM-Exit result as
> the following:
> >
> > Analyze events for all VCPUs:
> >
> > VM-EXIT Samples Samples% Time% Min Time
> Max Time A
> >
> > MSR_WRITE 3613908 57.53% 18.97%
> 5us 1362us 9.73
> > HLT 1399747 22.28% 74.90% 5us
> 432448us 99.24
> > CR_ACCESS 961203 15.30% 3.28%
> 4us 188us 6.33
> > EXTERNAL_INTERRUPT 213821 3.40% 2.25%
> 4us 4089us 19.54
> > EXCEPTION_NMI 25152 0.40% 0.12%
> 4us 71us 9.05
> > EPT_MISCONFIG 20104 0.32% 0.15%
> 8us 5628us 13.74
> > CPUID 19904 0.32% 0.07% 4us
> 220us 6.90
> > IO_INSTRUCTION 17097 0.27% 0.20% 13us
> 1008us 22.08
> > PAUSE_INSTRUCTION 10737 0.17% 0.05%
> 4us 53us 8.33
> > MSR_READ 48 0.00% 0.00%
> 4us 8us 5.62
> >
> > Total Samples:6281721, Total events handled time:185457820.41us.
> >
> > I also do some other experiments with different workload in the guest, I got
> the same results in terms of
> > HLT VM-Exit handling time. Does anyone know why the handling time for HLT
> VM-Exit is so high? Appreciate
> > You help!
>
> 432 ms sounds like a lot, but in general it is expected that HLT vmexits
> take a long time. After an HLT vmexit, the VCPU will not be reentered
> until the next interrupt comes. On hardware, the HLT instruction can
> also take many milliseconds.
Thanks for your answer, Paolo.
There is a general question for perf kvm. Do you know how the vmexit
handling time is measured? Such as what is the start point, and the
end point? If I got this information, I think I can better understand the
value of the handling time. Thanks a lot!
>
> If this is an SMP guest, it's possible that the maximum time is
> registered on the APs before Linux boots. With a UP guest I would
> expect a shorter maximum time, but still longer than other vmexits.
Yes, you are right, I get the data for an UP guest and the HLT vmexit
handling time is shorter. Here is the result:
Analyze events for all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max Time
EXTERNAL_INTERRUPT 90015 29.68% 41.92% 4us 660us 24.9
MSR_WRITE 89512 29.52% 14.25% 4us 162us 8.5
IO_INSTRUCTION 47823 15.77% 16.67% 11us 1907us 18.6
EPT_MISCONFIG 38504 12.70% 7.42% 7us 4848us 10.3
CPUID 32658 10.77% 3.61% 3us 304us 5.9
EXCEPTION_NMI 2232 0.74% 0.34% 4us 164us 8.2
PAUSE_INSTRUCTION 1594 0.53% 0.21% 3us 44us 6.9
HLT 547 0.18% 15.50% 8us 177098us 1516.5
CR_ACCESS 278 0.09% 0.05% 5us 26us 8.7
EPT_VIOLATION 71 0.02% 0.03% 8us 27us 20.3
MSR_READ 12 0.00% 0.00% 3us 4us 4.1
Total Samples:303246, Total events handled time:5350983.30us.
Thanks,
Feng
>
> Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: A question about HTL VM-Exit handling time
2014-10-23 4:56 ` Wu, Feng
@ 2014-10-23 12:23 ` Paolo Bonzini
2014-10-24 1:14 ` Wu, Feng
0 siblings, 1 reply; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-23 12:23 UTC (permalink / raw)
To: Wu, Feng, kvm@vger.kernel.org; +Cc: Xiao Guangrong
> Thanks for your answer, Paolo.
> There is a general question for perf kvm. Do you know how the vmexit
> handling time is measured? Such as what is the start point, and the
> end point? If I got this information, I think I can better understand the
> value of the handling time. Thanks a lot!
It is measured from kvm:kvm_exit to kvm:kvm_entry. See
exit_event_begin, exit_event_get_key, exit_event_end in
tools/perf/builtin-kvm.c.
Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: A question about HTL VM-Exit handling time
2014-10-23 12:23 ` Paolo Bonzini
@ 2014-10-24 1:14 ` Wu, Feng
0 siblings, 0 replies; 8+ messages in thread
From: Wu, Feng @ 2014-10-24 1:14 UTC (permalink / raw)
To: Paolo Bonzini, kvm@vger.kernel.org; +Cc: Xiao Guangrong
> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Thursday, October 23, 2014 8:23 PM
> To: Wu, Feng; kvm@vger.kernel.org
> Cc: Xiao Guangrong
> Subject: Re: A question about HTL VM-Exit handling time
>
>
> > Thanks for your answer, Paolo.
> > There is a general question for perf kvm. Do you know how the vmexit
> > handling time is measured? Such as what is the start point, and the
> > end point? If I got this information, I think I can better understand the
> > value of the handling time. Thanks a lot!
>
> It is measured from kvm:kvm_exit to kvm:kvm_entry. See
> exit_event_begin, exit_event_get_key, exit_event_end in
> tools/perf/builtin-kvm.c.
Got it, Thanks for the information!
Thanks,
Feng
>
> Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: A question about HTL VM-Exit handling time
2014-10-21 18:39 ` Paolo Bonzini
2014-10-23 4:56 ` Wu, Feng
@ 2014-10-27 23:52 ` Wanpeng Li
2014-10-28 10:16 ` Paolo Bonzini
2014-10-28 7:46 ` Christian Borntraeger
2 siblings, 1 reply; 8+ messages in thread
From: Wanpeng Li @ 2014-10-27 23:52 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kvm@vger.kernel.org, Xiao Guangrong
Hi Paolo,
On Tue, Oct 21, 2014 at 08:39:49PM +0200, Paolo Bonzini wrote:
>
>
>On 10/16/2014 10:15 AM, Wu, Feng wrote:
>> Hi folks,
>>
>> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
>>
>> Analyze events for all VCPUs:
>>
>> VM-EXIT Samples Samples% Time% Min Time Max Time A
>>
>> MSR_WRITE 3613908 57.53% 18.97% 5us 1362us 9.73
>> HLT 1399747 22.28% 74.90% 5us 432448us 99.24
>> CR_ACCESS 961203 15.30% 3.28% 4us 188us 6.33
>> EXTERNAL_INTERRUPT 213821 3.40% 2.25% 4us 4089us 19.54
>> EXCEPTION_NMI 25152 0.40% 0.12% 4us 71us 9.05
>> EPT_MISCONFIG 20104 0.32% 0.15% 8us 5628us 13.74
>> CPUID 19904 0.32% 0.07% 4us 220us 6.90
>> IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us 22.08
>> PAUSE_INSTRUCTION 10737 0.17% 0.05% 4us 53us 8.33
>> MSR_READ 48 0.00% 0.00% 4us 8us 5.62
>>
>> Total Samples:6281721, Total events handled time:185457820.41us.
>>
>> I also do some other experiments with different workload in the guest, I got the same results in terms of
>> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
>> You help!
>
>432 ms sounds like a lot, but in general it is expected that HLT vmexits
>take a long time. After an HLT vmexit, the VCPU will not be reentered
>until the next interrupt comes. On hardware, the HLT instruction can
>also take many milliseconds.
>
As you know, kernel build which Feng used is a CPU bound workload,
so why 22.28% samples is halt?
Regards,
Wanpeng Li
>If this is an SMP guest, it's possible that the maximum time is
>registered on the APs before Linux boots. With a UP guest I would
>expect a shorter maximum time, but still longer than other vmexits.
>
>Paolo
>--
>To unsubscribe from this list: send the line "unsubscribe kvm" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: A question about HTL VM-Exit handling time
2014-10-21 18:39 ` Paolo Bonzini
2014-10-23 4:56 ` Wu, Feng
2014-10-27 23:52 ` Wanpeng Li
@ 2014-10-28 7:46 ` Christian Borntraeger
2 siblings, 0 replies; 8+ messages in thread
From: Christian Borntraeger @ 2014-10-28 7:46 UTC (permalink / raw)
To: Paolo Bonzini, Wu, Feng, kvm@vger.kernel.org
Cc: Xiao Guangrong, Alexander Yarygin
Am 21.10.2014 20:39, schrieb Paolo Bonzini:
>
>
> On 10/16/2014 10:15 AM, Wu, Feng wrote:
>> Hi folks,
>>
>> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
>>
>> Analyze events for all VCPUs:
>>
>> VM-EXIT Samples Samples% Time% Min Time Max Time A
>>
>> MSR_WRITE 3613908 57.53% 18.97% 5us 1362us 9.73
>> HLT 1399747 22.28% 74.90% 5us 432448us 99.24
>> CR_ACCESS 961203 15.30% 3.28% 4us 188us 6.33
>> EXTERNAL_INTERRUPT 213821 3.40% 2.25% 4us 4089us 19.54
>> EXCEPTION_NMI 25152 0.40% 0.12% 4us 71us 9.05
>> EPT_MISCONFIG 20104 0.32% 0.15% 8us 5628us 13.74
>> CPUID 19904 0.32% 0.07% 4us 220us 6.90
>> IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us 22.08
>> PAUSE_INSTRUCTION 10737 0.17% 0.05% 4us 53us 8.33
>> MSR_READ 48 0.00% 0.00% 4us 8us 5.62
>>
>> Total Samples:6281721, Total events handled time:185457820.41us.
>>
>> I also do some other experiments with different workload in the guest, I got the same results in terms of
>> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
>> You help!
>
> 432 ms sounds like a lot, but in general it is expected that HLT vmexits
> take a long time. After an HLT vmexit, the VCPU will not be reentered
> until the next interrupt comes. On hardware, the HLT instruction can
> also take many milliseconds.
>
> If this is an SMP guest, it's possible that the maximum time is
> registered on the APs before Linux boots. With a UP guest I would
> expect a shorter maximum time, but still longer than other vmexits.
We have the same on s390 with wait state. The thing is, with an idle system and NOHZ the time spend in HLT/wait could be really long. So We might want to provide an option to filter this out.
(A similar filter already exitis for the --duration option)
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: A question about HTL VM-Exit handling time
2014-10-27 23:52 ` Wanpeng Li
@ 2014-10-28 10:16 ` Paolo Bonzini
0 siblings, 0 replies; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-28 10:16 UTC (permalink / raw)
To: Wanpeng Li, Xiao Guangrong, KVM list
On 10/28/2014 12:52 AM, Wanpeng Li wrote:
> As you know, kernel build which Feng used is a CPU bound workload,
> so why 22.28% samples is halt?
Because in a CPU bound workload you don't have many exits in the first
place. But there could also be some issues with the way he used perf.
213821 interrupts seem to be really few.
Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-10-28 10:16 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-16 8:15 A question about HTL VM-Exit handling time Wu, Feng
2014-10-21 18:39 ` Paolo Bonzini
2014-10-23 4:56 ` Wu, Feng
2014-10-23 12:23 ` Paolo Bonzini
2014-10-24 1:14 ` Wu, Feng
2014-10-27 23:52 ` Wanpeng Li
2014-10-28 10:16 ` Paolo Bonzini
2014-10-28 7:46 ` Christian Borntraeger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).