A question about HTL VM-Exit handling time

kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* A question about HTL VM-Exit handling time
@ 2014-10-16  8:15 Wu, Feng
  2014-10-21 18:39 ` Paolo Bonzini
  0 siblings, 1 reply; 8+ messages in thread
From: Wu, Feng @ 2014-10-16  8:15 UTC (permalink / raw)
  To: kvm@vger.kernel.org; +Cc: Xiao Guangrong

Hi folks,

I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:

Analyze events for all VCPUs:

             VM-EXIT    Samples  Samples%     Time%   Min Time   Max Time         A

           MSR_WRITE    3613908    57.53%    18.97%        5us     1362us      9.73
                 HLT    1399747    22.28%    74.90%        5us   432448us     99.24
           CR_ACCESS     961203    15.30%     3.28%        4us      188us      6.33
  EXTERNAL_INTERRUPT     213821     3.40%     2.25%        4us     4089us     19.54
       EXCEPTION_NMI      25152     0.40%     0.12%        4us       71us      9.05
       EPT_MISCONFIG      20104     0.32%     0.15%        8us     5628us     13.74
               CPUID      19904     0.32%     0.07%        4us      220us      6.90
      IO_INSTRUCTION      17097     0.27%     0.20%       13us     1008us     22.08
   PAUSE_INSTRUCTION      10737     0.17%     0.05%        4us       53us      8.33
            MSR_READ         48     0.00%     0.00%        4us        8us      5.62

Total Samples:6281721, Total events handled time:185457820.41us.

I also do some other experiments with different workload in the guest, I got the same results in terms of
HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
You help!

Thanks,
Feng

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: A question about HTL VM-Exit handling time
  2014-10-16  8:15 A question about HTL VM-Exit handling time Wu, Feng
@ 2014-10-21 18:39 ` Paolo Bonzini
  2014-10-23  4:56   ` Wu, Feng
                     ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-21 18:39 UTC (permalink / raw)
  To: Wu, Feng, kvm@vger.kernel.org; +Cc: Xiao Guangrong



On 10/16/2014 10:15 AM, Wu, Feng wrote:
> Hi folks,
> 
> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
> 
> Analyze events for all VCPUs:
> 
>              VM-EXIT    Samples  Samples%     Time%   Min Time   Max Time         A
> 
>            MSR_WRITE    3613908    57.53%    18.97%        5us     1362us      9.73
>                  HLT    1399747    22.28%    74.90%        5us   432448us     99.24
>            CR_ACCESS     961203    15.30%     3.28%        4us      188us      6.33
>   EXTERNAL_INTERRUPT     213821     3.40%     2.25%        4us     4089us     19.54
>        EXCEPTION_NMI      25152     0.40%     0.12%        4us       71us      9.05
>        EPT_MISCONFIG      20104     0.32%     0.15%        8us     5628us     13.74
>                CPUID      19904     0.32%     0.07%        4us      220us      6.90
>       IO_INSTRUCTION      17097     0.27%     0.20%       13us     1008us     22.08
>    PAUSE_INSTRUCTION      10737     0.17%     0.05%        4us       53us      8.33
>             MSR_READ         48     0.00%     0.00%        4us        8us      5.62
> 
> Total Samples:6281721, Total events handled time:185457820.41us.
> 
> I also do some other experiments with different workload in the guest, I got the same results in terms of
> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
> You help!

432 ms sounds like a lot, but in general it is expected that HLT vmexits
take a long time.  After an HLT vmexit, the VCPU will not be reentered
until the next interrupt comes.  On hardware, the HLT instruction can
also take many milliseconds.

If this is an SMP guest, it's possible that the maximum time is
registered on the APs before Linux boots.  With a UP guest I would
expect a shorter maximum time, but still longer than other vmexits.

Paolo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: A question about HTL VM-Exit handling time
  2014-10-21 18:39 ` Paolo Bonzini
@ 2014-10-23  4:56   ` Wu, Feng
  2014-10-23 12:23     ` Paolo Bonzini
  2014-10-27 23:52   ` Wanpeng Li
  2014-10-28  7:46   ` Christian Borntraeger
  2 siblings, 1 reply; 8+ messages in thread
From: Wu, Feng @ 2014-10-23  4:56 UTC (permalink / raw)
  To: Paolo Bonzini, kvm@vger.kernel.org; +Cc: Xiao Guangrong



> -----Original Message-----
> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Wednesday, October 22, 2014 2:40 AM
> To: Wu, Feng; kvm@vger.kernel.org
> Cc: Xiao Guangrong
> Subject: Re: A question about HTL VM-Exit handling time
> 
> 
> 
> On 10/16/2014 10:15 AM, Wu, Feng wrote:
> > Hi folks,
> >
> > I run kernel build in the guest and use perf kvm to get some VM-Exit result as
> the following:
> >
> > Analyze events for all VCPUs:
> >
> >              VM-EXIT    Samples  Samples%     Time%   Min Time
> Max Time         A
> >
> >            MSR_WRITE    3613908    57.53%    18.97%
> 5us     1362us      9.73
> >                  HLT    1399747    22.28%    74.90%        5us
> 432448us     99.24
> >            CR_ACCESS     961203    15.30%     3.28%
> 4us      188us      6.33
> >   EXTERNAL_INTERRUPT     213821     3.40%     2.25%
> 4us     4089us     19.54
> >        EXCEPTION_NMI      25152     0.40%     0.12%
> 4us       71us      9.05
> >        EPT_MISCONFIG      20104     0.32%     0.15%
> 8us     5628us     13.74
> >                CPUID      19904     0.32%     0.07%        4us
> 220us      6.90
> >       IO_INSTRUCTION      17097     0.27%     0.20%       13us
> 1008us     22.08
> >    PAUSE_INSTRUCTION      10737     0.17%     0.05%
> 4us       53us      8.33
> >             MSR_READ         48     0.00%     0.00%
> 4us        8us      5.62
> >
> > Total Samples:6281721, Total events handled time:185457820.41us.
> >
> > I also do some other experiments with different workload in the guest, I got
> the same results in terms of
> > HLT VM-Exit handling time. Does anyone know why the handling time for HLT
> VM-Exit is so high? Appreciate
> > You help!
> 
> 432 ms sounds like a lot, but in general it is expected that HLT vmexits
> take a long time.  After an HLT vmexit, the VCPU will not be reentered
> until the next interrupt comes.  On hardware, the HLT instruction can
> also take many milliseconds.

Thanks for your answer, Paolo.
There is a general question for perf kvm. Do you know how the vmexit
handling time is measured? Such as what is the start point, and the
end point? If I got this information, I think I can better understand the
value of the handling time. Thanks a lot!


> 
> If this is an SMP guest, it's possible that the maximum time is
> registered on the APs before Linux boots.  With a UP guest I would
> expect a shorter maximum time, but still longer than other vmexits.

Yes, you are right, I get the data for an UP guest and the HLT vmexit
handling time is shorter. Here is the result:

Analyze events for all VCPUs:

             VM-EXIT    Samples  Samples%     Time%   Min Time   Max Time

  EXTERNAL_INTERRUPT      90015    29.68%    41.92%        4us      660us     24.9
           MSR_WRITE      89512    29.52%    14.25%        4us      162us      8.5
      IO_INSTRUCTION      47823    15.77%    16.67%       11us     1907us     18.6
       EPT_MISCONFIG      38504    12.70%     7.42%        7us     4848us     10.3
               CPUID      32658    10.77%     3.61%        3us      304us      5.9
       EXCEPTION_NMI       2232     0.74%     0.34%        4us      164us      8.2
   PAUSE_INSTRUCTION       1594     0.53%     0.21%        3us       44us      6.9
                 HLT        547     0.18%    15.50%        8us   177098us   1516.5
           CR_ACCESS        278     0.09%     0.05%        5us       26us      8.7
       EPT_VIOLATION         71     0.02%     0.03%        8us       27us     20.3
            MSR_READ         12     0.00%     0.00%        3us        4us      4.1

Total Samples:303246, Total events handled time:5350983.30us.

Thanks,
Feng

> 
> Paolo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: A question about HTL VM-Exit handling time
  2014-10-23  4:56   ` Wu, Feng
@ 2014-10-23 12:23     ` Paolo Bonzini
  2014-10-24  1:14       ` Wu, Feng
  0 siblings, 1 reply; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-23 12:23 UTC (permalink / raw)
  To: Wu, Feng, kvm@vger.kernel.org; +Cc: Xiao Guangrong


> Thanks for your answer, Paolo.
> There is a general question for perf kvm. Do you know how the vmexit
> handling time is measured? Such as what is the start point, and the
> end point? If I got this information, I think I can better understand the
> value of the handling time. Thanks a lot!

It is measured from kvm:kvm_exit to kvm:kvm_entry.  See
exit_event_begin, exit_event_get_key, exit_event_end in
tools/perf/builtin-kvm.c.

Paolo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: A question about HTL VM-Exit handling time
  2014-10-23 12:23     ` Paolo Bonzini
@ 2014-10-24  1:14       ` Wu, Feng
  0 siblings, 0 replies; 8+ messages in thread
From: Wu, Feng @ 2014-10-24  1:14 UTC (permalink / raw)
  To: Paolo Bonzini, kvm@vger.kernel.org; +Cc: Xiao Guangrong



> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Thursday, October 23, 2014 8:23 PM
> To: Wu, Feng; kvm@vger.kernel.org
> Cc: Xiao Guangrong
> Subject: Re: A question about HTL VM-Exit handling time
> 
> 
> > Thanks for your answer, Paolo.
> > There is a general question for perf kvm. Do you know how the vmexit
> > handling time is measured? Such as what is the start point, and the
> > end point? If I got this information, I think I can better understand the
> > value of the handling time. Thanks a lot!
> 
> It is measured from kvm:kvm_exit to kvm:kvm_entry.  See
> exit_event_begin, exit_event_get_key, exit_event_end in
> tools/perf/builtin-kvm.c.

Got it, Thanks for the information!

Thanks,
Feng

> 
> Paolo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: A question about HTL VM-Exit handling time
  2014-10-21 18:39 ` Paolo Bonzini
  2014-10-23  4:56   ` Wu, Feng
@ 2014-10-27 23:52   ` Wanpeng Li
  2014-10-28 10:16     ` Paolo Bonzini
  2014-10-28  7:46   ` Christian Borntraeger
  2 siblings, 1 reply; 8+ messages in thread
From: Wanpeng Li @ 2014-10-27 23:52 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: kvm@vger.kernel.org, Xiao Guangrong

Hi Paolo,
On Tue, Oct 21, 2014 at 08:39:49PM +0200, Paolo Bonzini wrote:
>
>
>On 10/16/2014 10:15 AM, Wu, Feng wrote:
>> Hi folks,
>> 
>> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
>> 
>> Analyze events for all VCPUs:
>> 
>>              VM-EXIT    Samples  Samples%     Time%   Min Time   Max Time         A
>> 
>>            MSR_WRITE    3613908    57.53%    18.97%        5us     1362us      9.73
>>                  HLT    1399747    22.28%    74.90%        5us   432448us     99.24
>>            CR_ACCESS     961203    15.30%     3.28%        4us      188us      6.33
>>   EXTERNAL_INTERRUPT     213821     3.40%     2.25%        4us     4089us     19.54
>>        EXCEPTION_NMI      25152     0.40%     0.12%        4us       71us      9.05
>>        EPT_MISCONFIG      20104     0.32%     0.15%        8us     5628us     13.74
>>                CPUID      19904     0.32%     0.07%        4us      220us      6.90
>>       IO_INSTRUCTION      17097     0.27%     0.20%       13us     1008us     22.08
>>    PAUSE_INSTRUCTION      10737     0.17%     0.05%        4us       53us      8.33
>>             MSR_READ         48     0.00%     0.00%        4us        8us      5.62
>> 
>> Total Samples:6281721, Total events handled time:185457820.41us.
>> 
>> I also do some other experiments with different workload in the guest, I got the same results in terms of
>> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
>> You help!
>
>432 ms sounds like a lot, but in general it is expected that HLT vmexits
>take a long time.  After an HLT vmexit, the VCPU will not be reentered
>until the next interrupt comes.  On hardware, the HLT instruction can
>also take many milliseconds.
>

As you know, kernel build which Feng used is a CPU bound workload,
so why 22.28% samples is halt?

Regards,
Wanpeng Li 

>If this is an SMP guest, it's possible that the maximum time is
>registered on the APs before Linux boots.  With a UP guest I would
>expect a shorter maximum time, but still longer than other vmexits.
>
>Paolo
>--
>To unsubscribe from this list: send the line "unsubscribe kvm" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: A question about HTL VM-Exit handling time
  2014-10-27 23:52   ` Wanpeng Li
@ 2014-10-28 10:16     ` Paolo Bonzini
  0 siblings, 0 replies; 8+ messages in thread
From: Paolo Bonzini @ 2014-10-28 10:16 UTC (permalink / raw)
  To: Wanpeng Li, Xiao Guangrong, KVM list

On 10/28/2014 12:52 AM, Wanpeng Li wrote:
> As you know, kernel build which Feng used is a CPU bound workload,
> so why 22.28% samples is halt?

Because in a CPU bound workload you don't have many exits in the first
place.  But there could also be some issues with the way he used perf.
213821 interrupts seem to be really few.

Paolo

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: A question about HTL VM-Exit handling time
  2014-10-21 18:39 ` Paolo Bonzini
  2014-10-23  4:56   ` Wu, Feng
  2014-10-27 23:52   ` Wanpeng Li
@ 2014-10-28  7:46   ` Christian Borntraeger
  2 siblings, 0 replies; 8+ messages in thread
From: Christian Borntraeger @ 2014-10-28  7:46 UTC (permalink / raw)
  To: Paolo Bonzini, Wu, Feng, kvm@vger.kernel.org
  Cc: Xiao Guangrong, Alexander Yarygin

Am 21.10.2014 20:39, schrieb Paolo Bonzini:
> 
> 
> On 10/16/2014 10:15 AM, Wu, Feng wrote:
>> Hi folks,
>>
>> I run kernel build in the guest and use perf kvm to get some VM-Exit result as the following:
>>
>> Analyze events for all VCPUs:
>>
>>              VM-EXIT    Samples  Samples%     Time%   Min Time   Max Time         A
>>
>>            MSR_WRITE    3613908    57.53%    18.97%        5us     1362us      9.73
>>                  HLT    1399747    22.28%    74.90%        5us   432448us     99.24
>>            CR_ACCESS     961203    15.30%     3.28%        4us      188us      6.33
>>   EXTERNAL_INTERRUPT     213821     3.40%     2.25%        4us     4089us     19.54
>>        EXCEPTION_NMI      25152     0.40%     0.12%        4us       71us      9.05
>>        EPT_MISCONFIG      20104     0.32%     0.15%        8us     5628us     13.74
>>                CPUID      19904     0.32%     0.07%        4us      220us      6.90
>>       IO_INSTRUCTION      17097     0.27%     0.20%       13us     1008us     22.08
>>    PAUSE_INSTRUCTION      10737     0.17%     0.05%        4us       53us      8.33
>>             MSR_READ         48     0.00%     0.00%        4us        8us      5.62
>>
>> Total Samples:6281721, Total events handled time:185457820.41us.
>>
>> I also do some other experiments with different workload in the guest, I got the same results in terms of
>> HLT VM-Exit handling time. Does anyone know why the handling time for HLT VM-Exit is so high? Appreciate
>> You help!
> 
> 432 ms sounds like a lot, but in general it is expected that HLT vmexits
> take a long time.  After an HLT vmexit, the VCPU will not be reentered
> until the next interrupt comes.  On hardware, the HLT instruction can
> also take many milliseconds.
> 
> If this is an SMP guest, it's possible that the maximum time is
> registered on the APs before Linux boots.  With a UP guest I would
> expect a shorter maximum time, but still longer than other vmexits.

We have the same on s390 with wait state. The thing is, with an idle system and NOHZ the time spend in HLT/wait could be really long. So We might want to provide an option to filter this out.
(A similar filter already exitis for the --duration option)


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2014-10-28 10:16 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-16  8:15 A question about HTL VM-Exit handling time Wu, Feng
2014-10-21 18:39 ` Paolo Bonzini
2014-10-23  4:56   ` Wu, Feng
2014-10-23 12:23     ` Paolo Bonzini
2014-10-24  1:14       ` Wu, Feng
2014-10-27 23:52   ` Wanpeng Li
2014-10-28 10:16     ` Paolo Bonzini
2014-10-28  7:46   ` Christian Borntraeger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).