[Question] About the behavior of HLT in VMX guest mode

[Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
2) If it won't, then whether it would release the hardware resources shared with
another hyper-thread ?

Any suggestion would be greatly appreciated, thanks!

-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-03-13 15:12+0800, Longpeng (Mike):
> Hi guys,
> 
> I'm confusing about the behavior of HLT instruction in VMX guest mode.
> 
> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
> 
> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
> 
> My questions are
> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

> 2) If it won't, then whether it would release the hardware resources shared with
> another hyper-thread ?

No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
a nop-spinning VCPU won't hinder the other hyper-thread?

> Any suggestion would be greatly appreciated, thanks!

Mostly just more questions, sorry ...

I'd look at temperature sensors while halting inside guests on all cores
to see if they really enter a power saving mode -- I expect a noticeable
difference from idle=poll. :)

Re: [Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

Hi, Radim,

On 2017/3/16 1:32, Radim Krčmář wrote:

> 2017-03-13 15:12+0800, Longpeng (Mike):
>> Hi guys,
>>
>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>
>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>
>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>
>> My questions are
>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
> 
> Do you get a different result when running HLT outside VMX?
> 


Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
when executing in guest.

>> 2) If it won't, then whether it would release the hardware resources shared with
>> another hyper-thread ?
> 

> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
> a nop-spinning VCPU won't hinder the other hyper-thread?
> 


I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------
			(vcpu2)idle=poll	(vcpu2)idle=halt
(HLT exiting=1)
vcpu1 costs		3800931			1900209

(HLT exiting=0)
vcpu1 costs		3800193			1913514
-----------------------------------------------------------------------

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
"HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in
guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

>> Any suggestion would be greatly appreciated, thanks!
> 
> Mostly just more questions, sorry ...
> 
> I'd look at temperature sensors while halting inside guests on all cores
> to see if they really enter a power saving mode -- I expect a noticeable
> difference from idle=poll. :)
> 
> 


-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

On 2017/3/16 10:08, Longpeng (Mike) wrote:

> Hi, Radim,
> 
> On 2017/3/16 1:32, Radim Krčmář wrote:
> 
>> 2017-03-13 15:12+0800, Longpeng (Mike):
>>> Hi guys,
>>>
>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>>
>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>>
>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>>
>>> My questions are
>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
>>
>> Do you get a different result when running HLT outside VMX?
>>
> 
> 
> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
> when executing in guest.
> 
>>> 2) If it won't, then whether it would release the hardware resources shared with
>>> another hyper-thread ?
>>
> 
>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
>> a nop-spinning VCPU won't hinder the other hyper-thread?
>>
> 
> 
> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
> 
> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
> 
> (vcpu1)
> t1 = rdtsc();
> for (int i = 0; i < 10000000; ++i) ;
> t2 = rdtsc();
> costs = t2 - t1;
> 
> (vcpu2)
> "halt" or "while (1) ;"
> 
> The result is:
> -----------------------------------------------------------------------
> 			(vcpu2)idle=poll	(vcpu2)idle=halt
> (HLT exiting=1)
> vcpu1 costs		3800931			1900209
> 
> (HLT exiting=0)
> vcpu1 costs		3800193			1913514
> -----------------------------------------------------------------------
> 
> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
> "HLT exiting" configurable, while
> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
> redundant with CFS hardlimit.
> 
> I focus on the VM's performance. According the result, I think running HLT in
> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.
> 


comparing "HLT exiting=0 && idle=halt" with "HLT exiting=1 && idle=poll" :)

>>> Any suggestion would be greatly appreciated, thanks!
>>
>> Mostly just more questions, sorry ...
>>
>> I'd look at temperature sensors while halting inside guests on all cores
>> to see if they really enter a power saving mode -- I expect a noticeable
>> difference from idle=poll. :)
>>
>>
> 
> 


-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Wanpeng Li]

2017-03-16 10:08 GMT+08:00 Longpeng (Mike) <longpeng2@huawei.com>:
> Hi, Radim,
>
> On 2017/3/16 1:32, Radim Krčmář wrote:
>
>> 2017-03-13 15:12+0800, Longpeng (Mike):
>>> Hi guys,
>>>
>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>>
>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>>
>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>>
>>> My questions are
>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
>>
>> Do you get a different result when running HLT outside VMX?
>>
>
>
> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
> when executing in guest.

Execute hlt will enter C1 state, just mwait can enter C1E state.

Regards,
Wanpeng Li

>
>>> 2) If it won't, then whether it would release the hardware resources shared with
>>> another hyper-thread ?
>>
>
>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
>> a nop-spinning VCPU won't hinder the other hyper-thread?
>>
>
>
> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
>
> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
>
> (vcpu1)
> t1 = rdtsc();
> for (int i = 0; i < 10000000; ++i) ;
> t2 = rdtsc();
> costs = t2 - t1;
>
> (vcpu2)
> "halt" or "while (1) ;"
>
> The result is:
> -----------------------------------------------------------------------
>                         (vcpu2)idle=poll        (vcpu2)idle=halt
> (HLT exiting=1)
> vcpu1 costs             3800931                 1900209
>
> (HLT exiting=0)
> vcpu1 costs             3800193                 1913514
> -----------------------------------------------------------------------
>
> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
> "HLT exiting" configurable, while
> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
> redundant with CFS hardlimit.
>
> I focus on the VM's performance. According the result, I think running HLT in
> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.
>
>>> Any suggestion would be greatly appreciated, thanks!
>>
>> Mostly just more questions, sorry ...
>>
>> I'd look at temperature sensors while halting inside guests on all cores
>> to see if they really enter a power saving mode -- I expect a noticeable
>> difference from idle=poll. :)
>>
>>
>
>
> --
> Regards,
> Longpeng(Mike)
>

Re: [Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

Hi wangpeng,

On 2017/3/16 16:51, Wanpeng Li wrote:

> 2017-03-16 10:08 GMT+08:00 Longpeng (Mike) <longpeng2@huawei.com>:
>> Hi, Radim,
>>
>> On 2017/3/16 1:32, Radim Krčmář wrote:
>>
>>> 2017-03-13 15:12+0800, Longpeng (Mike):
>>>> Hi guys,
>>>>
>>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>>>
>>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>>>
>>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>>>
>>>> My questions are
>>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
>>>
>>> Do you get a different result when running HLT outside VMX?
>>>
>>
>>
>> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
>> when executing in guest.
> 
> Execute hlt will enter C1 state, just mwait can enter C1E state.
> 


You're right, thanks. :)

So...It is really more precise to say that executing HLT in host will enter C1
state and it won't when executing in guest if "HLT exiting = 0".

> Regards,
> Wanpeng Li
> 
>>
>>>> 2) If it won't, then whether it would release the hardware resources shared with
>>>> another hyper-thread ?
>>>
>>
>>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
>>> a nop-spinning VCPU won't hinder the other hyper-thread?
>>>
>>
>>
>> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
>> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
>>
>> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
>>
>> (vcpu1)
>> t1 = rdtsc();
>> for (int i = 0; i < 10000000; ++i) ;
>> t2 = rdtsc();
>> costs = t2 - t1;
>>
>> (vcpu2)
>> "halt" or "while (1) ;"
>>
>> The result is:
>> -----------------------------------------------------------------------
>>                         (vcpu2)idle=poll        (vcpu2)idle=halt
>> (HLT exiting=1)
>> vcpu1 costs             3800931                 1900209
>>
>> (HLT exiting=0)
>> vcpu1 costs             3800193                 1913514
>> -----------------------------------------------------------------------
>>
>> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
>> "HLT exiting" configurable, while
>> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
>> redundant with CFS hardlimit.
>>
>> I focus on the VM's performance. According the result, I think running HLT in
>> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.
>>
>>>> Any suggestion would be greatly appreciated, thanks!
>>>
>>> Mostly just more questions, sorry ...
>>>
>>> I'd look at temperature sensors while halting inside guests on all cores
>>> to see if they really enter a power saving mode -- I expect a noticeable
>>> difference from idle=poll. :)
>>>
>>>
>>
>>
>> --
>> Regards,
>> Longpeng(Mike)
>>
> 
> .
> 


-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-03-16 10:08+0800, Longpeng (Mike):
> Hi, Radim,
> 
> On 2017/3/16 1:32, Radim Krčmář wrote:
> 
>> 2017-03-13 15:12+0800, Longpeng (Mike):
>>> Hi guys,
>>>
>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>>
>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>>
>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>>
>>> My questions are
>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
>> 
>> Do you get a different result when running HLT outside VMX?
>> 
> 
> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
> when executing in guest.

I'd go for the thermal monitoring (ideally with constant fan speed) if
CPU counters are lacking.  Thermal sensors are easily accessible and far
more trustworthy for measuring power saving. :)

>>> 2) If it won't, then whether it would release the hardware resources shared with
>>> another hyper-thread ?
>> 
> 
>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
>> a nop-spinning VCPU won't hinder the other hyper-thread?
>> 
> 
> 
> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
> 
> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
> 
> (vcpu1)
> t1 = rdtsc();
> for (int i = 0; i < 10000000; ++i) ;
> t2 = rdtsc();
> costs = t2 - t1;
> 
> (vcpu2)
> "halt" or "while (1) ;"
> 
> The result is:
> -----------------------------------------------------------------------
> 			(vcpu2)idle=poll	(vcpu2)idle=halt
> (HLT exiting=1)
> vcpu1 costs		3800931			1900209
> 
> (HLT exiting=0)
> vcpu1 costs		3800193			1913514
> -----------------------------------------------------------------------

Oh, great results.
I wonder if the slightly better time on HLT exiting=1 is because the
other hyper-thread goes into deeper sleep after exit.
Btw. does adding pause() into the while loop bring the performance close
to halt?

> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
> "HLT exiting" configurable, while
> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
> redundant with CFS hardlimit.
> 
> I focus on the VM's performance. According the result, I think running HLT in
> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Yes, and using MWAIT for idle is even better than HLT (you can be woken
up without IPI) -- any reason to prefer HLT?

Thanks.

Re: [Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

Hi Radim,

On 2017/3/16 22:23, Radim Krčmář wrote:

> 2017-03-16 10:08+0800, Longpeng (Mike):
>> Hi, Radim,
>>
>> On 2017/3/16 1:32, Radim Krčmář wrote:
>>
>>> 2017-03-13 15:12+0800, Longpeng (Mike):
>>>> Hi guys,
>>>>
>>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
>>>>
>>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
>>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
>>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
>>>>
>>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
>>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
>>>>
>>>> My questions are
>>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
>>>
>>> Do you get a different result when running HLT outside VMX?
>>>
>>
>> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
>> when executing in guest.
> 
> I'd go for the thermal monitoring (ideally with constant fan speed) if
> CPU counters are lacking.  Thermal sensors are easily accessible and far
> more trustworthy for measuring power saving. :)
> 
>>>> 2) If it won't, then whether it would release the hardware resources shared with
>>>> another hyper-thread ?
>>>
>>
>>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
>>> a nop-spinning VCPU won't hinder the other hyper-thread?
>>>
>>
>>
>> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
>> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
>>
>> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
>>
>> (vcpu1)
>> t1 = rdtsc();
>> for (int i = 0; i < 10000000; ++i) ;
>> t2 = rdtsc();
>> costs = t2 - t1;
>>
>> (vcpu2)
>> "halt" or "while (1) ;"
>>
>> The result is:
>> -----------------------------------------------------------------------
>> 			(vcpu2)idle=poll	(vcpu2)idle=halt
>> (HLT exiting=1)
>> vcpu1 costs		3800931			1900209
>>
>> (HLT exiting=0)
>> vcpu1 costs		3800193			1913514
>> -----------------------------------------------------------------------
> 
> Oh, great results.
> I wonder if the slightly better time on HLT exiting=1 is because the
> other hyper-thread goes into deeper sleep after exit.


Yes, maybe.

Another potential reason is maybe the host's overhead is lower.
For "HLT exiting=1 && idle=halt" the host is idle and the cpu-usage close to 0%,
while for "HLT exiting=0 && idle=halt" the host is actually very busy and the
cpu-usage close to 100%. Maybe host kernel would do more work when it's busy.

> Btw. does adding pause() into the while loop bring the performance close
> to halt?


Good suggestion! :)

I tested pause() into poll loop and set "ple_gap=0" just now, the performance is
much better than "while(1) ;", but it's still obvious slower than halt.
-----------------------------------------------------------------------
 			(vcpu2)poll	(vcpu2)pause	(vcpu2)halt
 (HLT exiting=1)
 vcpu1 costs		3800931		2572812		1916724

 (HLT exiting=0)
 vcpu1 costs		3800193		2573685		1912443
-----------------------------------------------------------------------

> 
>> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
>> "HLT exiting" configurable, while
>> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
>> redundant with CFS hardlimit.
>>
>> I focus on the VM's performance. According the result, I think running HLT in
>> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.
> 
> Yes, and using MWAIT for idle is even better than HLT (you can be woken


Yes, agree.

> up without IPI) -- any reason to prefer HLT?
> 


In my humble opinion:

1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes
entering deeper sleep) under certain conditions.
Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache.
This is insecurity if we don't take other protective measures(such as limit the
guest's max-cstate, it's fortunately that power subsystem isn't supported by
QEMU, but we should be careful for some special-purpose in case). While HLT in
guest mode can't cause hardware into sleep.

2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in
guest mode can't work as perfect as in host sometimes.
For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose
this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor
address will be cleaned, so the MWAIT won't be waken up by a store operation to
the monitored address any more.

But I'm glad to do some tests if time permits, thanks :)

Radim, how about to make HLT-exiting configurable again in upstream ? If you
like it, there is a problem should be resolved, asynpf is conflict with
"HLT-exiting = 0" in certain situations.

> Thanks.
> 
> .
> 


-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-03-17 13:22+0800, Longpeng (Mike):
> Hi Radim,
> 
> On 2017/3/16 22:23, Radim Krčmář wrote:
> 
> > 2017-03-16 10:08+0800, Longpeng (Mike):
> >> Hi, Radim,
> >>
> >> On 2017/3/16 1:32, Radim Krčmář wrote:
> >>
> >>> 2017-03-13 15:12+0800, Longpeng (Mike):
> >>>> Hi guys,
> >>>>
> >>>> I'm confusing about the behavior of HLT instruction in VMX guest mode.
> >>>>
> >>>> I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute
> >>>> HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's
> >>>> c-states, it seems that the pcpu didn't enter C1/C1E state during this period.
> >>>>
> >>>> I searched the Intel spec vol-3, and only found that guest MWAIT won't entering
> >>>> a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.
> >>>>
> >>>> My questions are
> >>>> 1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?
> >>>
> >>> Do you get a different result when running HLT outside VMX?
> >>>
> >>
> >> Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't
> >> when executing in guest.
> > 
> > I'd go for the thermal monitoring (ideally with constant fan speed) if
> > CPU counters are lacking.  Thermal sensors are easily accessible and far
> > more trustworthy for measuring power saving. :)
> > 
> >>>> 2) If it won't, then whether it would release the hardware resources shared with
> >>>> another hyper-thread ?
> >>>
> >>
> >>> No idea.  Aren't hyperthreaded resources scheduled dynamically, so even
> >>> a nop-spinning VCPU won't hinder the other hyper-thread?
> >>>
> >>
> >>
> >> I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed
> >> vcpu won't compete the hardware resources( maybe including the pipeline ) any more.
> >>
> >> My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and
> >>
> >> (vcpu1)
> >> t1 = rdtsc();
> >> for (int i = 0; i < 10000000; ++i) ;
> >> t2 = rdtsc();
> >> costs = t2 - t1;
> >>
> >> (vcpu2)
> >> "halt" or "while (1) ;"
> >>
> >> The result is:
> >> -----------------------------------------------------------------------
> >> 			(vcpu2)idle=poll	(vcpu2)idle=halt
> >> (HLT exiting=1)
> >> vcpu1 costs		3800931			1900209
> >>
> >> (HLT exiting=0)
> >> vcpu1 costs		3800193			1913514
> >> -----------------------------------------------------------------------
> > 
> > Oh, great results.
> > I wonder if the slightly better time on HLT exiting=1 is because the
> > other hyper-thread goes into deeper sleep after exit.
> 
> 
> Yes, maybe.
> 
> Another potential reason is maybe the host's overhead is lower.
> For "HLT exiting=1 && idle=halt" the host is idle and the cpu-usage close to 0%,
> while for "HLT exiting=0 && idle=halt" the host is actually very busy and the
> cpu-usage close to 100%. Maybe host kernel would do more work when it's busy.
> 
> > Btw. does adding pause() into the while loop bring the performance close
> > to halt?
> 
> 
> Good suggestion! :)
> 
> I tested pause() into poll loop and set "ple_gap=0" just now, the performance is
> much better than "while(1) ;", but it's still obvious slower than halt.
> -----------------------------------------------------------------------
>  			(vcpu2)poll	(vcpu2)pause	(vcpu2)halt
>  (HLT exiting=1)
>  vcpu1 costs		3800931		2572812		1916724
> 
>  (HLT exiting=0)
>  vcpu1 costs		3800193		2573685		1912443
> -----------------------------------------------------------------------
> 
> > 
> >> I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked
> >> "HLT exiting" configurable, while
> >> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to
> >> redundant with CFS hardlimit.
> >>
> >> I focus on the VM's performance. According the result, I think running HLT in
> >> guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.
> > 
> > Yes, and using MWAIT for idle is even better than HLT (you can be woken
> 
> 
> Yes, agree.
> 
>> up without IPI) -- any reason to prefer HLT?
>> 
> 
> 
> In my humble opinion:
> 
> 1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes
> entering deeper sleep) under certain conditions.
> Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache.
> This is insecurity if we don't take other protective measures(such as limit the
> guest's max-cstate, it's fortunately that power subsystem isn't supported by
> QEMU, but we should be careful for some special-purpose in case). While HLT in
> guest mode can't cause hardware into sleep.

Good point.  I'm not aware of any VMX capabilities to prevent deeper
C-states, so we'd always hope that guests obey provided information.

> 2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in
> guest mode can't work as perfect as in host sometimes.
> For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose
> this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor
> address will be cleaned, so the MWAIT won't be waken up by a store operation to
> the monitored address any more.

It's not as perfect, but should not cause a bug (well, there is a
discussion with suspicious MWAIT behavior :]).
MWAIT on all Intels I tested would just behave as a nop if exit happened
between MONITOR and MWAIT, like it does if you skip the MONITOR (MWAIT
instruction desciption):

  If the preceding MONITOR instruction did not successfully arm an
  address range or if the MONITOR instruction has not been executed
  prior to executing MWAIT, then the processor will not enter the
  implementation-dependent-optimized state. Execution will resume at the
  instruction following the MWAIT.

> But I'm glad to do some tests if time permits, thanks :)
> 
> Radim, how about to make HLT-exiting configurable again in upstream ? If you
> like it, there is a problem should be resolved, asynpf is conflict with
> "HLT-exiting = 0" in certain situations.

Go ahead.  KVM should provide access to hardware features and
no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a
good case) capability.  I'm interested in the asyncpf conflict.

Thanks.

Re: [Question] About the behavior of HLT in VMX guest mode

[Longpeng (Mike)]

Hi Radim,

On 2017/3/20 23:18, Radim Krčmář wrote:

> 2017-03-17 13:22+0800, Longpeng (Mike):
>> Hi Radim,
...
>> In my humble opinion:
>>
>> 1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes
>> entering deeper sleep) under certain conditions.
>> Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache.
>> This is insecurity if we don't take other protective measures(such as limit the
>> guest's max-cstate, it's fortunately that power subsystem isn't supported by
>> QEMU, but we should be careful for some special-purpose in case). While HLT in
>> guest mode can't cause hardware into sleep.
> 
> Good point.  I'm not aware of any VMX capabilities to prevent deeper
> C-states, so we'd always hope that guests obey provided information.
> 


I'll do some tests this weekend.
I plan to use MWAIT to enter deeper C-states in a testcase of kvm-unit-tests,
and start a memory-sensitive workload on another hyper-thread, then use
intel-pcm or perf to observe the count of cache miss on that core.

>> 2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in
>> guest mode can't work as perfect as in host sometimes.
>> For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose
>> this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor
>> address will be cleaned, so the MWAIT won't be waken up by a store operation to
>> the monitored address any more.
> 
> It's not as perfect, but should not cause a bug (well, there is a
> discussion with suspicious MWAIT behavior :]).
> MWAIT on all Intels I tested would just behave as a nop if exit happened
> between MONITOR and MWAIT, like it does if you skip the MONITOR (MWAIT
> instruction desciption):
> 
>   If the preceding MONITOR instruction did not successfully arm an
>   address range or if the MONITOR instruction has not been executed
>   prior to executing MWAIT, then the processor will not enter the
>   implementation-dependent-optimized state. Execution will resume at the
>   instruction following the MWAIT.
> 


OK. :)

>> But I'm glad to do some tests if time permits, thanks :)
>>
>> Radim, how about to make HLT-exiting configurable again in upstream ? If you
>> like it, there is a problem should be resolved, asynpf is conflict with
>> "HLT-exiting = 0" in certain situations.
> 
> Go ahead.  KVM should provide access to hardware features and
> no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a
> good case) capability.  I'm interested in the asyncpf conflict.
> 


I had did some offline discussion with Wanpeng Li, he's interesting to write a
path for this feature. :)

> Thanks.
> 
> .
> 


-- 
Regards,
Longpeng(Mike)

Re: [Question] About the behavior of HLT in VMX guest mode

[Wanpeng Li]

2017-03-21 9:47 GMT+08:00 Longpeng (Mike) <longpeng2@huawei.com>:
> Hi Radim,
>
> On 2017/3/20 23:18, Radim Krčmář wrote:
>
>> 2017-03-17 13:22+0800, Longpeng (Mike):
>>> Hi Radim,
> ...
>>> In my humble opinion:
>>>
>>> 1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes
>>> entering deeper sleep) under certain conditions.
>>> Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache.
>>> This is insecurity if we don't take other protective measures(such as limit the
>>> guest's max-cstate, it's fortunately that power subsystem isn't supported by
>>> QEMU, but we should be careful for some special-purpose in case). While HLT in
>>> guest mode can't cause hardware into sleep.
>>
>> Good point.  I'm not aware of any VMX capabilities to prevent deeper
>> C-states, so we'd always hope that guests obey provided information.
>>
>
>
> I'll do some tests this weekend.
> I plan to use MWAIT to enter deeper C-states in a testcase of kvm-unit-tests,
> and start a memory-sensitive workload on another hyper-thread, then use
> intel-pcm or perf to observe the count of cache miss on that core.
>
>>> 2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in
>>> guest mode can't work as perfect as in host sometimes.
>>> For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose
>>> this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor
>>> address will be cleaned, so the MWAIT won't be waken up by a store operation to
>>> the monitored address any more.
>>
>> It's not as perfect, but should not cause a bug (well, there is a
>> discussion with suspicious MWAIT behavior :]).
>> MWAIT on all Intels I tested would just behave as a nop if exit happened
>> between MONITOR and MWAIT, like it does if you skip the MONITOR (MWAIT
>> instruction desciption):
>>
>>   If the preceding MONITOR instruction did not successfully arm an
>>   address range or if the MONITOR instruction has not been executed
>>   prior to executing MWAIT, then the processor will not enter the
>>   implementation-dependent-optimized state. Execution will resume at the
>>   instruction following the MWAIT.
>>
>
>
> OK. :)
>
>>> But I'm glad to do some tests if time permits, thanks :)
>>>
>>> Radim, how about to make HLT-exiting configurable again in upstream ? If you
>>> like it, there is a problem should be resolved, asynpf is conflict with
>>> "HLT-exiting = 0" in certain situations.
>>
>> Go ahead.  KVM should provide access to hardware features and
>> no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a
>> good case) capability.  I'm interested in the asyncpf conflict.

After async pf setup successfully, there is a broadcast wakeup w/
special token 0xffffffff which tells vCPU that it should wake up all
processes waiting for APFs though there is no real process waiting at
the moment.

Refer to SDM 26.3.1.5:

HLT. The only events allowed are the following:

— Those with interruption type external interrupt or non-maskable
interrupt (NMI).
— Those with interruption type hardware exception and vector 1 (debug
exception) or vector 18 (machine-check exception).
— Those with interruption type other event and vector 0 (pending MTF VM exit).

So if the guest activity state is hlt and delivery #PF event during
vmentry, the vmentry will fail.

Refer to the original "KVM: VMX: add module parameter to avoid
trapping HLT instructions"
https://www.spinics.net/lists/kvm-commits/msg00137.html, it will set
guest activity state to active if it is hlt before manually. Actually
I wonder who set guest activity state to active when there is
HLT-exiting.

In addition, what's your design for per-VM non HLT-exiting capability?

> I had did some offline discussion with Wanpeng Li, he's interesting to write a
> path for this feature. :)

Thanks Longpeng. :)

Regards,
Wanpeng Li

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-03-21 14:21+0800, Wanpeng Li:
> 2017-03-21 9:47 GMT+08:00 Longpeng (Mike) <longpeng2@huawei.com>:
>> On 2017/3/20 23:18, Radim Krčmář wrote:
>>> 2017-03-17 13:22+0800, Longpeng (Mike):
>>>> Radim, how about to make HLT-exiting configurable again in upstream ? If you
>>>> like it, there is a problem should be resolved, asynpf is conflict with
>>>> "HLT-exiting = 0" in certain situations.
>>>
>>> Go ahead.  KVM should provide access to hardware features and
>>> no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a
>>> good case) capability.  I'm interested in the asyncpf conflict.
> 
> After async pf setup successfully, there is a broadcast wakeup w/
> special token 0xffffffff which tells vCPU that it should wake up all
> processes waiting for APFs though there is no real process waiting at
> the moment.
> 
> Refer to SDM 26.3.1.5:
> 
> HLT. The only events allowed are the following:
> 
> — Those with interruption type external interrupt or non-maskable
> interrupt (NMI).
> — Those with interruption type hardware exception and vector 1 (debug
> exception) or vector 18 (machine-check exception).
> — Those with interruption type other event and vector 0 (pending MTF VM exit).
> 
> So if the guest activity state is hlt and delivery #PF event during
> vmentry, the vmentry will fail.

I see, so we also need to change the activity state to ACTIVE when
setting most interrupts in VM_ENTRY_INTR_INFO_FIELD ...
This adds overhead to the common case and gets a little tricky with
cancel_injection(), but should turn out to be bearable.

No word about GUEST_INTR_STATUS, so posted interrupts hopefully work
without additional changes.

> Refer to the original "KVM: VMX: add module parameter to avoid
> trapping HLT instructions"
> https://www.spinics.net/lists/kvm-commits/msg00137.html, it will set
> guest activity state to active if it is hlt before manually. Actually
> I wonder who set guest activity state to active when there is
> HLT-exiting.
> 
> In addition, what's your design for per-VM non HLT-exiting capability?

I would go with an enableable per-VM KVM_CAP_X86_GUEST_HLT that can be
used by userspace to set a variable in struct kvm.

Also, please take a look at SVM -- I think that errata #415 for family
0x10 forces us to introduce a vendor specific KVM_CHECK_EXTENSION and
KVM_ENABLE_CAP, because HLT must be intercepted in that case.

Thanks.

Re: [Question] About the behavior of HLT in VMX guest mode

[Wanpeng Li]

2017-03-22 0:45 GMT+08:00 Radim Krčmář <rkrcmar@redhat.com>:
> 2017-03-21 14:21+0800, Wanpeng Li:
>> 2017-03-21 9:47 GMT+08:00 Longpeng (Mike) <longpeng2@huawei.com>:
>>> On 2017/3/20 23:18, Radim Krčmář wrote:
>>>> 2017-03-17 13:22+0800, Longpeng (Mike):
>>>>> Radim, how about to make HLT-exiting configurable again in upstream ? If you
>>>>> like it, there is a problem should be resolved, asynpf is conflict with
>>>>> "HLT-exiting = 0" in certain situations.
>>>>
>>>> Go ahead.  KVM should provide access to hardware features and
>>>> no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a
>>>> good case) capability.  I'm interested in the asyncpf conflict.
>>
>> After async pf setup successfully, there is a broadcast wakeup w/
>> special token 0xffffffff which tells vCPU that it should wake up all
>> processes waiting for APFs though there is no real process waiting at
>> the moment.
>>
>> Refer to SDM 26.3.1.5:
>>
>> HLT. The only events allowed are the following:
>>
>> — Those with interruption type external interrupt or non-maskable
>> interrupt (NMI).
>> — Those with interruption type hardware exception and vector 1 (debug
>> exception) or vector 18 (machine-check exception).
>> — Those with interruption type other event and vector 0 (pending MTF VM exit).
>>
>> So if the guest activity state is hlt and delivery #PF event during
>> vmentry, the vmentry will fail.
>
> I see, so we also need to change the activity state to ACTIVE when
> setting most interrupts in VM_ENTRY_INTR_INFO_FIELD ...
> This adds overhead to the common case and gets a little tricky with
> cancel_injection(), but should turn out to be bearable.
>
> No word about GUEST_INTR_STATUS, so posted interrupts hopefully work
> without additional changes.
>
>> Refer to the original "KVM: VMX: add module parameter to avoid
>> trapping HLT instructions"
>> https://www.spinics.net/lists/kvm-commits/msg00137.html, it will set
>> guest activity state to active if it is hlt before manually. Actually
>> I wonder who set guest activity state to active when there is
>> HLT-exiting.
>>
>> In addition, what's your design for per-VM non HLT-exiting capability?
>
> I would go with an enableable per-VM KVM_CAP_X86_GUEST_HLT that can be
> used by userspace to set a variable in struct kvm.
>
> Also, please take a look at SVM -- I think that errata #415 for family
> 0x10 forces us to introduce a vendor specific KVM_CHECK_EXTENSION and
> KVM_ENABLE_CAP, because HLT must be intercepted in that case.

I observed that the hang during the boot of guest since pvspinlock
depends on trapping the halt as mentioned in this patch:
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang
disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order
to disable the pvspinlock in guest from kvm. So if disable the
pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and
pCPU 1:1 pin can be acceptable?

Regards,
Wanpeng Li

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-07-04 12:24+0800, Wanpeng Li:
> I observed that the hang during the boot of guest since pvspinlock
> depends on trapping the halt as mentioned in this patch:
> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang
> disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order
> to disable the pvspinlock in guest from kvm. So if disable the
> pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and
> pCPU 1:1 pin can be acceptable?

I think the hang happened because one VCPU did a halt with disabled
interrupts and then ignored KVM_HC_KICK_CPU from another VCPU.
The target VCPU does a VM exit, so handling the PV unhalt should be a
simple setting of GUEST_ACTIVITY_ACTIVE upon arch.pv.pv_unhalted.

If it turns out to be complicated, then forbidding the userspace to
configure both is acceptable, (like you said, it currently doesn't make
much sense to combine them)

thanks.

Re: [Question] About the behavior of HLT in VMX guest mode

[Wanpeng Li]

2017-07-11 1:08 GMT+08:00 Radim Krčmář <rkrcmar@redhat.com>:
> 2017-07-04 12:24+0800, Wanpeng Li:
>> I observed that the hang during the boot of guest since pvspinlock
>> depends on trapping the halt as mentioned in this patch:
>> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang
>> disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order
>> to disable the pvspinlock in guest from kvm. So if disable the
>> pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and
>> pCPU 1:1 pin can be acceptable?
>
> I think the hang happened because one VCPU did a halt with disabled
> interrupts and then ignored KVM_HC_KICK_CPU from another VCPU.
> The target VCPU does a VM exit, so handling the PV unhalt should be a
> simple setting of GUEST_ACTIVITY_ACTIVE upon arch.pv.pv_unhalted.
>
> If it turns out to be complicated, then forbidding the userspace to
> configure both is acceptable, (like you said, it currently doesn't make
> much sense to combine them)

kvm_spinlock_init() just check KVM_FEATURE_PV_UNHALT once to determine
whether to enable pvspinlock, however, there is no easy way to check
per-VM halt-exiting flag in the function __do_cpuid_ent().

Regards,
Wanpeng Li

Re: [Question] About the behavior of HLT in VMX guest mode

[Radim Krčmář]

2017-07-11 18:41+0800, Wanpeng Li:
> 2017-07-11 1:08 GMT+08:00 Radim Krčmář <rkrcmar@redhat.com>:
> > 2017-07-04 12:24+0800, Wanpeng Li:
> >> I observed that the hang during the boot of guest since pvspinlock
> >> depends on trapping the halt as mentioned in this patch:
> >> http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang
> >> disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order
> >> to disable the pvspinlock in guest from kvm. So if disable the
> >> pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and
> >> pCPU 1:1 pin can be acceptable?
> >
> > I think the hang happened because one VCPU did a halt with disabled
> > interrupts and then ignored KVM_HC_KICK_CPU from another VCPU.
> > The target VCPU does a VM exit, so handling the PV unhalt should be a
> > simple setting of GUEST_ACTIVITY_ACTIVE upon arch.pv.pv_unhalted.
> >
> > If it turns out to be complicated, then forbidding the userspace to
> > configure both is acceptable, (like you said, it currently doesn't make
> > much sense to combine them)
> 
> kvm_spinlock_init() just check KVM_FEATURE_PV_UNHALT once to determine
> whether to enable pvspinlock, however, there is no easy way to check
> per-VM halt-exiting flag in the function __do_cpuid_ent().

Right, but we do not need to eliminate the flag from __do_cpuid_ent().

The documentation could say that if you enable per-VM halt-exiting, then
you cannot set KVM_FEATURE_PV_UNHALT for VCPUs in that VM.  And KVM can
check when userspace sets the CPUID.  (And at least forbid enablement of
halt-exiting if there already are created VCPUs.)

It's nothing pretty, so I hope that supporting the combination will
work. :)