Host latency peaks due to kvm-intel

Host latency peaks due to kvm-intel

[Jan Kiszka]

Hi,

did anyone recently tried current KVM for Intel over some real-time
Linux? I'm seeing more than 500 us latency peaks on the host,
specifically during VM startup. This applies to both 2.6.29.6-rt23 and
Xenomai/I-pipe. For -rt, I both tried the included (patched) KVM modules
as well as kvm.git head with some additionally required -rt fixes.
Xenomai ran over a 2.6.30 kernel with my own KVM-enabler patch.

Early instrumentation actually points to the guest exit itself: I added
markers right before and after the assembly part of vmx_vcpu_run, and
further instrumentation reports that the next host APIC tick should go
off right inside guest mode. But KVM leaves the switching part 500 us
too late in that case - as if guest exit on external IRQs was disabled.

Will debug this further, but I'm also curious to hear other user
experiences.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: Host latency peaks due to kvm-intel

[Gregory Haskins]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

Jan Kiszka wrote:
> Hi,
>
> did anyone recently tried current KVM for Intel over some real-time
> Linux? I'm seeing more than 500 us latency peaks on the host,
> specifically during VM startup. This applies to both 2.6.29.6-rt23 and
> Xenomai/I-pipe. For -rt, I both tried the included (patched) KVM modules
> as well as kvm.git head with some additionally required -rt fixes.
> Xenomai ran over a 2.6.30 kernel with my own KVM-enabler patch.
>
> Early instrumentation actually points to the guest exit itself: I added
> markers right before and after the assembly part of vmx_vcpu_run, and
> further instrumentation reports that the next host APIC tick should go
> off right inside guest mode. But KVM leaves the switching part 500 us
> too late in that case - as if guest exit on external IRQs was disabled.
>
> Will debug this further, but I'm also curious to hear other user
> experiences.
>
> Jan
>
>   
Hi Jan,
  Did you try to run with latency-tracer enabled?  If not, this may
pinpoint the source for you.

Regards,
-Greg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

Re: Host latency peaks due to kvm-intel

[Jan Kiszka]

Gregory Haskins wrote:
> Jan Kiszka wrote:
>> Hi,
>>
>> did anyone recently tried current KVM for Intel over some real-time
>> Linux? I'm seeing more than 500 us latency peaks on the host,
>> specifically during VM startup. This applies to both 2.6.29.6-rt23 and
>> Xenomai/I-pipe. For -rt, I both tried the included (patched) KVM modules
>> as well as kvm.git head with some additionally required -rt fixes.
>> Xenomai ran over a 2.6.30 kernel with my own KVM-enabler patch.
>>
>> Early instrumentation actually points to the guest exit itself: I added
>> markers right before and after the assembly part of vmx_vcpu_run, and
>> further instrumentation reports that the next host APIC tick should go
>> off right inside guest mode. But KVM leaves the switching part 500 us
>> too late in that case - as if guest exit on external IRQs was disabled.
>>
>> Will debug this further, but I'm also curious to hear other user
>> experiences.
>>
>> Jan
>>
>>   
> Hi Jan,
>   Did you try to run with latency-tracer enabled?  If not, this may
> pinpoint the source for you.

I did, see above.

It finally turned out that I got burned once again by wbinvd: My test
CPUs (and likely also some of my customer) are too "old" to support
SECONDARY_VM_EXEC_CONTROL, which includes trapping guest's wbinvd
invocations. Too bad.

I vaguely recall that someone promised to add a feature reporting
facility for all those nice things, modern VM-extensions may or may not
support (something like or even an extension of /proc/cpuinfo). What is
the state of this plan? Would be specifically interesting for Intel CPUs
as there seem to be many of them out there with restrictions for special
use cases - like real-time.

Jan (who is now patching his guest to avoid wbinvd where possible)

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: Host latency peaks due to kvm-intel

[Gregory Haskins]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

Jan Kiszka wrote:
> Gregory Haskins wrote:
>   
>> Jan Kiszka wrote:
>>     
>>> Hi,
>>>
>>> did anyone recently tried current KVM for Intel over some real-time
>>> Linux? I'm seeing more than 500 us latency peaks on the host,
>>> specifically during VM startup. This applies to both 2.6.29.6-rt23 and
>>> Xenomai/I-pipe. For -rt, I both tried the included (patched) KVM modules
>>> as well as kvm.git head with some additionally required -rt fixes.
>>> Xenomai ran over a 2.6.30 kernel with my own KVM-enabler patch.
>>>
>>> Early instrumentation actually points to the guest exit itself: I added
>>> markers right before and after the assembly part of vmx_vcpu_run, and
>>> further instrumentation reports that the next host APIC tick should go
>>> off right inside guest mode. But KVM leaves the switching part 500 us
>>> too late in that case - as if guest exit on external IRQs was disabled.
>>>
>>> Will debug this further, but I'm also curious to hear other user
>>> experiences.
>>>
>>> Jan
>>>
>>>   
>>>       
>> Hi Jan,
>>   Did you try to run with latency-tracer enabled?  If not, this may
>> pinpoint the source for you.
>>     
>
> I did, see above.
>   

Ah, sorry.  It wasn't clear what "instrumenation" was or if you felt it
was definitively pinpointed.  :P

Regards,
-Greg



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]

Re: Host latency peaks due to kvm-intel

[Avi Kivity]

On 07/24/2009 12:41 PM, Jan Kiszka wrote:
> I vaguely recall that someone promised to add a feature reporting
> facility for all those nice things, modern VM-extensions may or may not
> support (something like or even an extension of /proc/cpuinfo). What is
> the state of this plan? Would be specifically interesting for Intel CPUs
> as there seem to be many of them out there with restrictions for special
> use cases - like real-time.
>    

Newer kernels do report some vmx features (like flexpriority) in 
/proc/cpuinfo but not all.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Host latency peaks due to kvm-intel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3117 bytes --]

Avi Kivity wrote:
> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>> I vaguely recall that someone promised to add a feature reporting
>> facility for all those nice things, modern VM-extensions may or may not
>> support (something like or even an extension of /proc/cpuinfo). What is
>> the state of this plan? Would be specifically interesting for Intel CPUs
>> as there seem to be many of them out there with restrictions for special
>> use cases - like real-time.
>>    
> 
> Newer kernels do report some vmx features (like flexpriority) in
> /proc/cpuinfo but not all.
> 

Ah, nice. Then we just need this?

------------>

From: Jan Kiszka <jan.kiszka@siemens.com>
Subject: [PATCH] x86: Report VMX feature vwbinvd

Not all VMX-capable CPUs support guest exists on wbinvd execution. If
this is not supported, the instruction will run natively on behalf of
the guest. This can cause multi-millisecond latencies to the host which
is very problematic in real-time scenarios.

Report the wbinvd trapping feature along with other VMX feature flags,
calling it 'vwbinvd' ('virtual wbinvd').

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

 arch/x86/include/asm/cpufeature.h |    1 +
 arch/x86/kernel/cpu/intel.c       |    4 ++++
 2 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h
index 4a28d22..8647524 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -165,6 +165,7 @@
 #define X86_FEATURE_FLEXPRIORITY (8*32+ 2) /* Intel FlexPriority */
 #define X86_FEATURE_EPT         (8*32+ 3) /* Intel Extended Page Table */
 #define X86_FEATURE_VPID        (8*32+ 4) /* Intel Virtual Processor ID */
+#define X86_FEATURE_VWBINVD     (8*32+ 5) /* Guest Exiting on WBINVD */
 
 #if defined(__KERNEL__) && !defined(__ASSEMBLY__)
 
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 3260ab0..2d921b0 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -297,6 +297,7 @@ static void __cpuinit detect_vmx_virtcap(struct cpuinfo_x86 *c)
 #define X86_VMX_FEATURE_PROC_CTLS2_VIRT_APIC	0x00000001
 #define X86_VMX_FEATURE_PROC_CTLS2_EPT		0x00000002
 #define X86_VMX_FEATURE_PROC_CTLS2_VPID		0x00000020
+#define X86_VMX_FEATURE_PROC_CTLS2_VWBINVD	0x00000040
 
 	u32 vmx_msr_low, vmx_msr_high, msr_ctl, msr_ctl2;
 
@@ -305,6 +306,7 @@ static void __cpuinit detect_vmx_virtcap(struct cpuinfo_x86 *c)
 	clear_cpu_cap(c, X86_FEATURE_FLEXPRIORITY);
 	clear_cpu_cap(c, X86_FEATURE_EPT);
 	clear_cpu_cap(c, X86_FEATURE_VPID);
+	clear_cpu_cap(c, X86_FEATURE_VWBINVD);
 
 	rdmsr(MSR_IA32_VMX_PROCBASED_CTLS, vmx_msr_low, vmx_msr_high);
 	msr_ctl = vmx_msr_high | vmx_msr_low;
@@ -323,6 +325,8 @@ static void __cpuinit detect_vmx_virtcap(struct cpuinfo_x86 *c)
 			set_cpu_cap(c, X86_FEATURE_EPT);
 		if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_VPID)
 			set_cpu_cap(c, X86_FEATURE_VPID);
+		if (msr_ctl2 & X86_VMX_FEATURE_PROC_CTLS2_VWBINVD)
+			set_cpu_cap(c, X86_FEATURE_VWBINVD);
 	}
 }
 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Host latency peaks due to kvm-intel

[Avi Kivity]

On 07/25/2009 12:55 PM, Jan Kiszka wrote:
> Avi Kivity wrote:
>    
>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>      
>>> I vaguely recall that someone promised to add a feature reporting
>>> facility for all those nice things, modern VM-extensions may or may not
>>> support (something like or even an extension of /proc/cpuinfo). What is
>>> the state of this plan? Would be specifically interesting for Intel CPUs
>>> as there seem to be many of them out there with restrictions for special
>>> use cases - like real-time.
>>>
>>>        
>> Newer kernels do report some vmx features (like flexpriority) in
>> /proc/cpuinfo but not all.
>>
>>      
>
> Ah, nice. Then we just need this?
>
> ------------>
>
> From: Jan Kiszka<jan.kiszka@siemens.com>
> Subject: [PATCH] x86: Report VMX feature vwbinvd
>
> Not all VMX-capable CPUs support guest exists on wbinvd execution. If
> this is not supported, the instruction will run natively on behalf of
> the guest. This can cause multi-millisecond latencies to the host which
> is very problematic in real-time scenarios.
>
> Report the wbinvd trapping feature along with other VMX feature flags,
> calling it 'vwbinvd' ('virtual wbinvd').
>
>    

What about AMD cpus that can always trap wbinvd?  do we set the bit or 
do we trust the user to know that it isn't needed on AMD (I suppose the 
latter)?

This should go in via tip.git, it isn't really kvm related (except that 
kvm should start reading these caps one day instead of querying the 
hardware directly).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Host latency peaks due to kvm-intel

[Avi Kivity]

On 07/24/2009 12:41 PM, Jan Kiszka wrote:
> Jan (who is now patching his guest to avoid wbinvd where possible)
>
>    

Is there ever a case where it is required?  What about under a 
hypervisor (i.e. check the hypervisor enabled bit).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Host latency peaks due to kvm-intel

[Sujit Karataparambil]

> Do not meddle in the internals of kernels, for they are subtle and quick to
> panic.
Also the kvm code. Are you sure that the processor supports KVM
Extension. I know of a lot of intel architectures where KVM is not
support. Especially the HW_CHECK_SUM. Might not be sure, but this sure
seems an problem. Also there is no dependency check with the KVM on
Linux. What I mean by this is that KVM Install on an Architecture that
donot support the extension without problem. So compiling KVM alone
does not mean it works on an architecture.


-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)

Re: Host latency peaks due to kvm-intel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

Avi Kivity wrote:
> On 07/25/2009 12:55 PM, Jan Kiszka wrote:
>> Avi Kivity wrote:
>>   
>>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>>     
>>>> I vaguely recall that someone promised to add a feature reporting
>>>> facility for all those nice things, modern VM-extensions may or may not
>>>> support (something like or even an extension of /proc/cpuinfo). What is
>>>> the state of this plan? Would be specifically interesting for Intel
>>>> CPUs
>>>> as there seem to be many of them out there with restrictions for
>>>> special
>>>> use cases - like real-time.
>>>>
>>>>        
>>> Newer kernels do report some vmx features (like flexpriority) in
>>> /proc/cpuinfo but not all.
>>>
>>>      
>>
>> Ah, nice. Then we just need this?
>>
>> ------------>
>>
>> From: Jan Kiszka<jan.kiszka@siemens.com>
>> Subject: [PATCH] x86: Report VMX feature vwbinvd
>>
>> Not all VMX-capable CPUs support guest exists on wbinvd execution. If
>> this is not supported, the instruction will run natively on behalf of
>> the guest. This can cause multi-millisecond latencies to the host which
>> is very problematic in real-time scenarios.
>>
>> Report the wbinvd trapping feature along with other VMX feature flags,
>> calling it 'vwbinvd' ('virtual wbinvd').
>>
>>    
> 
> What about AMD cpus that can always trap wbinvd?  do we set the bit or
> do we trust the user to know that it isn't needed on AMD (I suppose the
> latter)?

I also think that the feature flags should remain vendor-specific.

> 
> This should go in via tip.git, it isn't really kvm related (except that
> kvm should start reading these caps one day instead of querying the
> hardware directly).
> 

OK, will go that way. Probably I will also add some flags for AMD's NPT,
Intel's EPT and they new unrestricted guest mode at this chance.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Host latency peaks due to kvm-intel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

Avi Kivity wrote:
> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>> Jan (who is now patching his guest to avoid wbinvd where possible)
>>
>>    
> 
> Is there ever a case where it is required?  What about under a
> hypervisor (i.e. check the hypervisor enabled bit).
> 

Reminds me of the discussion in '07 when I first stumbled over this :) :
Yes, the bochs bios could safely skip the wbinvd in qemu mode. But that
won't safe us from Linux and - far more problematic - Windows or any
binary-only guest which think they have to issue it.

One may the close eyes, fire up the guest and then start the
time-critical host application in the hope that the guest remains calm
as long as it's up and running. But, well...

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Host latency peaks due to kvm-intel

[Avi Kivity]

On 07/26/2009 05:34 PM, Jan Kiszka wrote:
> Avi Kivity wrote:
>    
>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>      
>>> Jan (who is now patching his guest to avoid wbinvd where possible)
>>>
>>>
>>>        
>> Is there ever a case where it is required?  What about under a
>> hypervisor (i.e. check the hypervisor enabled bit).
>>
>>      
>
> Reminds me of the discussion in '07 when I first stumbled over this :) :
> Yes, the bochs bios could safely skip the wbinvd in qemu mode. But that
> won't safe us from Linux and - far more problematic - Windows or any
> binary-only guest which think they have to issue it.
>
> One may the close eyes, fire up the guest and then start the
> time-critical host application in the hope that the guest remains calm
> as long as it's up and running. But, well...
>    

Given that it's now '09, how critical is the problem?  Don't most cpus 
have vwbinvd now?

If so, the real-time management application can simply refuse to run on 
such an old processor.

-- 
error compiling committee.c: too many arguments to function

Re: Host latency peaks due to kvm-intel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

Avi Kivity wrote:
> On 07/26/2009 05:34 PM, Jan Kiszka wrote:
>> Avi Kivity wrote:
>>   
>>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>>     
>>>> Jan (who is now patching his guest to avoid wbinvd where possible)
>>>>
>>>>
>>>>        
>>> Is there ever a case where it is required?  What about under a
>>> hypervisor (i.e. check the hypervisor enabled bit).
>>>
>>>      
>>
>> Reminds me of the discussion in '07 when I first stumbled over this :) :
>> Yes, the bochs bios could safely skip the wbinvd in qemu mode. But that
>> won't safe us from Linux and - far more problematic - Windows or any
>> binary-only guest which think they have to issue it.
>>
>> One may the close eyes, fire up the guest and then start the
>> time-critical host application in the hope that the guest remains calm
>> as long as it's up and running. But, well...
>>    
> 
> Given that it's now '09, how critical is the problem?  Don't most cpus
> have vwbinvd now?

Sadly, in (embedded) industry you have to live with "old" hardware for
quite a long time. And I would have to throw my only 2-years-old
notebook from the table to have a more decent portable test environment.

> 
> If so, the real-time management application can simply refuse to run on
> such an old processor.
> 

At least one could go and collect the cpuinfo from some box that suffers
from high latencies. Normally, you go through extensive testing anyway,
also checking for issues like crazy SMI BIOS code that runs for eternities.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Host latency peaks due to kvm-intel

[H. Peter Anvin]

Jan Kiszka wrote:
> Avi Kivity wrote:
>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>> I vaguely recall that someone promised to add a feature reporting
>>> facility for all those nice things, modern VM-extensions may or may not
>>> support (something like or even an extension of /proc/cpuinfo). What is
>>> the state of this plan? Would be specifically interesting for Intel CPUs
>>> as there seem to be many of them out there with restrictions for special
>>> use cases - like real-time.
>>>    
>> Newer kernels do report some vmx features (like flexpriority) in
>> /proc/cpuinfo but not all.
>>
> 
> Ah, nice. Then we just need this?
> 

Fine with me.

Acked-by: H. Peter Anvin <hpa@zytor.com>

However, I guess the real question if we shouldn't export ALL VMX
features in a consistent way instead?

	-hpa

Re: Host latency peaks due to kvm-intel

[Yang, Sheng]

On Monday 27 July 2009 03:16:27 H. Peter Anvin wrote:
> Jan Kiszka wrote:
> > Avi Kivity wrote:
> >> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
> >>> I vaguely recall that someone promised to add a feature reporting
> >>> facility for all those nice things, modern VM-extensions may or may not
> >>> support (something like or even an extension of /proc/cpuinfo). What is
> >>> the state of this plan? Would be specifically interesting for Intel
> >>> CPUs as there seem to be many of them out there with restrictions for
> >>> special use cases - like real-time.
> >>
> >> Newer kernels do report some vmx features (like flexpriority) in
> >> /proc/cpuinfo but not all.
> >
> > Ah, nice. Then we just need this?
>
> Fine with me.
>
> Acked-by: H. Peter Anvin <hpa@zytor.com>
>
> However, I guess the real question if we shouldn't export ALL VMX
> features in a consistent way instead?
>
When I add feature reporting to cpuinfo, I just put highlight features there, 
otherwise the VMX feature list would at least as long as CPU one.

I have also suggested another field for virtualization feature for it, but 
some concern again userspace tools raised.

For we got indeed quite a lot features, and would get more, would it better to 
export the part of struct vmcs_config entries(that's pin_based_exec_ctrl, 
cpu_based_exec_ctrl, and cpu_based_2nd_exec_ctrl) through 
sys/module/kvm_intel/? Put every feature to cpuinfo seems not that necessary 
for such a big list.

-- 
regards
Yang, Sheng

cpuinfo and HVM features (was: Host latency peaks due to kvm-intel)

[Jan Kiszka]

[ carrying this to LKML ]

Yang, Sheng wrote:
> On Monday 27 July 2009 03:16:27 H. Peter Anvin wrote:
>> Jan Kiszka wrote:
>>> Avi Kivity wrote:
>>>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
>>>>> I vaguely recall that someone promised to add a feature reporting
>>>>> facility for all those nice things, modern VM-extensions may or may not
>>>>> support (something like or even an extension of /proc/cpuinfo). What is
>>>>> the state of this plan? Would be specifically interesting for Intel
>>>>> CPUs as there seem to be many of them out there with restrictions for
>>>>> special use cases - like real-time.
>>>> Newer kernels do report some vmx features (like flexpriority) in
>>>> /proc/cpuinfo but not all.
>>> Ah, nice. Then we just need this?
>> Fine with me.
>>
>> Acked-by: H. Peter Anvin <hpa@zytor.com>
>>
>> However, I guess the real question if we shouldn't export ALL VMX
>> features in a consistent way instead?
>>
> When I add feature reporting to cpuinfo, I just put highlight features there, 
> otherwise the VMX feature list would at least as long as CPU one.

That could become true. But the question is always what the highlights
are. Often this depends on the hypervisor as it may implement
workarounds for missing features differently (or not at all). So I'm
also for exposing feature information consistently.

> 
> I have also suggested another field for virtualization feature for it, but 
> some concern again userspace tools raised.
> 
> For we got indeed quite a lot features, and would get more, would it better to 
> export the part of struct vmcs_config entries(that's pin_based_exec_ctrl, 
> cpu_based_exec_ctrl, and cpu_based_2nd_exec_ctrl) through 
> sys/module/kvm_intel/? Put every feature to cpuinfo seems not that necessary 
> for such a big list.

I don't think this information should only come from KVM. Consider you
didn't build it into some kernel but still want to find out what your
system is able to provide.

What about adding some dedicated /proc entry for CPU virtualization
features, say /proc/hvminfo?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: cpuinfo and HVM features (was: Host latency peaks due to kvm-intel)

[Yang, Sheng]

On Monday 27 July 2009 17:08:42 Jan Kiszka wrote:
> [ carrying this to LKML ]
>
> Yang, Sheng wrote:
> > On Monday 27 July 2009 03:16:27 H. Peter Anvin wrote:
> >> Jan Kiszka wrote:
> >>> Avi Kivity wrote:
> >>>> On 07/24/2009 12:41 PM, Jan Kiszka wrote:
> >>>>> I vaguely recall that someone promised to add a feature reporting
> >>>>> facility for all those nice things, modern VM-extensions may or may
> >>>>> not support (something like or even an extension of /proc/cpuinfo).
> >>>>> What is the state of this plan? Would be specifically interesting for
> >>>>> Intel CPUs as there seem to be many of them out there with
> >>>>> restrictions for special use cases - like real-time.
> >>>>
> >>>> Newer kernels do report some vmx features (like flexpriority) in
> >>>> /proc/cpuinfo but not all.
> >>>
> >>> Ah, nice. Then we just need this?
> >>
> >> Fine with me.
> >>
> >> Acked-by: H. Peter Anvin <hpa@zytor.com>
> >>
> >> However, I guess the real question if we shouldn't export ALL VMX
> >> features in a consistent way instead?
> >
> > When I add feature reporting to cpuinfo, I just put highlight features
> > there, otherwise the VMX feature list would at least as long as CPU one.
>
> That could become true. But the question is always what the highlights
> are. Often this depends on the hypervisor as it may implement
> workarounds for missing features differently (or not at all). So I'm
> also for exposing feature information consistently.

(CC Andi and Ingo)

The highlight means the feature we would gain a lot, like FlexPriority, EPT, 
VPID. They can be vendor specific. And I am talking about hardware capability 
here, so what's hypervisor did for workaround is not in scope.
>
> > I have also suggested another field for virtualization feature for it,
> > but some concern again userspace tools raised.
> >
> > For we got indeed quite a lot features, and would get more, would it
> > better to export the part of struct vmcs_config entries(that's
> > pin_based_exec_ctrl, cpu_based_exec_ctrl, and cpu_based_2nd_exec_ctrl)
> > through
> > sys/module/kvm_intel/? Put every feature to cpuinfo seems not that
> > necessary for such a big list.
>
> I don't think this information should only come from KVM. Consider you
> didn't build it into some kernel but still want to find out what your
> system is able to provide.

Yes, agree.
>
> What about adding some dedicated /proc entry for CPU virtualization
> features, say /proc/hvminfo?

Well, compared to this, I may still prefer a new item in /proc/cpuinfo, for 
it's still CPU feature, like Andi did for power management(IIRC).

Any more preferred location?

-- 
regards
Yang, Sheng

Re: cpuinfo and HVM features

[Avi Kivity]

On 07/27/2009 12:08 PM, Jan Kiszka wrote:
>> When I add feature reporting to cpuinfo, I just put highlight features there,
>> otherwise the VMX feature list would at least as long as CPU one.
>>      
>
> That could become true. But the question is always what the highlights
> are. Often this depends on the hypervisor as it may implement
> workarounds for missing features differently (or not at all). So I'm
> also for exposing feature information consistently.
>    


I'd put everything in there.  It's information that is often useful.  
Even minor features can expose bugs in the hypervisor.

>> I have also suggested another field for virtualization feature for it, but
>> some concern again userspace tools raised.
>>
>> For we got indeed quite a lot features, and would get more, would it better to
>> export the part of struct vmcs_config entries(that's pin_based_exec_ctrl,
>> cpu_based_exec_ctrl, and cpu_based_2nd_exec_ctrl) through
>> sys/module/kvm_intel/? Put every feature to cpuinfo seems not that necessary
>> for such a big list.
>>      
>
> I don't think this information should only come from KVM. Consider you
> didn't build it into some kernel but still want to find out what your
> system is able to provide.
>
> What about adding some dedicated /proc entry for CPU virtualization
> features, say /proc/hvminfo?
>    

The flags line is already very long, and already has some virt features, 
so I see no problem extending it.  If we don't want that. I'd prefer a 
virtualization  line in /proc/cpuinfo rather than a new file.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.