* PVH cpuid feature flags @ 2014-01-21 19:28 Roger Pau Monné 2014-01-21 22:17 ` Mukesh Rathor 2014-01-24 18:31 ` Konrad Rzeszutek Wilk 0 siblings, 2 replies; 5+ messages in thread From: Roger Pau Monné @ 2014-01-21 19:28 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, Mukesh Rathor, xen-devel Hello, While doing some benchmarks on PV/PVH/PVHVM, I've realized that the cpuid feature flags exposed to PVH guests are kind of strange, this is the output of the feature flags as seen by an HVM domain: Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> And this is what a PVH domain sees when running on the same hardware: Features=0x1fc98b75<FPU,DE,TSC,MSR,PAE,CX8,APIC,SEP,CMOV,PAT,CLFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> I would expect the feature flags to be quite similar between an HVM domain and a PV domain (since they both run inside of a HVM container). AFAIK, there's no reason to disable PSE, PGE, PSE36 and RDTSCP for PVH guests. Also, is there any reason why PVH guests have the ACPI, SS and CLFLUSH feature flags but not HVM? Most (if not all) of this probably comes from the fact that we are reporting the same feature flags as pure PV guests, but I see no reason to do that for PVH guests, we should decide what's supported on PVH and set the feature flags accordingly. Roger. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PVH cpuid feature flags 2014-01-21 19:28 PVH cpuid feature flags Roger Pau Monné @ 2014-01-21 22:17 ` Mukesh Rathor 2014-01-24 18:31 ` Konrad Rzeszutek Wilk 1 sibling, 0 replies; 5+ messages in thread From: Mukesh Rathor @ 2014-01-21 22:17 UTC (permalink / raw) To: Roger Pau Monné; +Cc: xen-devel On Tue, 21 Jan 2014 20:28:14 +0100 Roger Pau Monné <roger.pau@citrix.com> wrote: > Hello, > > While doing some benchmarks on PV/PVH/PVHVM, I've realized that the > cpuid feature flags exposed to PVH guests are kind of strange, this is > the output of the feature flags as seen by an HVM domain: > > Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> > Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV> > AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > AMD Features2=0x1<LAHF> > > And this is what a PVH domain sees when running on the same hardware: > > Features=0x1fc98b75<FPU,DE,TSC,MSR,PAE,CX8,APIC,SEP,CMOV,PAT,CLFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> > Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > > I would expect the feature flags to be quite similar between an HVM > domain and a PV domain (since they both run inside of a HVM > container). AFAIK, there's no reason to disable PSE, PGE, PSE36 and > RDTSCP for PVH guests. Also, is there any reason why PVH guests have > the ACPI, SS and CLFLUSH feature flags but not HVM? > > Most (if not all) of this probably comes from the fact that we are > reporting the same feature flags as pure PV guests, but I see no Right, I already said I was working on xen patch to export CR4 and other features. But, it will take couple days, as I'm debugging somethig else right now. If you can't wait, go for it. BTW, if you have time to work on PVH, can you pl help with migration? Would be very much appreciated.... thanks Mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PVH cpuid feature flags 2014-01-21 19:28 PVH cpuid feature flags Roger Pau Monné 2014-01-21 22:17 ` Mukesh Rathor @ 2014-01-24 18:31 ` Konrad Rzeszutek Wilk 2014-01-27 12:06 ` Roger Pau Monné 1 sibling, 1 reply; 5+ messages in thread From: Konrad Rzeszutek Wilk @ 2014-01-24 18:31 UTC (permalink / raw) To: Roger Pau Monné; +Cc: xen-devel On Tue, Jan 21, 2014 at 08:28:14PM +0100, Roger Pau Monné wrote: > Hello, > > While doing some benchmarks on PV/PVH/PVHVM, I've realized that the > cpuid feature flags exposed to PVH guests are kind of strange, this is > the output of the feature flags as seen by an HVM domain: > What about a PV guest? I presume if you ran an NetBSD PV guest it would give a format similar to this? > Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> > Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV> > AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > AMD Features2=0x1<LAHF> > > And this is what a PVH domain sees when running on the same hardware: > > Features=0x1fc98b75<FPU,DE,TSC,MSR,PAE,CX8,APIC,SEP,CMOV,PAT,CLFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> > Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > > I would expect the feature flags to be quite similar between an HVM > domain and a PV domain (since they both run inside of a HVM container). > AFAIK, there's no reason to disable PSE, PGE, PSE36 and RDTSCP for PVH > guests. Also, is there any reason why PVH guests have the ACPI, SS and > CLFLUSH feature flags but not HVM? S5? ACPI is enabled for PV I think, but Linux PV guests disable them as there is no ACPI tables in PV mode: 429 if (!xen_initial_domain()) 430 cpuid_leaf1_edx_mask &= 431 ~((1 << X86_FEATURE_ACPI)); /* disable ACPI */ 432 434 CLFLUSH - no idea why it would be disabled. The rdsctp should be enabled. In the past I think it was related to 'timer=' option. We would either trap it, or emulate it with a constant value, or passthrough. It should be passing it through but maybe there is a bug? PSE, PGE, PSE36, PG1GB, etc, should all be exposed. Actually the PG1TB is not exposed because of another bug: http://www.gossamer-threads.com/lists/xen/devel/313596 > > Most (if not all) of this probably comes from the fact that we are > reporting the same feature flags as pure PV guests, but I see no reason > to do that for PVH guests, we should decide what's supported on PVH and > set the feature flags accordingly. Right and also have a nice policy. The problem is that we set/unset the cpuid flags in the toolstack (and in two places - depending on the architecture) and also in the hypervisor. Anyhow, these I know we disable: 425 cpuid_leaf1_edx_mask = 426 ~((1 << X86_FEATURE_MTRR) | /* disable MTRR */ 427 (1 << X86_FEATURE_ACC)); /* thermal monitoring */ And I think your patch to the Xen hypervisor does it too - it clears the MTRR by default now. The ACC is (if my memory is correct) for the Pentium 4 and such - we can disable it as well. 428 433 cpuid_leaf1_ecx_mask &= ~(1 << (X86_FEATURE_X2APIC % 32)); And this we definitly need to disable. The x2APIC should not be exposed as we want to use the Xen's version of APIC ops. And if the x2APIC bit is enabled in Linux, there is some other code (NMI handler) that will use that without using the APIC ops. And blow up :-( Then there is the MWAIT. Actually we can put that on the side. I know it is important for dom0, but since we don't have those patches yet in, we can ignore that. However, the hypervisor (pv_cpuid) disables it. There is also the XSAVE business: save_mask = 440 (1 << (X86_FEATURE_XSAVE % 32)) | 441 (1 << (X86_FEATURE_OSXSAVE % 32)); 442 443 /* Xen will set CR4.OSXSAVE if supported and not disabled by force */ 444 if ((cx & xsave_mask) != xsave_mask) 445 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & OSXSAVE */ Which I am not clear about. This looks like to make a uniform 'cpuid' look in the hypervisor we need somehow to glue hvm_cpuid and pv_cpuid with some form of tables/lookups. And make sure that the same logic is reflected in xc_cpuid_x86.c (toolstack). > > Roger. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PVH cpuid feature flags 2014-01-24 18:31 ` Konrad Rzeszutek Wilk @ 2014-01-27 12:06 ` Roger Pau Monné 2014-01-31 15:49 ` Konrad Rzeszutek Wilk 0 siblings, 1 reply; 5+ messages in thread From: Roger Pau Monné @ 2014-01-27 12:06 UTC (permalink / raw) To: Konrad Rzeszutek Wilk; +Cc: xen-devel On 24/01/14 19:31, Konrad Rzeszutek Wilk wrote: > On Tue, Jan 21, 2014 at 08:28:14PM +0100, Roger Pau Monné wrote: >> Hello, >> >> While doing some benchmarks on PV/PVH/PVHVM, I've realized that the >> cpuid feature flags exposed to PVH guests are kind of strange, this is >> the output of the feature flags as seen by an HVM domain: >> > > What about a PV guest? I presume if you ran an NetBSD PV guest it would > give a format similar to this? I guess so, the feature flags reported by NetBSD PV will probably be the same of the ones reported by FreeBSD PVH (unless NetBSD PV also does some kind of pre-filtering). > >> Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> >> Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV> >> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> >> AMD Features2=0x1<LAHF> >> >> And this is what a PVH domain sees when running on the same hardware: >> >> Features=0x1fc98b75<FPU,DE,TSC,MSR,PAE,CX8,APIC,SEP,CMOV,PAT,CLFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> >> Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV> >> AMD Features=0x20100800<SYSCALL,NX,LM> >> AMD Features2=0x1<LAHF> >> >> I would expect the feature flags to be quite similar between an HVM >> domain and a PV domain (since they both run inside of a HVM container). >> AFAIK, there's no reason to disable PSE, PGE, PSE36 and RDTSCP for PVH >> guests. Also, is there any reason why PVH guests have the ACPI, SS and >> CLFLUSH feature flags but not HVM? > > S5? SS - CPU cache supports self-snoop. Not sure if that should be enabled or not for PVH. > > ACPI is enabled for PV I think, but Linux PV guests disable them > as there is no ACPI tables in PV mode: > > 429 if (!xen_initial_domain()) > 430 cpuid_leaf1_edx_mask &= > 431 ~((1 << X86_FEATURE_ACPI)); /* disable ACPI */ > 432 > 434 > > CLFLUSH - no idea why it would be disabled. > > > The rdsctp should be enabled. In the past I think it was related to > 'timer=' option. We would either trap it, or emulate it with a constant > value, or passthrough. It should be passing it through but maybe there > is a bug? > > PSE, PGE, PSE36, PG1GB, etc, should all be exposed. Actually the PG1TB > is not exposed because of another bug: > http://www.gossamer-threads.com/lists/xen/devel/313596 Think so, now that we run inside of an HVM container we should be able to make use of all those. > >> >> Most (if not all) of this probably comes from the fact that we are >> reporting the same feature flags as pure PV guests, but I see no reason >> to do that for PVH guests, we should decide what's supported on PVH and >> set the feature flags accordingly. > > Right and also have a nice policy. The problem is that we set/unset > the cpuid flags in the toolstack (and in two places - depending on the > architecture) and also in the hypervisor. Yes, all this cpuid flag stuff seems like a mess to me, there are so many places where we enable or blacklist certain cpu flags that makes me wonder if it would be more sane to define a set of flags that an HVM container supports and maybe then blacklist some of them if they are not actually implemented/usable on the specific kind of guest. > Anyhow, these I know we disable: > > 425 cpuid_leaf1_edx_mask = > 426 ~((1 << X86_FEATURE_MTRR) | /* disable MTRR */ > 427 (1 << X86_FEATURE_ACC)); /* thermal monitoring */ > > And I think your patch to the Xen hypervisor does it too - it clears > the MTRR by default now. The ACC is (if my memory is correct) for > the Pentium 4 and such - we can disable it as well. > > 428 > 433 cpuid_leaf1_ecx_mask &= ~(1 << (X86_FEATURE_X2APIC % 32)); > > And this we definitly need to disable. The x2APIC should not > be exposed as we want to use the Xen's version of APIC ops. And > if the x2APIC bit is enabled in Linux, there is some other code > (NMI handler) that will use that without using the APIC ops. > And blow up :-( > > > Then there is the MWAIT. Actually we can put that on the side. > I know it is important for dom0, but since we don't have those > patches yet in, we can ignore that. However, the hypervisor > (pv_cpuid) disables it. > > > There is also the XSAVE business: > > save_mask = > 440 (1 << (X86_FEATURE_XSAVE % 32)) | > 441 (1 << (X86_FEATURE_OSXSAVE % 32)); > 442 > 443 /* Xen will set CR4.OSXSAVE if supported and not disabled by force */ > 444 if ((cx & xsave_mask) != xsave_mask) > 445 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & OSXSAVE */ > > Which I am not clear about. > > > This looks like to make a uniform 'cpuid' look in the hypervisor > we need somehow to glue hvm_cpuid and pv_cpuid with some > form of tables/lookups. > > And make sure that the same logic is reflected in > xc_cpuid_x86.c (toolstack). Agree. On a slight different topic, why do we enable the APIC flags for PV(H) guests? We certainly have no APIC, and makes me wonder if we should disable it now and enable it once we have hardware APIC virtualization in place. This would allow PVH to either use the traditional PV style, or a hardware virtualized APIC if we enable it for PVH guests (and make the guest aware of it by turning the flag). Roger. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: PVH cpuid feature flags 2014-01-27 12:06 ` Roger Pau Monné @ 2014-01-31 15:49 ` Konrad Rzeszutek Wilk 0 siblings, 0 replies; 5+ messages in thread From: Konrad Rzeszutek Wilk @ 2014-01-31 15:49 UTC (permalink / raw) To: Roger Pau Monné; +Cc: xen-devel On Mon, Jan 27, 2014 at 01:06:55PM +0100, Roger Pau Monné wrote: > On 24/01/14 19:31, Konrad Rzeszutek Wilk wrote: > > On Tue, Jan 21, 2014 at 08:28:14PM +0100, Roger Pau Monné wrote: > >> Hello, > >> > >> While doing some benchmarks on PV/PVH/PVHVM, I've realized that the > >> cpuid feature flags exposed to PVH guests are kind of strange, this is > >> the output of the feature flags as seen by an HVM domain: > >> > > > > What about a PV guest? I presume if you ran an NetBSD PV guest it would > > give a format similar to this? > > I guess so, the feature flags reported by NetBSD PV will probably be the > same of the ones reported by FreeBSD PVH (unless NetBSD PV also does > some kind of pre-filtering). > > > > >> Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT> > >> Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV> > >> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> > >> AMD Features2=0x1<LAHF> > >> > >> And this is what a PVH domain sees when running on the same hardware: > >> > >> Features=0x1fc98b75<FPU,DE,TSC,MSR,PAE,CX8,APIC,SEP,CMOV,PAT,CLFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT> > >> Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,HV> > >> AMD Features=0x20100800<SYSCALL,NX,LM> > >> AMD Features2=0x1<LAHF> > >> > >> I would expect the feature flags to be quite similar between an HVM > >> domain and a PV domain (since they both run inside of a HVM container). > >> AFAIK, there's no reason to disable PSE, PGE, PSE36 and RDTSCP for PVH > >> guests. Also, is there any reason why PVH guests have the ACPI, SS and > >> CLFLUSH feature flags but not HVM? > > > > S5? > > SS - CPU cache supports self-snoop. > > Not sure if that should be enabled or not for PVH. Not even sure what does means for a guest. I thought all CPUs do snooping except when they go in C states? > > > > > ACPI is enabled for PV I think, but Linux PV guests disable them > > as there is no ACPI tables in PV mode: > > > > 429 if (!xen_initial_domain()) > > 430 cpuid_leaf1_edx_mask &= > > 431 ~((1 << X86_FEATURE_ACPI)); /* disable ACPI */ > > 432 > > 434 > > > > CLFLUSH - no idea why it would be disabled. > > > > > > The rdsctp should be enabled. In the past I think it was related to > > 'timer=' option. We would either trap it, or emulate it with a constant > > value, or passthrough. It should be passing it through but maybe there > > is a bug? > > > > PSE, PGE, PSE36, PG1GB, etc, should all be exposed. Actually the PG1TB > > is not exposed because of another bug: > > http://www.gossamer-threads.com/lists/xen/devel/313596 > > Think so, now that we run inside of an HVM container we should be able > to make use of all those. Agreed. > > > > >> > >> Most (if not all) of this probably comes from the fact that we are > >> reporting the same feature flags as pure PV guests, but I see no reason > >> to do that for PVH guests, we should decide what's supported on PVH and > >> set the feature flags accordingly. > > > > Right and also have a nice policy. The problem is that we set/unset > > the cpuid flags in the toolstack (and in two places - depending on the > > architecture) and also in the hypervisor. > > Yes, all this cpuid flag stuff seems like a mess to me, there are so > many places where we enable or blacklist certain cpu flags that makes me > wonder if it would be more sane to define a set of flags that an HVM > container supports and maybe then blacklist some of them if they are not > actually implemented/usable on the specific kind of guest. So, first go through HVM list and then follow with the PV? > > > Anyhow, these I know we disable: > > > > 425 cpuid_leaf1_edx_mask = > > 426 ~((1 << X86_FEATURE_MTRR) | /* disable MTRR */ > > 427 (1 << X86_FEATURE_ACC)); /* thermal monitoring */ > > > > And I think your patch to the Xen hypervisor does it too - it clears > > the MTRR by default now. The ACC is (if my memory is correct) for > > the Pentium 4 and such - we can disable it as well. > > > > 428 > > 433 cpuid_leaf1_ecx_mask &= ~(1 << (X86_FEATURE_X2APIC % 32)); > > > > And this we definitly need to disable. The x2APIC should not > > be exposed as we want to use the Xen's version of APIC ops. And > > if the x2APIC bit is enabled in Linux, there is some other code > > (NMI handler) that will use that without using the APIC ops. > > And blow up :-( > > > > > > Then there is the MWAIT. Actually we can put that on the side. > > I know it is important for dom0, but since we don't have those > > patches yet in, we can ignore that. However, the hypervisor > > (pv_cpuid) disables it. > > > > > > There is also the XSAVE business: > > > > save_mask = > > 440 (1 << (X86_FEATURE_XSAVE % 32)) | > > 441 (1 << (X86_FEATURE_OSXSAVE % 32)); > > 442 > > 443 /* Xen will set CR4.OSXSAVE if supported and not disabled by force */ > > 444 if ((cx & xsave_mask) != xsave_mask) > > 445 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & OSXSAVE */ > > > > Which I am not clear about. > > > > > > This looks like to make a uniform 'cpuid' look in the hypervisor > > we need somehow to glue hvm_cpuid and pv_cpuid with some > > form of tables/lookups. > > > > And make sure that the same logic is reflected in > > xc_cpuid_x86.c (toolstack). > > Agree. On a slight different topic, why do we enable the APIC flags for > PV(H) guests? > > We certainly have no APIC, and makes me wonder if we should disable it > now and enable it once we have hardware APIC virtualization in place. > This would allow PVH to either use the traditional PV style, or a > hardware virtualized APIC if we enable it for PVH guests (and make the > guest aware of it by turning the flag). I thought it was off for PV guests (except dom0). That is my recollection when 'perf' starts and realizes it can't sample the APIC. But when it comes to dom0 it needs that otherwise it won't even parse the ACPI MADT tables - and you need to parse those as you need to get INT_SRV_OVR right. The reason you need that is because once the ACPI AML code kicks in, it has to figure out which GSI is the ACPI SCI and the INT_SRV_OVR might have an override (like pin 9 might be hard-wired to pin 20 instead of 9). In other words - need this for dom0 to let it do its work. > > Roger. > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-01-31 15:49 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-21 19:28 PVH cpuid feature flags Roger Pau Monné 2014-01-21 22:17 ` Mukesh Rathor 2014-01-24 18:31 ` Konrad Rzeszutek Wilk 2014-01-27 12:06 ` Roger Pau Monné 2014-01-31 15:49 ` Konrad Rzeszutek Wilk
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).