Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Zhao Liu]

(I'd like to CC Sean again to discuss the possibility of user space
 removing arch-capabilities completely for AMD)

On Mon, Jun 30, 2025 at 03:30:25PM +0200, Alexandre Chartre wrote:
> Date: Mon, 30 Jun 2025 15:30:25 +0200
> From: Alexandre Chartre <alexandre.chartre@oracle.com>
> Subject: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD
> X-Mailer: git-send-email 2.43.5
> 
> KVM emulates the ARCH_CAPABILITIES on x86 for both Intel and AMD
> cpus, although the IA32_ARCH_CAPABILITIES MSR is an Intel-specific
> MSR and it makes no sense to emulate it on AMD.
> 
> As a consequence, VMs created on AMD with qemu -cpu host and using
> KVM will advertise the ARCH_CAPABILITIES feature and provide the
> IA32_ARCH_CAPABILITIES MSR. This can cause issues (like Windows BSOD)
> as the guest OS might not expect this MSR to exist on such cpus (the
> AMD documentation specifies that ARCH_CAPABILITIES feature and MSR
> are not defined on the AMD architecture).

This issue looks very similar to this one that others in the community
reported:

https://gitlab.com/qemu-project/qemu/-/issues/3001

But there's a little difference, pls see the below comment...

> A fix was proposed in KVM code, however KVM maintainers don't want to
> change this behavior that exists for 6+ years and suggest changes to be
> done in qemu instead.
>
> So this commit changes the behavior in qemu so that ARCH_CAPABILITIES
> is not provided by default on AMD cpus when the hypervisor emulates it,
> but it can still be provided by explicitly setting arch-capabilities=on.
> 
> Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
> ---
>  target/i386/cpu.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 0d35e95430..7e136c48df 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -8324,6 +8324,20 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>          }
>      }
>  
> +    /*
> +     * For years, KVM has inadvertently emulated the ARCH_CAPABILITIES
> +     * MSR on AMD although this is an Intel-specific MSR; and KVM will
> +     * continue doing so to not change its ABI for existing setups.
> +     *
> +     * So ensure that the ARCH_CAPABILITIES MSR is disabled on AMD cpus
> +     * to prevent providing a cpu with an MSR which is not supposed to
> +     * be there,

Yes, disabling this feature bit makes sense on AMD platform. It's fine
for -cpu host.

> unless it was explicitly requested by the user.

But this could still break Windows, just like issue #3001, which enables
arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
value would even break something.

So even for named CPUs, arch-capabilities=on doesn't reflect the fact
that it is purely emulated, and is (maybe?) harmful.

> +     */
> +    if (IS_AMD_CPU(env) &&
> +        !(env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
> +        env->features[FEAT_7_0_EDX] &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
> +    }
> +

I was considering whether we should tweak it in kvm_arch_get_supported_cpuid()
until I saw this:

else if (function == 7 && index == 0 && reg == R_EDX) {
        /* Not new instructions, just an optimization.  */
        uint32_t edx;
        host_cpuid(7, 0, &unused, &unused, &unused, &edx);
        ret |= edx & CPUID_7_0_EDX_FSRM;

        /*
         * Linux v4.17-v4.20 incorrectly return ARCH_CAPABILITIES on SVM hosts.
         * We can detect the bug by checking if MSR_IA32_ARCH_CAPABILITIES is
         * returned by KVM_GET_MSR_INDEX_LIST.
         */
        if (!has_msr_arch_capabs) {
            ret &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
        }
}

What a pity! QEMU had previously workedaround CPUID_7_0_EDX_ARCH_CAPABILITIES
correctly, but since then kvm's commit 0cf9135b773b("KVM: x86: Emulate
MSR_IA32_ARCH_CAPABILITIES on AMD hosts") breaks the balance once again. 
I understand the commit, and it makes up for the mismatch between the
emulated feature bit and the MSR. Now the Windows exposes the problem of
such emulation.

So, to avoid endless workaround thereafter, I think it's time to just
disable arch-capabilities for AMD Guest (after all, closer to the real
hardware environment is better).

Further, it helps to eliminate kernel/kvm concerns when user space resolves
the legacy issues first. At least, IMO, pushing ABI changes in kernel/kvm
needs to show that there is no destruction of pre-existing user space, so
I believe a complete cleanup of QEMU is the appropriate approach.

The attached code is just some simple example to show what I think:
Starting with QEMU v10.1 for AMD Guest, to disable arch-capabilties
feature bit and MSR.

I don't have an AMD CPU, so it's untested. You can feel free to squash
it in your patch. If so, it's better to add a "Resolves" tag in your
commit message:

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3001

Thanks,
Zhao
---
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index b2116335752d..c175e7d9e7b8 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -81,7 +81,9 @@
     { "qemu64-" TYPE_X86_CPU, "model-id", "QEMU Virtual CPU version " v, },\
     { "athlon-" TYPE_X86_CPU, "model-id", "QEMU Virtual CPU version " v, },

-GlobalProperty pc_compat_10_0[] = {};
+GlobalProperty pc_compat_10_0[] = {
+    { TYPE_X86_CPU, "x-amd-disable-arch-capabs", "false" },
+};
 const size_t pc_compat_10_0_len = G_N_ELEMENTS(pc_compat_10_0);

 GlobalProperty pc_compat_9_2[] = {};
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 9aa0ea447860..a8e83efd83f6 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -8336,10 +8336,12 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
      *
      * So ensure that the ARCH_CAPABILITIES MSR is disabled on AMD cpus
      * to prevent providing a cpu with an MSR which is not supposed to
-     * be there, unless it was explicitly requested by the user.
+     * be there.
      */
-    if (IS_AMD_CPU(env) &&
-        !(env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
+    if (cpu->amd_disable_arch_capabs && IS_AMD_CPU(env)) {
+        mark_unavailable_features(cpu, FEAT_7_0_EDX,
+            env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES,
+            "This feature is not available for AMD Guest");
         env->features[FEAT_7_0_EDX] &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
     }

@@ -9414,6 +9416,8 @@ static const Property x86_cpu_properties[] = {
     DEFINE_PROP_BOOL("x-intel-pt-auto-level", X86CPU, intel_pt_auto_level,
                      true),
     DEFINE_PROP_BOOL("x-l1-cache-per-thread", X86CPU, l1_cache_per_core, true),
+    DEFINE_PROP_BOOL("x-amd-disable-arch-capabs", X86CPU, amd_disable_arch_capabs,
+                     true),
 };

 #ifndef CONFIG_USER_ONLY
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 51e10139dfdf..a3fc80de3a75 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -2306,6 +2306,13 @@ struct ArchCPU {
      */
     uint32_t guest_phys_bits;

+    /*
+     * Compatibility bits for old machine types.
+     * If true disable CPUID_7_0_EDX_ARCH_CAPABILITIES and
+     * MSR_IA32_ARCH_CAPABILITIES for AMD Guest.
+     */
+    bool amd_disable_arch_capabs;
+
     /* in order to simplify APIC support, we leave this pointer to the
        user */
     struct DeviceState *apic_state;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 234878c613f6..40a50ae193c7 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -2368,6 +2368,11 @@ int kvm_arch_init_vcpu(CPUState *cs)

     cpu->kvm_msr_buf = g_malloc0(MSR_BUF_SIZE);

+    if (cpu->amd_disable_arch_capabs &&
+        !(env->features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
+        has_msr_arch_capabs = false;
+    }
+
     if (!(env->features[FEAT_8000_0001_EDX] & CPUID_EXT2_RDTSCP)) {
         has_msr_tsc_aux = false;
     }

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Xiaoyao Li]

On 7/1/2025 6:26 PM, Zhao Liu wrote:
>> unless it was explicitly requested by the user.
> But this could still break Windows, just like issue #3001, which enables
> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> value would even break something.
> 
> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> that it is purely emulated, and is (maybe?) harmful.

It is because Windows adds wrong code. So it breaks itself and it's just 
the regression of Windows.

KVM and QEMU are not supposed to be blamed.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Alexandre Chartre]

On 7/1/25 13:12, Xiaoyao Li wrote:
> On 7/1/2025 6:26 PM, Zhao Liu wrote:
>>> unless it was explicitly requested by the user.
>> But this could still break Windows, just like issue #3001, which enables
>> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
>> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
>> value would even break something.
>>
>> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
>> that it is purely emulated, and is (maybe?) harmful.
> 
> It is because Windows adds wrong code. So it breaks itself and it's just the regression of Windows.
> 
> KVM and QEMU are not supposed to be blamed.

I can understand the Windows code logic, and I don't think it is necessarily wrong,
because it finds that the system has:

- an AMD cpu
- an Intel-only feature/MSR

Then what should the code do? Trust the cpu type (AMD) or trust the MSR (Intel).
They decided not to choose, and for safety they stop because they have an unexpected
configuration.

alex.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Xiaoyao Li]

On 7/1/2025 8:12 PM, Alexandre Chartre wrote:
> 
> On 7/1/25 13:12, Xiaoyao Li wrote:
>> On 7/1/2025 6:26 PM, Zhao Liu wrote:
>>>> unless it was explicitly requested by the user.
>>> But this could still break Windows, just like issue #3001, which enables
>>> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
>>> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
>>> value would even break something.
>>>
>>> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
>>> that it is purely emulated, and is (maybe?) harmful.
>>
>> It is because Windows adds wrong code. So it breaks itself and it's 
>> just the regression of Windows.
>>
>> KVM and QEMU are not supposed to be blamed.
> 
> I can understand the Windows code logic, and I don't think it is 
> necessarily wrong,
> because it finds that the system has:
> 
> - an AMD cpu
> - an Intel-only feature/MSR
> 
> Then what should the code do? Trust the cpu type (AMD) or trust the MSR 
> (Intel).
> They decided not to choose, and for safety they stop because they have 
> an unexpected
> configuration.

It's not how software/OS is supposed to work with x86 architecture.

Though there are different vendors for x86, like Intel and AMD, they 
both implement x86 architecture. For x86 architecture, architectural 
features are enumerated by CPUID. If you read Intel SDM and AMD APM, you 
will find that Intel defines most features at range [0, x] while AMD 
defines most features at range [0x8000 000, 0x8000 000y]. But if a bit 
is defined by both Intel and AMD, it must have same meaning and 
enumerate the same feature.

Usually, a feature is first introduced by one vendor, then other vendors 
might implement the same one later. E.g., bus lock detection, which is 
enumerated via CPUID.7_0:ECX[bit 24] and first introduced by Intel in 
2020. Later, AMD implemented the same one from Zen 5. Before AMD 
implemented it, it was an Intel-only feature. Can we make code as below

    if (is_AMD && cpuid_enumerates_bus_lock_detect)
	error(unsupported CPU);

at that time? If we wrote such code, then it will fail on all the AMD 
Zen 5 CPUs.

Besides, I would like to talk about how software is supposed to deal 
with reserved bits on x86 architecture. In general, software should not 
set any expectation on the reserved bit. The value cannot be relied upon 
to be 0 since any reserved bit can have a meaning in the future. As Igor 
said:

   software shouldn't even try to use it or make any decisions
   based on that

For more information, you can refer to Intel SDM vol1. chapter 1.3.2 
Reserved Bits and Software compatibility. For AMD APM, you would need 
search yourself.

OK, back to the original question "what should the code do?"

My answer is, it can behave with any of below option:

- Be vendor agnostic and stick to x86 architecture. If CPUID enumerates 
a feature, then the feature is available architecturally.

- Based on AMD spec. Ignore the bit since it's a reserved bit. (Expect a 
reserved bit to be zero if not explicitly state by spec is totally wrong!)

> alex.
> 

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Konrad Rzeszutek Wilk]

..snip..
> OK, back to the original question "what should the code do?"
> 
> My answer is, it can behave with any of below option:
> 
> - Be vendor agnostic and stick to x86 architecture. If CPUID enumerates a
> feature, then the feature is available architecturally.

Exactly. That is what we believe Windows does.


By this logic KVM is at fault for exposing this irregardless of the
platform (when using -cpu host). And Sean (the KVM maintainer) agrees it is
a bug. But he does not want it in the kernel due to guest ABI and hence
the ask is to put this in QEMU.


> 
> - Based on AMD spec. Ignore the bit since it's a reserved bit. (Expect a
> reserved bit to be zero if not explicitly state by spec is totally wrong!)

Which may change in the future as AMD may expose this CPUID in say X
years. And at that point the first option you enumerated is the more
safer one.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Daniel P. Berrangé]

On Tue, Jul 01, 2025 at 03:59:12PM -0400, Konrad Rzeszutek Wilk wrote:
> ..snip..
> > OK, back to the original question "what should the code do?"
> > 
> > My answer is, it can behave with any of below option:
> > 
> > - Be vendor agnostic and stick to x86 architecture. If CPUID enumerates a
> > feature, then the feature is available architecturally.
> 
> Exactly. That is what we believe Windows does.
> 
> 
> By this logic KVM is at fault for exposing this irregardless of the
> platform (when using -cpu host). And Sean (the KVM maintainer) agrees it is
> a bug. But he does not want it in the kernel due to guest ABI and hence
> the ask is to put this in QEMU.

If QEMU unconditionally disables this on AMD, and a future AMD CPU
does implement it, then QEMU is now broken because it won't be fully
exposing valid features impl by the host CPU and supported by KVM.

IOW, if we're going to have QEMU workaround the KVM mistake, then
the code change needs to be more refined.

QEMU needs to first check whether the host CPU implements
ARCH_CAPABILITIES and conditionally disable it in the guest CPU
based on that host CPU check. Of course that would re-expose the
Windows guest bug, but that ceases to be KVM/QEMU's problem at
that point, as we'd be following a genuine physical CPU impl.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Sean Christopherson]

On Mon, Jul 07, 2025, Daniel P. Berrangé wrote:
> On Tue, Jul 01, 2025 at 03:59:12PM -0400, Konrad Rzeszutek Wilk wrote:
> > ..snip..
> > > OK, back to the original question "what should the code do?"
> > > 
> > > My answer is, it can behave with any of below option:
> > > 
> > > - Be vendor agnostic and stick to x86 architecture. If CPUID enumerates a
> > > feature, then the feature is available architecturally.
> > 
> > Exactly. That is what we believe Windows does.
> > 
> > 
> > By this logic KVM is at fault for exposing this irregardless of the
> > platform (when using -cpu host). And Sean (the KVM maintainer) agrees it is
> > a bug. But he does not want it in the kernel due to guest ABI and hence
> > the ask is to put this in QEMU.
> 
> If QEMU unconditionally disables this on AMD, and a future AMD CPU
> does implement it, then QEMU is now broken because it won't be fully
> exposing valid features impl by the host CPU and supported by KVM.
> 
> IOW, if we're going to have QEMU workaround the KVM mistake, then
> the code change needs to be more refined.
> 
> QEMU needs to first check whether the host CPU implements
> ARCH_CAPABILITIES and conditionally disable it in the guest CPU
> based on that host CPU check. Of course that would re-expose the
> Windows guest bug, but that ceases to be KVM/QEMU's problem at
> that point, as we'd be following a genuine physical CPU impl.

+1

In a perfect world, we'd quirk this in KVM.  But to avoid a potentially breaking
ABI change, KVM's quirky behavior would need to remain the default behavior, i.e.
wouldn't actually help because QEMU would still need to be updated to opt out of
the quirk.

That, and KVM's quirk system is per-VM, whereas KVM_GET_SUPPORTED_CPUID is a
/dev/kvm ioctl.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Zhao Liu]

On Tue, Jul 01, 2025 at 07:12:44PM +0800, Xiaoyao Li wrote:
> Date: Tue, 1 Jul 2025 19:12:44 +0800
> From: Xiaoyao Li <xiaoyao.li@intel.com>
> Subject: Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised
>  on AMD
> 
> On 7/1/2025 6:26 PM, Zhao Liu wrote:
> > > unless it was explicitly requested by the user.
> > But this could still break Windows, just like issue #3001, which enables
> > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > value would even break something.
> > 
> > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > that it is purely emulated, and is (maybe?) harmful.
> 
> It is because Windows adds wrong code. So it breaks itself and it's just the
> regression of Windows.

Could you please tell me what the Windows's wrong code is? And what's
wrong when someone is following the hardware spec?

Do you expect software developers to make special modifications for QEMU
after following the hardware spec? Or do you categorize this behavior as
paravirtualization?

Resolving this issue within QEMU is already a win-win approach. I don't
understand why you're shifting the blame onto Windows.

> KVM and QEMU are not supposed to be blamed.

I do not think I'm blaming anything. So many people report
this bug issue in QEMU community, and maintainer suggested a solution.

I totally agree on this way, and provide feedback to help thoroughly
resolve the issue and prevent similar situations from happening again.

That's all.

Thanks,
Zhao

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Igor Mammedov]

On Tue, 1 Jul 2025 20:36:43 +0800
Zhao Liu <zhao1.liu@intel.com> wrote:

> On Tue, Jul 01, 2025 at 07:12:44PM +0800, Xiaoyao Li wrote:
> > Date: Tue, 1 Jul 2025 19:12:44 +0800
> > From: Xiaoyao Li <xiaoyao.li@intel.com>
> > Subject: Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised
> >  on AMD
> > 
> > On 7/1/2025 6:26 PM, Zhao Liu wrote:  
> > > > unless it was explicitly requested by the user.  
> > > But this could still break Windows, just like issue #3001, which enables
> > > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > > value would even break something.
> > > 
> > > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > > that it is purely emulated, and is (maybe?) harmful.  
> > 
> > It is because Windows adds wrong code. So it breaks itself and it's just the
> > regression of Windows.  
> 
> Could you please tell me what the Windows's wrong code is? And what's
> wrong when someone is following the hardware spec?

the reason is that it's reserved on AMD hence software shouldn't even try
to use it or make any decisions based on that.


PS:
on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
guest would actually complicate QEMU for no big reason.

Also
KVM does do have plenty of such code, and it's not actively preventing guests from using it.
Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.


> Do you expect software developers to make special modifications for QEMU
> after following the hardware spec? Or do you categorize this behavior as
> paravirtualization?
> 
> Resolving this issue within QEMU is already a win-win approach. I don't
> understand why you're shifting the blame onto Windows.
> 
> > KVM and QEMU are not supposed to be blamed.  
> 
> I do not think I'm blaming anything. So many people report
> this bug issue in QEMU community, and maintainer suggested a solution.
> 
> I totally agree on this way, and provide feedback to help thoroughly
> resolve the issue and prevent similar situations from happening again.
> 
> That's all.
> 
> Thanks,
> Zhao
> 
> 
> 

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Konrad Rzeszutek Wilk]

On Tue, Jul 01, 2025 at 03:05:00PM +0200, Igor Mammedov wrote:
> On Tue, 1 Jul 2025 20:36:43 +0800
> Zhao Liu <zhao1.liu@intel.com> wrote:
> 
> > On Tue, Jul 01, 2025 at 07:12:44PM +0800, Xiaoyao Li wrote:
> > > Date: Tue, 1 Jul 2025 19:12:44 +0800
> > > From: Xiaoyao Li <xiaoyao.li@intel.com>
> > > Subject: Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised
> > >  on AMD
> > > 
> > > On 7/1/2025 6:26 PM, Zhao Liu wrote:  
> > > > > unless it was explicitly requested by the user.  
> > > > But this could still break Windows, just like issue #3001, which enables
> > > > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > > > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > > > value would even break something.
> > > > 
> > > > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > > > that it is purely emulated, and is (maybe?) harmful.  
> > > 
> > > It is because Windows adds wrong code. So it breaks itself and it's just the
> > > regression of Windows.  
> > 
> > Could you please tell me what the Windows's wrong code is? And what's
> > wrong when someone is following the hardware spec?
> 
> the reason is that it's reserved on AMD hence software shouldn't even try
> to use it or make any decisions based on that.
> 
> 
> PS:
> on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
> guest would actually complicate QEMU for no big reason.

The guest is not misbehaving. It is following the spec.
> 
> Also
> KVM does do have plenty of such code, and it's not actively preventing guests from using it.
> Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.

Because KVM maintainer does not want to touch the guest ABI. He agrees
this is a bug.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Zhao Liu]

Thanks Igor for looking here and thanks Konrad's explanation.

> > > > On 7/1/2025 6:26 PM, Zhao Liu wrote:  
> > > > > > unless it was explicitly requested by the user.  
> > > > > But this could still break Windows, just like issue #3001, which enables
> > > > > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > > > > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > > > > value would even break something.
> > > > > 
> > > > > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > > > > that it is purely emulated, and is (maybe?) harmful.  
> > > > 
> > > > It is because Windows adds wrong code. So it breaks itself and it's just the
> > > > regression of Windows.  
> > > 
> > > Could you please tell me what the Windows's wrong code is? And what's
> > > wrong when someone is following the hardware spec?
> > 
> > the reason is that it's reserved on AMD hence software shouldn't even try
> > to use it or make any decisions based on that.
> > 
> > 
> > PS:
> > on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
> > guest would actually complicate QEMU for no big reason.
> 
> The guest is not misbehaving. It is following the spec.

(That's my thinking, and please feel free to correct me.)

I had the same thought. Windows guys could also say they didn't access
the reserved MSR unconditionally, and they followed the CPUID feature
bit to access that MSR. When CPUID is set, it indicates that feature is
implemented.

At least I think it makes sense to rely on the CPUID to access the MSR.
Just as an example, it's unlikely that after the software finds a CPUID
of 1, it still need to download the latest spec version to confirm
whether the feature is actually implemented or reserved.

Based on the above point, this CPUID feature bit is set to 1 in KVM and
KVM also adds emulation (as a fix) specifically for this MSR. This means
that Guest is considered to have valid access to this feature MSR,
except that if Guest doesn't get what it wants, then it is reasonable
for Guest to assume that the current (v)CPU lacks hardware support and
mark it as "unsupported processor".

As Konrad's mentioned, there's the previous explanation about why KVM
sets this feature bit (it started with a little accident):

https://lore.kernel.org/kvm/CALMp9eRjDczhSirSismObZnzimxq4m+3s6Ka7OxwPj5Qj6X=BA@mail.gmail.com/#t

So I think the question is where this fix should be applied (KVM or
QEMU) or if it should be applied at all, rather than whether Windows has
the bug.

But I do agree, such "cleanups" would complicate QEMU, as I listed
Eduardo as having done similar workaround six years ago:

https://lore.kernel.org/qemu-devel/20190125220606.4864-1-ehabkost@redhat.com/

Complexity and technical debt is an important consideration, and another
consideration is the impact of this issue. Luckily, newer versions of
Windows are actively compatible with KVM + QEMU:

https://blogs.windows.com/windows-insider/2025/06/23/announcing-windows-11-insider-preview-build-26120-4452-beta-channel/

But it's also hard to say if such a problem will happen again.
Especially if the software works fine on real hardware but fails in
"-host cpu" (which is supposed synchronized with host as much as
possible).

> > Also
> > KVM does do have plenty of such code, and it's not actively preventing guests from using it.
> > Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.
> 
> Because KVM maintainer does not want to touch the guest ABI. He agrees
> this is a bug.

If we agree on this fix should be applied on Linux virtualization stack,
then the question of whether the fix should land in KVM or QEMU is a bit
like the chicken and egg dilemma.

I personally think it might be better to roll it out in QEMU first — it
feels like the safer bet:

 * Currently, the -cpu host option enables this feature by default, and
   it's hard to say if anyone is actually relying on this emulated
   feature (though issue #3001 suggests it causes trouble for Windows).
   So only when the ABI changes, it's uncertain if anything will break.

 * Similarly, if only the ABI is changed, I'm a bit unsure if there's
   any migration based on "-cpu host" and between different versions of
   kernel. And, based on my analysis at the beginning reply, named CPUs
   also have the effect if user actively sets "arch-capbilities=on". But
   the key here point is the migration happens between different kernel
   versions.

 * Additionally, handling different versions of ABI can sometimes be
   quite complex. After changing the ABI, there might be differences
   between the new kernel and the old stable kernels (and according to
   doc, the oldest supported kernel is v4.5 - docs/system/target-i386.rst).
   It's similar to what Ewan previously complained about:

   https://lore.kernel.org/qemu-devel/53119b66-3528-41d6-ac44-df166699500a@zhaoxin.com/

So, if anyone is relying on the current emulated feature, changing the
ABI will inevitably break existing things, and QEMU might have to bear
the cost of maintaining compatibility with the old ABI. :-(

Personally, I think the safer approach is to first handle potential old
dependencies in QEMU through a compat option. Once the use case is
eliminated in user space, it can clearly demonstrate that the ABI change
won't disrupt user space.

The workaround change I proposed to Alexandre isn't meant to be
permanent. If we upgrade the supported kernel version to >6.17 (assuming
the ABI can be changed in 6.17), then the workaround can be removed —
though I admit that day might never come...

Thanks for your patience.
Zhao

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Zhao Liu]

> > > > Could you please tell me what the Windows's wrong code is? And what's
> > > > wrong when someone is following the hardware spec?
> > > 
> > > the reason is that it's reserved on AMD hence software shouldn't even try
> > > to use it or make any decisions based on that.
> > > 
> > > 
> > > PS:
> > > on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
> > > guest would actually complicate QEMU for no big reason.
> > 
> > The guest is not misbehaving. It is following the spec.
> 
> (That's my thinking, and please feel free to correct me.)
> 
> I had the same thought. Windows guys could also say they didn't access
> the reserved MSR unconditionally, and they followed the CPUID feature
> bit to access that MSR. When CPUID is set, it indicates that feature is
> implemented.
> 
> At least I think it makes sense to rely on the CPUID to access the MSR.
> Just as an example, it's unlikely that after the software finds a CPUID
> of 1, it still need to download the latest spec version to confirm
> whether the feature is actually implemented or reserved.

If the encountered feature bit is indeed not expected (truly reserved),
the processor would be considered faulty and may be fixed in a new
stepping. This is similar to the debate over whether software should
adhere to the spec or whether hardware (emulation) should comply.

> Based on the above point, this CPUID feature bit is set to 1 in KVM and
> KVM also adds emulation (as a fix) specifically for this MSR. This means
> that Guest is considered to have valid access to this feature MSR,
> except that if Guest doesn't get what it wants, then it is reasonable
> for Guest to assume that the current (v)CPU lacks hardware support and
> mark it as "unsupported processor".
 

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Xiaoyao Li]

On 7/2/2025 1:01 PM, Zhao Liu wrote:
> Thanks Igor for looking here and thanks Konrad's explanation.
> 
>>>>> On 7/1/2025 6:26 PM, Zhao Liu wrote:
>>>>>>> unless it was explicitly requested by the user.
>>>>>> But this could still break Windows, just like issue #3001, which enables
>>>>>> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
>>>>>> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
>>>>>> value would even break something.
>>>>>>
>>>>>> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
>>>>>> that it is purely emulated, and is (maybe?) harmful.
>>>>>
>>>>> It is because Windows adds wrong code. So it breaks itself and it's just the
>>>>> regression of Windows.
>>>>
>>>> Could you please tell me what the Windows's wrong code is? And what's
>>>> wrong when someone is following the hardware spec?
>>>
>>> the reason is that it's reserved on AMD hence software shouldn't even try
>>> to use it or make any decisions based on that.
>>>
>>>
>>> PS:
>>> on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
>>> guest would actually complicate QEMU for no big reason.
>>
>> The guest is not misbehaving. It is following the spec.
> 
> (That's my thinking, and please feel free to correct me.)

I think we need firstly aligned on what the behavior of the Windows that 
hit "unsupported processor" is.

My understanding is, the Windows is doing something like

	if (is_AMD && CPUID(arch_capabilities))
		error(unsupported processor)

And I think this behavior is not correct.

However, it seems not the behavior of the Windows from your 
understanding. So what's the behavior in you mind?

> I had the same thought. Windows guys could also say they didn't access
> the reserved MSR unconditionally, and they followed the CPUID feature
> bit to access that MSR. When CPUID is set, it indicates that feature is
> implemented.
> 
> At least I think it makes sense to rely on the CPUID to access the MSR.
> Just as an example, it's unlikely that after the software finds a CPUID
> of 1, it still need to download the latest spec version to confirm
> whether the feature is actually implemented or reserved.
> 
> Based on the above point, this CPUID feature bit is set to 1 in KVM and
> KVM also adds emulation (as a fix) specifically for this MSR. This means
> that Guest is considered to have valid access to this feature MSR,
> except that if Guest doesn't get what it wants, then it is reasonable
> for Guest to assume that the current (v)CPU lacks hardware support and
> mark it as "unsupported processor".
> 
> As Konrad's mentioned, there's the previous explanation about why KVM
> sets this feature bit (it started with a little accident):
> 
> https://lore.kernel.org/kvm/CALMp9eRjDczhSirSismObZnzimxq4m+3s6Ka7OxwPj5Qj6X=BA@mail.gmail.com/#t
> 
> So I think the question is where this fix should be applied (KVM or
> QEMU) or if it should be applied at all, rather than whether Windows has
> the bug.

If we are agreed it's the bug of Windows, then no fix in QEMU/KVM at all.

> But I do agree, such "cleanups" would complicate QEMU, as I listed
> Eduardo as having done similar workaround six years ago:
> 
> https://lore.kernel.org/qemu-devel/20190125220606.4864-1-ehabkost@redhat.com/
> 
> Complexity and technical debt is an important consideration, and another
> consideration is the impact of this issue. Luckily, newer versions of
> Windows are actively compatible with KVM + QEMU:
> 
> https://blogs.windows.com/windows-insider/2025/06/23/announcing-windows-11-insider-preview-build-26120-4452-beta-channel/
> 
> But it's also hard to say if such a problem will happen again.
> Especially if the software works fine on real hardware but fails in
> "-host cpu" (which is supposed synchronized with host as much as
> possible).

work fine on real hardware but have issue with virtualization doesn't 
mean it is the problem of virtualization unless we figure out the 
root-cause and prove that software/OS's behavior is correct.

If the problem is due to the wrong behavior of guest OS, then it has 
nothing to do QEMU/KVM and QEMU/KVM cannot avoid such problem.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Zhao Liu]

> I think we need firstly aligned on what the behavior of the Windows that hit
> "unsupported processor" is.
> 
> My understanding is, the Windows is doing something like
> 
> 	if (is_AMD && CPUID(arch_capabilities))
> 		error(unsupported processor)

This is just a guess; it's also possible that Windows checked this MSR
and found the necessary feature missing. Windows 11 has very strict
hardware support requirements.

> And I think this behavior is not correct.
> 
> However, it seems not the behavior of the Windows from your understanding.
> So what's the behavior in you mind?

Guessing and discussing what Windows' code actually does is unlikely to
yield results. It's closed-source, and even if someone knows the answer,
he probably won't disclose it due to contractual restrictions.

Thanks,
Zhao

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Sean Christopherson]

On Wed, Jul 02, 2025, Zhao Liu wrote:
> > I think we need firstly aligned on what the behavior of the Windows that hit
> > "unsupported processor" is.
> > 
> > My understanding is, the Windows is doing something like
> > 
> > 	if (is_AMD && CPUID(arch_capabilities))
> > 		error(unsupported processor)
> 
> This is just a guess; it's also possible that Windows checked this MSR
> and found the necessary feature missing. Windows 11 has very strict
> hardware support requirements.
> 
> > And I think this behavior is not correct.

It's not really a matter of correct versus incorrect in this case.  Software is
well within its rights to refuse to run on unsupported hardware.  E.g. many
hypervisors now require EPT/NPT and other modern features.  Even KVM requires a
minimum set of features; KVM just happens to have a *very* low minimum, and KVM
tries to be gentle and friendly when unsupported or incompatible hardware is
encountered.

Windows' behavior is arguably flawed, misguided, and user hostile, but we can't
say it's wrong.  E.g. even if the argument is that AMD could ship a future CPU
that supports ARCH_CAPABILITIES, it's not a violation of AMD's architecture for
Windows to not support such a processor.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Alexandre Chartre]

On 7/2/25 07:01, Zhao Liu wrote:
> Thanks Igor for looking here and thanks Konrad's explanation.
> 
>>>>> On 7/1/2025 6:26 PM, Zhao Liu wrote:
>>>>>>> unless it was explicitly requested by the user.
>>>>>> But this could still break Windows, just like issue #3001, which enables
>>>>>> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
>>>>>> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
>>>>>> value would even break something.
>>>>>>
>>>>>> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
>>>>>> that it is purely emulated, and is (maybe?) harmful.
>>>>>
>>>>> It is because Windows adds wrong code. So it breaks itself and it's just the
>>>>> regression of Windows.
>>>>
>>>> Could you please tell me what the Windows's wrong code is? And what's
>>>> wrong when someone is following the hardware spec?
>>>
>>> the reason is that it's reserved on AMD hence software shouldn't even try
>>> to use it or make any decisions based on that.
>>>
>>>
>>> PS:
>>> on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
>>> guest would actually complicate QEMU for no big reason.
>>
>> The guest is not misbehaving. It is following the spec.
> 
> (That's my thinking, and please feel free to correct me.)
> 
> I had the same thought. Windows guys could also say they didn't access
> the reserved MSR unconditionally, and they followed the CPUID feature
> bit to access that MSR. When CPUID is set, it indicates that feature is
> implemented.
> 
> At least I think it makes sense to rely on the CPUID to access the MSR.
> Just as an example, it's unlikely that after the software finds a CPUID
> of 1, it still need to download the latest spec version to confirm
> whether the feature is actually implemented or reserved.
> 
> Based on the above point, this CPUID feature bit is set to 1 in KVM and
> KVM also adds emulation (as a fix) specifically for this MSR. This means
> that Guest is considered to have valid access to this feature MSR,
> except that if Guest doesn't get what it wants, then it is reasonable
> for Guest to assume that the current (v)CPU lacks hardware support and
> mark it as "unsupported processor".
> 
> As Konrad's mentioned, there's the previous explanation about why KVM
> sets this feature bit (it started with a little accident):
> 
> https://lore.kernel.org/kvm/CALMp9eRjDczhSirSismObZnzimxq4m+3s6Ka7OxwPj5Qj6X=BA@mail.gmail.com/#t
> 
> So I think the question is where this fix should be applied (KVM or
> QEMU) or if it should be applied at all, rather than whether Windows has
> the bug.
> 
> But I do agree, such "cleanups" would complicate QEMU, as I listed
> Eduardo as having done similar workaround six years ago:
> 
> https://lore.kernel.org/qemu-devel/20190125220606.4864-1-ehabkost@redhat.com/
> 
> Complexity and technical debt is an important consideration, and another
> consideration is the impact of this issue. Luckily, newer versions of
> Windows are actively compatible with KVM + QEMU:
> 
> https://blogs.windows.com/windows-insider/2025/06/23/announcing-windows-11-insider-preview-build-26120-4452-beta-channel/
> 
> But it's also hard to say if such a problem will happen again.
> Especially if the software works fine on real hardware but fails in
> "-host cpu" (which is supposed synchronized with host as much as
> possible).
> 
>>> Also
>>> KVM does do have plenty of such code, and it's not actively preventing guests from using it.
>>> Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.
>>
>> Because KVM maintainer does not want to touch the guest ABI. He agrees
>> this is a bug.
> 
> If we agree on this fix should be applied on Linux virtualization stack,
> then the question of whether the fix should land in KVM or QEMU is a bit
> like the chicken and egg dilemma.
> 
> I personally think it might be better to roll it out in QEMU first — it
> feels like the safer bet:
> 
>   * Currently, the -cpu host option enables this feature by default, and
>     it's hard to say if anyone is actually relying on this emulated
>     feature (though issue #3001 suggests it causes trouble for Windows).
>     So only when the ABI changes, it's uncertain if anything will break.

I wouldn't expect that any code rely on the presence of this feature (emulated
or or) because it is not available on some cpus (AMD but also some Intel cpus),
so the code should be able to handle the absence of this feature.

Also KVM emulates this MSR only to advertise that the cpu is not impacted by
some speculative execution vulnerabilities, none of which (except one) applies
to AMD cpus. So on AMD, the MSR always says: this cpu is not impacted by
all these vulnerabilities; but you can already figure that because this is an
AMD cpu. The only vulnerability in this MSR that can be present on AMD is SSB,
but it also has an AMD-specific feature to indicate if the cpu is not impacted
(and it is set accordingly by KVM).

Anyway, changing QEMU first is certainly safer before eventually changing KVM.

alex.


>   * Similarly, if only the ABI is changed, I'm a bit unsure if there's
>     any migration based on "-cpu host" and between different versions of
>     kernel. And, based on my analysis at the beginning reply, named CPUs
>     also have the effect if user actively sets "arch-capbilities=on". But
>     the key here point is the migration happens between different kernel
>     versions.
> 
>   * Additionally, handling different versions of ABI can sometimes be
>     quite complex. After changing the ABI, there might be differences
>     between the new kernel and the old stable kernels (and according to
>     doc, the oldest supported kernel is v4.5 - docs/system/target-i386.rst).
>     It's similar to what Ewan previously complained about:
> 
>     https://lore.kernel.org/qemu-devel/53119b66-3528-41d6-ac44-df166699500a@zhaoxin.com/
> 
> So, if anyone is relying on the current emulated feature, changing the
> ABI will inevitably break existing things, and QEMU might have to bear
> the cost of maintaining compatibility with the old ABI. :-(
> 
> Personally, I think the safer approach is to first handle potential old
> dependencies in QEMU through a compat option. Once the use case is
> eliminated in user space, it can clearly demonstrate that the ABI change
> won't disrupt user space.
> 
> The workaround change I proposed to Alexandre isn't meant to be
> permanent. If we upgrade the supported kernel version to >6.17 (assuming
> the ABI can be changed in 6.17), then the workaround can be removed —
> though I admit that day might never come...
> 
> Thanks for your patience.
> Zhao
> 

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Igor Mammedov]

On Tue, 1 Jul 2025 16:01:21 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Tue, Jul 01, 2025 at 03:05:00PM +0200, Igor Mammedov wrote:
> > On Tue, 1 Jul 2025 20:36:43 +0800
> > Zhao Liu <zhao1.liu@intel.com> wrote:
> >   
> > > On Tue, Jul 01, 2025 at 07:12:44PM +0800, Xiaoyao Li wrote:  
> > > > Date: Tue, 1 Jul 2025 19:12:44 +0800
> > > > From: Xiaoyao Li <xiaoyao.li@intel.com>
> > > > Subject: Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised
> > > >  on AMD
> > > > 
> > > > On 7/1/2025 6:26 PM, Zhao Liu wrote:    
> > > > > > unless it was explicitly requested by the user.    
> > > > > But this could still break Windows, just like issue #3001, which enables
> > > > > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > > > > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > > > > value would even break something.
> > > > > 
> > > > > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > > > > that it is purely emulated, and is (maybe?) harmful.    
> > > > 
> > > > It is because Windows adds wrong code. So it breaks itself and it's just the
> > > > regression of Windows.    
> > > 
> > > Could you please tell me what the Windows's wrong code is? And what's
> > > wrong when someone is following the hardware spec?  
> > 
> > the reason is that it's reserved on AMD hence software shouldn't even try
> > to use it or make any decisions based on that.
> > 
> > 
> > PS:
> > on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
> > guest would actually complicate QEMU for no big reason.  
> 
> The guest is not misbehaving. It is following the spec.

that's not how I read spec:

"
AMD64 Architecture Programmer’s Manual Volume 3: General-Purpose and System Instructions
24594—Rev. 3.36—March 2024
...
Appendix E Obtaining Processor Information Via the CPUID Instruction
...
All bit positions that are not defined as fields are
reserved. The value of bits within reserved ranges cannot be relied upon to be zero.
Software must mask off all reserved bits in the return value prior to making any value comparisons of represented
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
information.
...
E.3.6 Function 7h—Structured Extended Feature Identifiers
...
The value returned in EDX is undefined and is reserved.
"

what actually happens is guest side being lazy and blindly following CPUID.


> > Also
> > KVM does do have plenty of such code, and it's not actively preventing guests from using it.
> > Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.  
> 
> Because KVM maintainer does not want to touch the guest ABI. He agrees
> this is a bug.
one can say both guest and hypervisor are to blame,
  1st is not masking reserved bits
  2nd provides 'hybrid' cpu that doesn't exists in real world,
  but then 'host' cpu model has never been the exact match for physical cpu.

what I dislike is ad-hoc fixups in generic code, 
if consensus were to implement _out of spec_ fixup for already fixed issue in Windows,
it should be better be done in host cpumodel code.

Or even better a single KVM optin feature 'do_not_advertise_features_not_supported_by_host_cpu',
and then QEMU could use that for disabling all nonsense in one go.
Plus all of that won't be breaking KVM ABI nor qemu had to add fixups for this and that feature.

After some time when old machine types are deprecated/gone, KVM could make it default and eventually
remove advertising 'fake' features.

PS:
On QEMU side we usually tolerant to such fixups if it's not fixable on guest side.
but that's not the case here.

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Sean Christopherson]

On Wed, Jul 02, 2025, Igor Mammedov wrote:
> Or even better a single KVM optin feature 'do_not_advertise_features_not_supported_by_host_cpu',
> and then QEMU could use that for disabling all nonsense in one go.
> Plus all of that won't be breaking KVM ABI nor qemu had to add fixups for this and that feature.
> 
> After some time when old machine types are deprecated/gone, KVM could make it default and eventually
> remove advertising 'fake' features.

Such a feature/quirk wouldn't be useful in practice.  There are several features
that KVM emulates irrespective of hardware support, and that generally speaking
every VMM will want to enable whenever possible, e.g. x2APIC, TSC deadline timer,
TSC adjust, and the amusing "no SMM_CTL MSR" anti-feature.  Throwing out x2APIC
and TSC deadline timer in particular would be a significant regression, i.e. not
something any end user actually wants.

If QEMU or any other VMM wants to filter KVM's support against bare metal, then
QEMU can simply do CPUID itself.

Somewhat of a side topic, the somewhat confusingly-named KVM_GET_EMULATED_CPUID
exists to allow KVM to differentiate between features that KVM can emulate and/or
virtualize without additional overhead, and those that KVM can emulate but with
non-trivial cost.  E.g. KVM advertises MOVBE and RDPID in KVM_GET_SUPPORTED_CPUID
if and only if they are natively suppored, because enabling those instructions in
the guest's CPUID model turns exitless instruction execution into expensive #UD
VM-Exits and slow emulation.  But KVM unconditionally advertises support for them
in KVM_GET_EMULATED_CPUID so that userspace can run a "newer" VM on old hardware.

I mention KVM_GET_EMULATED_CPUID purely to stave off any exploration into trying
to move ARCH_CAPABILITIES to KVM_GET_EMULATED_CPUID.  Name aside, it's not the
correct fit (and would still be an ABI change).

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Sean Christopherson]

On Tue, Jul 01, 2025, Konrad Rzeszutek Wilk wrote:
> On Tue, Jul 01, 2025 at 03:05:00PM +0200, Igor Mammedov wrote:
> > On Tue, 1 Jul 2025 20:36:43 +0800
> > Zhao Liu <zhao1.liu@intel.com> wrote:
> > 
> > > On Tue, Jul 01, 2025 at 07:12:44PM +0800, Xiaoyao Li wrote:
> > > > Date: Tue, 1 Jul 2025 19:12:44 +0800
> > > > From: Xiaoyao Li <xiaoyao.li@intel.com>
> > > > Subject: Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised
> > > >  on AMD
> > > > 
> > > > On 7/1/2025 6:26 PM, Zhao Liu wrote:  
> > > > > > unless it was explicitly requested by the user.  
> > > > > But this could still break Windows, just like issue #3001, which enables
> > > > > arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> > > > > turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> > > > > value would even break something.
> > > > > 
> > > > > So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> > > > > that it is purely emulated, and is (maybe?) harmful.  
> > > > 
> > > > It is because Windows adds wrong code. So it breaks itself and it's just the
> > > > regression of Windows.  
> > > 
> > > Could you please tell me what the Windows's wrong code is? And what's
> > > wrong when someone is following the hardware spec?
> > 
> > the reason is that it's reserved on AMD hence software shouldn't even try
> > to use it or make any decisions based on that.
> > 
> > 
> > PS:
> > on contrary, doing such ad-hoc 'cleanups' for the sake of misbehaving
> > guest would actually complicate QEMU for no big reason.
> 
> The guest is not misbehaving. It is following the spec.
> > 
> > Also
> > KVM does do have plenty of such code, and it's not actively preventing guests from using it.
> > Given that KVM is not welcoming such change, I think QEMU shouldn't do that either.
> 
> Because KVM maintainer does not want to touch the guest ABI. He agrees
> this is a bug.

No, I agreed that KVM's behavior is pointless, annoying, and odd[*].  I do not
agree that KVM's behavior is an outright bug.

[*] https://lore.kernel.org/all/aF1S2EIJWN47zLDG@google.com

Re: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD

[Alexandre Chartre]

On 7/1/25 12:26, Zhao Liu wrote:
> (I'd like to CC Sean again to discuss the possibility of user space
>   removing arch-capabilities completely for AMD)
> 
> On Mon, Jun 30, 2025 at 03:30:25PM +0200, Alexandre Chartre wrote:
>> Date: Mon, 30 Jun 2025 15:30:25 +0200
>> From: Alexandre Chartre <alexandre.chartre@oracle.com>
>> Subject: [PATCH] i386/cpu: ARCH_CAPABILITIES should not be advertised on AMD
>> X-Mailer: git-send-email 2.43.5
>>
>> KVM emulates the ARCH_CAPABILITIES on x86 for both Intel and AMD
>> cpus, although the IA32_ARCH_CAPABILITIES MSR is an Intel-specific
>> MSR and it makes no sense to emulate it on AMD.
>>
>> As a consequence, VMs created on AMD with qemu -cpu host and using
>> KVM will advertise the ARCH_CAPABILITIES feature and provide the
>> IA32_ARCH_CAPABILITIES MSR. This can cause issues (like Windows BSOD)
>> as the guest OS might not expect this MSR to exist on such cpus (the
>> AMD documentation specifies that ARCH_CAPABILITIES feature and MSR
>> are not defined on the AMD architecture).
> 
> This issue looks very similar to this one that others in the community
> reported:
> 
> https://urldefense.com/v3/__https://gitlab.com/qemu-project/qemu/-/issues/3001__;!!ACWV5N9M2RV99hQ!IniD7c-8rcBUxEfJSXIBw2nLjN3la2lNKdPCWBhis7bs4j7k5tCISUMRRt7RrJjeONhumXlVH9x-wzPJSvDpq5s$
> 
> But there's a little difference, pls see the below comment...
> 
>> A fix was proposed in KVM code, however KVM maintainers don't want to
>> change this behavior that exists for 6+ years and suggest changes to be
>> done in qemu instead.
>>
>> So this commit changes the behavior in qemu so that ARCH_CAPABILITIES
>> is not provided by default on AMD cpus when the hypervisor emulates it,
>> but it can still be provided by explicitly setting arch-capabilities=on.
>> Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
>> ---
>>   target/i386/cpu.c | 14 ++++++++++++++
>>   1 file changed, 14 insertions(+)
>>
>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>> index 0d35e95430..7e136c48df 100644
>> --- a/target/i386/cpu.c
>> +++ b/target/i386/cpu.c
>> @@ -8324,6 +8324,20 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>>           }
>>       }
>>   
>> +    /*
>> +     * For years, KVM has inadvertently emulated the ARCH_CAPABILITIES
>> +     * MSR on AMD although this is an Intel-specific MSR; and KVM will
>> +     * continue doing so to not change its ABI for existing setups.
>> +     *
>> +     * So ensure that the ARCH_CAPABILITIES MSR is disabled on AMD cpus
>> +     * to prevent providing a cpu with an MSR which is not supposed to
>> +     * be there,
> 
> Yes, disabling this feature bit makes sense on AMD platform. It's fine
> for -cpu host.
> 
>> unless it was explicitly requested by the user.
> 
> But this could still break Windows, just like issue #3001, which enables
> arch-capabilities for EPYC-Genoa. This fact shows that even explicitly
> turning on arch-capabilities in AMD Guest and utilizing KVM's emulated
> value would even break something.
> 
> So even for named CPUs, arch-capabilities=on doesn't reflect the fact
> that it is purely emulated, and is (maybe?) harmful.
> 
>> +     */
>> +    if (IS_AMD_CPU(env) &&
>> +        !(env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
>> +        env->features[FEAT_7_0_EDX] &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
>> +    }
>> +
> 
> I was considering whether we should tweak it in kvm_arch_get_supported_cpuid()
> until I saw this:
> 
> else if (function == 7 && index == 0 && reg == R_EDX) {
>          /* Not new instructions, just an optimization.  */
>          uint32_t edx;
>          host_cpuid(7, 0, &unused, &unused, &unused, &edx);
>          ret |= edx & CPUID_7_0_EDX_FSRM;
> 
>          /*
>           * Linux v4.17-v4.20 incorrectly return ARCH_CAPABILITIES on SVM hosts.
>           * We can detect the bug by checking if MSR_IA32_ARCH_CAPABILITIES is
>           * returned by KVM_GET_MSR_INDEX_LIST.
>           */
>          if (!has_msr_arch_capabs) {
>              ret &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
>          }
> }
> 
> What a pity! QEMU had previously workedaround CPUID_7_0_EDX_ARCH_CAPABILITIES
> correctly, but since then kvm's commit 0cf9135b773b("KVM: x86: Emulate
> MSR_IA32_ARCH_CAPABILITIES on AMD hosts") breaks the balance once again.
> I understand the commit, and it makes up for the mismatch between the
> emulated feature bit and the MSR. Now the Windows exposes the problem of
> such emulation.
> 
> So, to avoid endless workaround thereafter, I think it's time to just
> disable arch-capabilities for AMD Guest (after all, closer to the real
> hardware environment is better).
> 
> Further, it helps to eliminate kernel/kvm concerns when user space resolves
> the legacy issues first. At least, IMO, pushing ABI changes in kernel/kvm
> needs to show that there is no destruction of pre-existing user space, so
> I believe a complete cleanup of QEMU is the appropriate approach.
> 
> The attached code is just some simple example to show what I think:
> Starting with QEMU v10.1 for AMD Guest, to disable arch-capabilties
> feature bit and MSR.
> 
> I don't have an AMD CPU, so it's untested. You can feel free to squash
> it in your patch. If so, it's better to add a "Resolves" tag in your
> commit message:
> 

Thanks for the fix update, I will give it a try and submit a new patch version.

alex.

> Resolves: https://urldefense.com/v3/__https://gitlab.com/qemu-project/qemu/-/issues/3001__;!!ACWV5N9M2RV99hQ!IniD7c-8rcBUxEfJSXIBw2nLjN3la2lNKdPCWBhis7bs4j7k5tCISUMRRt7RrJjeONhumXlVH9x-wzPJSvDpq5s$
> 
> Thanks,
> Zhao
> ---
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index b2116335752d..c175e7d9e7b8 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -81,7 +81,9 @@
>       { "qemu64-" TYPE_X86_CPU, "model-id", "QEMU Virtual CPU version " v, },\
>       { "athlon-" TYPE_X86_CPU, "model-id", "QEMU Virtual CPU version " v, },
> 
> -GlobalProperty pc_compat_10_0[] = {};
> +GlobalProperty pc_compat_10_0[] = {
> +    { TYPE_X86_CPU, "x-amd-disable-arch-capabs", "false" },
> +};
>   const size_t pc_compat_10_0_len = G_N_ELEMENTS(pc_compat_10_0);
> 
>   GlobalProperty pc_compat_9_2[] = {};
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 9aa0ea447860..a8e83efd83f6 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -8336,10 +8336,12 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>        *
>        * So ensure that the ARCH_CAPABILITIES MSR is disabled on AMD cpus
>        * to prevent providing a cpu with an MSR which is not supposed to
> -     * be there, unless it was explicitly requested by the user.
> +     * be there.
>        */
> -    if (IS_AMD_CPU(env) &&
> -        !(env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
> +    if (cpu->amd_disable_arch_capabs && IS_AMD_CPU(env)) {
> +        mark_unavailable_features(cpu, FEAT_7_0_EDX,
> +            env->user_features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES,
> +            "This feature is not available for AMD Guest");
>           env->features[FEAT_7_0_EDX] &= ~CPUID_7_0_EDX_ARCH_CAPABILITIES;
>       }
> 
> @@ -9414,6 +9416,8 @@ static const Property x86_cpu_properties[] = {
>       DEFINE_PROP_BOOL("x-intel-pt-auto-level", X86CPU, intel_pt_auto_level,
>                        true),
>       DEFINE_PROP_BOOL("x-l1-cache-per-thread", X86CPU, l1_cache_per_core, true),
> +    DEFINE_PROP_BOOL("x-amd-disable-arch-capabs", X86CPU, amd_disable_arch_capabs,
> +                     true),
>   };
> 
>   #ifndef CONFIG_USER_ONLY
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index 51e10139dfdf..a3fc80de3a75 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -2306,6 +2306,13 @@ struct ArchCPU {
>        */
>       uint32_t guest_phys_bits;
> 
> +    /*
> +     * Compatibility bits for old machine types.
> +     * If true disable CPUID_7_0_EDX_ARCH_CAPABILITIES and
> +     * MSR_IA32_ARCH_CAPABILITIES for AMD Guest.
> +     */
> +    bool amd_disable_arch_capabs;
> +
>       /* in order to simplify APIC support, we leave this pointer to the
>          user */
>       struct DeviceState *apic_state;
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 234878c613f6..40a50ae193c7 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -2368,6 +2368,11 @@ int kvm_arch_init_vcpu(CPUState *cs)
> 
>       cpu->kvm_msr_buf = g_malloc0(MSR_BUF_SIZE);
> 
> +    if (cpu->amd_disable_arch_capabs &&
> +        !(env->features[FEAT_7_0_EDX] & CPUID_7_0_EDX_ARCH_CAPABILITIES)) {
> +        has_msr_arch_capabs = false;
> +    }
> +
>       if (!(env->features[FEAT_8000_0001_EDX] & CPUID_EXT2_RDTSCP)) {
>           has_msr_tsc_aux = false;
>       }
> 
> 
> 
>