* Re: [PATCH v7 16/22] x86/virt/tdx: Update tdx_sysinfo and check features post-update
From: Dave Hansen @ 2026-04-07 15:53 UTC (permalink / raw)
To: Chao Gao, linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <adT1Tkz+/ysSZ1Ua@intel.com>
On 4/7/26 05:15, Chao Gao wrote:
> Dave's comment on another patch applies here too: don't preemptively handle
> errors that never occur. The custom error message is unnecessary, and
> propagating the error isn't worth it. Will simplify it to:
>
> /* Shouldn't fail as the update has succeeded. */
> WARN_ON_ONCE(get_tdx_sys_info(info));
This is nit territory, but I don't like that either.
Actual, important, normal-program-flow logic should stand on its own,
separate from warnings.
OK:
ret = foo()
WARN_ON(ret);
Not OK:
WARN_ON(foo());
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Sean Christopherson @ 2026-04-07 13:20 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Chang S. Bae, Kiryl Shutsemau, kvm, the arch/x86 maintainers,
linux-coco, Kernel Mailing List, Linux, Andrew Cooper
In-Reply-To: <CABgObfaFqrSENS=_eNgkyxebqL1vFauNqG3XAgZm0EHfkbQ_gw@mail.gmail.com>
On Tue, Apr 07, 2026, Paolo Bonzini wrote:
> Il mar 7 apr 2026, 00:00 Sean Christopherson <seanjc@google.com> ha scritto:
> >
> > > > . So unless I'm missing something (or hardware is flawed and lets the
> > > > guest speculative consume R16-R31, which would be sad), it's perfectly safe to
> > > > run the guest with host state in R16-R31.
> > > >
> > > > That would avoid pointlessly context switching 16 registers when APX is not being
> > > > used by the guest, and would avoid having to write XCR0 in the fastpath.
> > >
> > > For now yes, but once/if the kernel starts using the registers there's
> > > no way out of writing XCR0 for APX-disabled guests in the fast path.
> >
> > Why's that? So long as KVM uses vcpu->arch.regs[R16-R31] as the source of truth
> > when emulating anything, there's no danger of taking a #UD in the host due to
> > accessing R16-R31 with XCR0.APX=0.
>
> Yes I agree with that. But the unavoidable part is the XSETBV because
> only the assembly code can run with XCR0.APX=0. As soon as you go back
> to C, including during the fast path, you have to ensure XCR0.APX=1
> again if the kernel is compiled with -mapxf.
/facepalm
I got so focused on register state that I completely forgot about actually
using the registers...
^ permalink raw reply
* Re: [PATCH v7 16/22] x86/virt/tdx: Update tdx_sysinfo and check features post-update
From: Chao Gao @ 2026-04-07 12:15 UTC (permalink / raw)
To: linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260331124214.117808-17-chao.gao@intel.com>
>+int tdx_module_post_update(struct tdx_sys_info *info)
>+{
>+ struct tdx_sys_info_version *old, *new;
>+ int ret;
>+
>+ /* Shouldn't fail as the update has succeeded. */
>+ ret = get_tdx_sys_info(info);
>+ if (WARN_ONCE(ret, "version retrieval failed after update, replace the TDX module\n"))
>+ return ret;
Dave's comment on another patch applies here too: don't preemptively handle
errors that never occur. The custom error message is unnecessary, and
propagating the error isn't worth it. Will simplify it to:
/* Shouldn't fail as the update has succeeded. */
WARN_ON_ONCE(get_tdx_sys_info(info));
>+
>+ old = &tdx_sysinfo.version;
>+ new = &info->version;
>+ pr_info("version " TDX_VERSION_FMT " -> " TDX_VERSION_FMT "\n",
>+ old->major_version, old->minor_version, old->update_version,
>+ new->major_version, new->minor_version, new->update_version);
>+
>+ /*
>+ * Blindly refreshing the entire tdx_sysinfo could disrupt running
>+ * software, as it may subtly rely on the previous state unless
>+ * proven otherwise.
>+ *
>+ * Only refresh update_version and handoff version. They don't
>+ * affect TDX functionality. Major/minor versions do not change
>+ * across updates, so no refresh is needed.
>+ */
>+ tdx_sysinfo.version.update_version = info->version.update_version;
>+ tdx_sysinfo.handoff = info->handoff;
>+
>+ if (!memcmp(&tdx_sysinfo, info, sizeof(*info)))
>+ return 0;
>+
>+ pr_info("TDX module features have changed after updates, but might not take effect.\n");
>+ pr_info("Please consider updating your BIOS to install the TDX module.\n");
>+ return 0;
>+}
^ permalink raw reply
* Re: [PATCH v7 15/22] x86/virt/tdx: Restore TDX module state
From: Chao Gao @ 2026-04-07 12:07 UTC (permalink / raw)
To: linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260331124214.117808-16-chao.gao@intel.com>
>+int tdx_module_run_update(void)
>+{
>+ struct tdx_module_args args = {};
>+ int ret;
>+
>+ ret = seamcall_prerr(TDH_SYS_UPDATE, &args);
>+ if (ret) {
>+ pr_err("update failed (%d)\n", ret);
>+ tdx_module_status = TDX_MODULE_ERROR;
>+ return ret;
>+ }
The pr_err() isn't needed as seamcall_prerr() will emit a
message. and no need to set tdx_module_status to ERROR on
failure as it is already done during shutdown.
so, this can be simplified to:
ret = seamcall_prerr(TDH_SYS_UPDATE, &args);
if (ret)
return ret;
^ permalink raw reply
* Re: [PATCH v7 12/22] x86/virt/tdx: Reset software states during TDX module shutdown
From: Chao Gao @ 2026-04-07 12:02 UTC (permalink / raw)
To: linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260331124214.117808-13-chao.gao@intel.com>
> int tdx_module_shutdown(void)
> {
> struct tdx_module_args args = {};
>+ int ret, cpu;
>
> /*
> * Shut down the TDX module and prepare handoff data for the next
>@@ -1188,7 +1189,31 @@ int tdx_module_shutdown(void)
> * modules as new modules likely have higher handoff version.
> */
> args.rcx = tdx_sysinfo.handoff.module_hv;
>- return seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
>+ ret = seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
>+ if (ret)
>+ return ret;
>+
>+ /*
>+ * Mark the module is unavailable (in ERROR status) to prevent
>+ * re-initialization and tdx_sysinfo reporting. Note the status
>+ * will be restored after a successful update.
>+ *
>+ * No need to acquire tdx_module_lock here since this runs in
>+ * stop_machine() where no concurrent initialization can occur.
>+ */
>+ tdx_module_status = TDX_MODULE_ERROR;
>+ sysinit_done = false;
>+ sysinit_ret = 0;
>+
>+ /*
>+ * Since the TDX module is shut down and gone, mark all CPUs
>+ * (including offlined ones) as uninitialized. This is called in
>+ * stop_machine() (where CPU hotplug is disabled), preventing
>+ * races with other tdx_lp_initialized accesses.
>+ */
>+ for_each_possible_cpu(cpu)
>+ per_cpu(tdx_lp_initialized, cpu) = false;
I would like to merge the two comments and make them more concise:
/*
* Clear global and per-CPU initialization flags so the new module
* can be fully re-initialized after a successful update. The ERROR
* status prevents re-init if the update ultimately fails.
*
* No locks needed as no concurrent accesses can occur here.
*/
tdx_module_status = TDX_MODULE_ERROR;
sysinit_done = false;
sysinit_ret = 0;
for_each_possible_cpu(cpu)
per_cpu(tdx_lp_initialized, cpu) = false;
>+ return 0;
> }
>
> static bool is_pamt_page(unsigned long phys)
>--
>2.47.3
>
^ permalink raw reply
* Re: [PATCH v7 11/22] x86/virt/seamldr: Shut down the current TDX module
From: Chao Gao @ 2026-04-07 11:51 UTC (permalink / raw)
To: linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260331124214.117808-12-chao.gao@intel.com>
>+int tdx_module_shutdown(void)
>+{
>+ struct tdx_module_args args = {};
>+
>+ /*
>+ * Shut down the TDX module and prepare handoff data for the next
>+ * TDX module. This SEAMCALL requires a handoff version. Use the
>+ * module's handoff version, as it is the highest version the
>+ * module can produce and is more likely to be supported by new
>+ * modules as new modules likely have higher handoff version.
>+ */
Will change this comment to:
/*
* Use the module's handoff version as it is the highest the
* module can produce and most likely supported by newer modules.
*/
>+ args.rcx = tdx_sysinfo.handoff.module_hv;
>+ return seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
>+}
^ permalink raw reply
* Re: [PATCH v7 09/22] x86/virt/seamldr: Introduce skeleton for TDX module updates
From: Chao Gao @ 2026-04-07 11:49 UTC (permalink / raw)
To: linux-kernel, linux-coco, kvm
Cc: binbin.wu, dan.j.williams, dave.hansen, ira.weiny, kai.huang, kas,
nik.borisov, paulmck, pbonzini, reinette.chatre, rick.p.edgecombe,
sagis, seanjc, tony.lindgren, vannapurve, vishal.l.verma,
yilun.xu, xiaoyao.li, yan.y.zhao, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260331124214.117808-10-chao.gao@intel.com>
>@@ -214,7 +287,14 @@ int seamldr_install_module(const u8 *data, u32 size)
> if (IS_ERR(params))
> return PTR_ERR(params);
>
>- /* TODO: Update TDX module here */
>- return 0;
>+ /*
>+ * Prevent CPU hotplug. If a CPU goes offline after thread_ack
>+ * initialization, thread_ack will exceed the online count and
>+ * never decrement to zero, causing all CPUs spinning forever
>+ * with IRQs disabled.
>+ */
Applying Dave's feedback to simplify comments across the series. I will change
this to:
/* The lockstep update needs a stable set of online CPUs. */
>+ guard(cpus_read_lock)();
>+ set_target_state(MODULE_UPDATE_START + 1);
>+ return stop_machine_cpuslocked(do_seamldr_install_module, params, cpu_online_mask);
> }
> EXPORT_SYMBOL_FOR_MODULES(seamldr_install_module, "tdx-host");
>--
>2.47.3
>
^ permalink raw reply
* [PATCH v2] dma-buf: heaps: system: document system_cc_shared heap
From: Jiri Pirko @ 2026-04-07 9:26 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco
From: Jiri Pirko <jiri@nvidia.com>
Document the system_cc_shared dma-buf heap that was introduced
recently. Describe its purpose, availability conditions and
relation to confidential computing VMs.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: T.J.Mercier <tjmercier@google.com>
---
Documentation/userspace-api/dma-buf-heaps.rst | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Documentation/userspace-api/dma-buf-heaps.rst b/Documentation/userspace-api/dma-buf-heaps.rst
index 05445c83b79a..f56b743cdb36 100644
--- a/Documentation/userspace-api/dma-buf-heaps.rst
+++ b/Documentation/userspace-api/dma-buf-heaps.rst
@@ -16,6 +16,13 @@ following heaps:
- The ``system`` heap allocates virtually contiguous, cacheable, buffers.
+ - The ``system_cc_shared`` heap allocates virtually contiguous, cacheable,
+ buffers using shared (decrypted) memory. It is only present on
+ confidential computing (CoCo) VMs where memory encryption is active
+ (e.g., AMD SEV, Intel TDX). The allocated pages have the encryption
+ bit cleared, making them accessible for device DMA without TDISP
+ support. On non-CoCo VM configurations, this heap is not registered.
+
- The ``default_cma_region`` heap allocates physically contiguous,
cacheable, buffers. Only present if a CMA region is present. Such a
region is usually created either through the kernel commandline
--
2.51.1
^ permalink raw reply related
* Re: [PATCH] dma-buf: heaps: system: document system_cc_shared heap
From: Jiri Pirko @ 2026-04-07 9:25 UTC (permalink / raw)
To: T.J. Mercier
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, christian.koenig,
m.szyprowski, robin.murphy, jgg, leon, sean.anderson, ptesarik,
catalin.marinas, aneesh.kumar, suzuki.poulose, steven.price,
thomas.lendacky, john.allen, ashish.kalra, suravee.suthikulpanit,
linux-coco
In-Reply-To: <CABdmKX3N70j8ZZs5DNhx6fhRi=Aa_+2xY1JHcW+uDoaV2+Sngw@mail.gmail.com>
Mon, Apr 06, 2026 at 10:20:33PM +0200, tjmercier@google.com wrote:
>On Thu, Apr 2, 2026 at 7:11 AM Jiri Pirko <jiri@resnulli.us> wrote:
>>
>> From: Jiri Pirko <jiri@nvidia.com>
>>
>> Document the system_cc_shared dma-buf heap that was introduced
>> recently. Describe its purpose, availability conditions and
>> relation to confidential computing VMs.
>>
>> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
>> ---
>> Documentation/userspace-api/dma-buf-heaps.rst | 8 ++++++++
>> 1 file changed, 8 insertions(+)
>>
>> diff --git a/Documentation/userspace-api/dma-buf-heaps.rst b/Documentation/userspace-api/dma-buf-heaps.rst
>> index 05445c83b79a..591732393e7d 100644
>> --- a/Documentation/userspace-api/dma-buf-heaps.rst
>> +++ b/Documentation/userspace-api/dma-buf-heaps.rst
>> @@ -16,6 +16,14 @@ following heaps:
>>
>> - The ``system`` heap allocates virtually contiguous, cacheable, buffers.
>>
>> + - The ``system_cc_shared`` heap allocates virtually contiguous, cacheable,
>> + buffers using shared (decrypted) memory. It is only present on
>> + confidential computing (CoCo) VMs where memory encryption is active
>> + (e.g., AMD SEV, Intel TDX). The allocated pages have the encryption
>> + bit cleared, making them accessible for device DMA without TDISP
>> + support. On non-CoCo VMs configurations, this heap is
>
>"non-CoCo VM configurations"
>
>> + not registered.
>
>Doesn't seem like you need to wrap this line.
>
>with that: Reviewed-by: T.J.Mercier <tjmercier@google.com>
Okay. Thanks!
>
>> +
>> - The ``default_cma_region`` heap allocates physically contiguous,
>> cacheable, buffers. Only present if a CMA region is present. Such a
>> region is usually created either through the kernel commandline
>
>Each paragraph starting with '-' confused me for a second there. Those
>aren't part of the diff. :)
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Paolo Bonzini @ 2026-04-07 7:18 UTC (permalink / raw)
To: Sean Christopherson
Cc: Chang S. Bae, Kiryl Shutsemau, kvm, the arch/x86 maintainers,
linux-coco, Kernel Mailing List, Linux, Andrew Cooper
In-Reply-To: <adQs4LQgy3mS2t89@google.com>
Il mar 7 apr 2026, 00:00 Sean Christopherson <seanjc@google.com> ha scritto:
>
> > > . So unless I'm missing something (or hardware is flawed and lets the
> > > guest speculative consume R16-R31, which would be sad), it's perfectly safe to
> > > run the guest with host state in R16-R31.
> > >
> > > That would avoid pointlessly context switching 16 registers when APX is not being
> > > used by the guest, and would avoid having to write XCR0 in the fastpath.
> >
> > For now yes, but once/if the kernel starts using the registers there's
> > no way out of writing XCR0 for APX-disabled guests in the fast path.
>
> Why's that? So long as KVM uses vcpu->arch.regs[R16-R31] as the source of truth
> when emulating anything, there's no danger of taking a #UD in the host due to
> accessing R16-R31 with XCR0.APX=0.
Yes I agree with that. But the unavoidable part is the XSETBV because
only the assembly code can run with XCR0.APX=0. As soon as you go back
to C, including during the fast path, you have to ensure XCR0.APX=1
again if the kernel is compiled with -mapxf.
For now, I agree that early_xcr0 isn't needed and you can run all the
time with XCR0.APX=0.
Paolo
^ permalink raw reply
* Re: [PATCH 2/2] x86/virt/tdx: Use PFN directly for unmapping guest private memory
From: Yan Zhao @ 2026-04-07 0:44 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Xiaoyao Li, seanjc, dave.hansen, tglx, mingo, bp, kas, x86,
linux-kernel, kvm, linux-coco, kai.huang, rick.p.edgecombe,
yilun.xu, vannapurve, ackerleytng, sagis, binbin.wu,
isaku.yamahata
In-Reply-To: <5b3110f4-4e46-4573-b68e-54e220ae1c19@redhat.com>
On Sat, Apr 04, 2026 at 08:39:00AM +0200, Paolo Bonzini wrote:
> On 3/19/26 09:56, Yan Zhao wrote:
> > On Thu, Mar 19, 2026 at 04:56:10PM +0800, Xiaoyao Li wrote:
> > > So why not considering option 2?
> > >
> > > 2. keep tdx_quirk_reset_page() as-is for the cases of
> > > tdx_reclaim_page() and tdx_reclaim_td_control_pages() that have the
> > > struct page. But only change tdx_sept_remove_private_spte() to use
> > > tdx_quirk_reset_paddr() directly.
> > >
> > > It will need export tdx_quirk_reset_paddr() for KVM. I think it will be OK?
> > I don't think it's necessary. But if we have to export an extra API, IMHO,
> > tdx_quirk_reset_pfn() is better than tdx_quirk_reset_paddr(). Otherwise,
> > why not only expose tdx_quirk_reset_paddr()?
>
> That works for me, it seems the cleanest.
Hi Paolo,
To avoid misunderstanding: you think only exporting tdx_quirk_reset_paddr() is
the cleanest, right? :)
Thanks
Yan
^ permalink raw reply
* Re: [PATCH] KVM: TDX: Fix APIC MSR ranges in tdx_has_emulated_msr()
From: Sean Christopherson @ 2026-04-06 23:07 UTC (permalink / raw)
To: Rick P Edgecombe
Cc: kvm@vger.kernel.org, Dave Hansen, Isaku Yamahata,
dmaluka@chromium.org, x86@kernel.org, kas@kernel.org,
bp@alien8.de, linux-kernel@vger.kernel.org, mingo@redhat.com,
dave.hansen@linux.intel.com, binbin.wu@linux.intel.com,
linux-coco@lists.linux.dev, hpa@zytor.com, tglx@kernel.org,
pbonzini@redhat.com
In-Reply-To: <faac37c7ba9e3a2e8d996e18c74301d9cefad3dc.camel@intel.com>
On Sat, Apr 04, 2026, Rick P Edgecombe wrote:
> On Fri, 2026-04-03 at 16:07 -0700, Sean Christopherson wrote:
> > > So... "Reduced #VE" (also called "VE reduction") reduces which things cause
> > > a #VE. The guest opts into it and the TDX module starts behaving
> > > differently. It's kind of grab bag of changes including changing CPUID
> > > behavior, which is another wrinkle. It was intended to fixup guest side TDX
> > > arch issues.
> >
> > And KVM has no visilibity into which mode the guest has selected? That's
> > awful.
>
> Yea, on both accounts. So where we are at with this is, starting to reject
> changes that build on the pattern. We haven't gone so far as to ask for a
> feature to notify the host of the guest opt-ins. But I wouldn't say we have a
> grand design in mind either. If you have any clarity, please feel free to drop a
> quotable.
I got nothing, probably best to just deal with things on a case-by-case basis
unless we end up with a recurring theme.
> > If KVM has no visiblity, then I don't see an option other than for KVM to
> > advertise and emulate what it can at all times, and it becomes the guest's
> > responsibility to not screw up. I guess it's not really any different from
> > not trying to use MMIO accesses after switching to x2APIC mode.
>
> Like your diff? Expose any MSRs that might be emulated in the TDX paradigm. But
> don't expose all MSRs that KVM supports.
Yep.
^ permalink raw reply
* Re: [PATCH v2 00/16] fs,x86/resctrl: Add kernel-mode (e.g., PLZA) support to the resctrl subsystem
From: Babu Moger @ 2026-04-06 22:45 UTC (permalink / raw)
To: Reinette Chatre, corbet, tony.luck, Dave.Martin, james.morse,
tglx, mingo, bp, dave.hansen
Cc: skhan, x86, hpa, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kas,
rick.p.edgecombe, akpm, pmladek, rdunlap, dapeng1.mi, kees, elver,
paulmck, lirongqing, safinaskar, fvdl, seanjc, pawan.kumar.gupta,
xin, tiala, Neeraj.Upadhyay, chang.seok.bae, thomas.lendacky,
elena.reshetova, linux-doc, linux-kernel, linux-coco, kvm,
eranian, peternewman
In-Reply-To: <83ae0c18-5c5e-4b52-901d-4126fe7c141b@intel.com>
Hi Reinette,
Sorry for the late response. I was trying to get confirmation about the
use case.
On 3/31/26 17:24, Reinette Chatre wrote:
> Hi Babu,
>
> On 3/30/26 11:46 AM, Babu Moger wrote:
>> On 3/27/26 17:11, Reinette Chatre wrote:
>>> On 3/26/26 10:12 AM, Babu Moger wrote:
>>>> On 3/24/26 17:51, Reinette Chatre wrote:
>>>>> On 3/12/26 1:36 PM, Babu Moger wrote:
>
>>>>>> Tony suggested using global variables to store the kernel mode
>>>>>> CLOSID and RMID. However, the kernel mode CLOSID and RMID are
>>>>>> coming from rdtgroup structure with the new interface. Accessing
>>>>>> them requires holding the associated lock, which would make the
>>>>>> context switch path unnecessarily expensive. So, dropped the idea.
>>>>>> https://lore.kernel.org/lkml/aXuxVSbk1GR2ttzF@agluck-desk3/
>>>>>> Let me know if there are other ways to optimize this.
>>>>> I do not see why the context switch path needs to be touched at all with this
>>>>> implementation. Since PLZA only supports global assignment does it not mean that resctrl
>>>>> only needs to update PQR_PLZA_ASSOC when user writes to info/kernel_mode and
>>>>> info/kernel_mode_assignment?
>>>> Each thread has an MSR to configure whether to associate privilege level zero execution with a separate COS and/or RMID, and the value of the COS and/or RMID. PLZA may be enabled or disabled on a per-thread basis. However, the COS and RMID association and configuration must be the same for all threads in the QOS Domain.
>>> Based on previous comment in https://lore.kernel.org/lkml/abb049fa-3a3d-4601-9ae3-61eeb7fd8fcf@amd.com/
>>> and this implementation all fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_en must be the
>>> same for all CPUs on the system, not just per QoS domain. Could you please confirm?
>>
>> Sorry for the confusion. It is "per QoS domain".
>>
>> All the fields of PQR_PLZA_ASSOC except PQR_PLZA_ASSOC.plza_enmust be set to the same value for all HW threads in the QOS domain for consistent operation (Per-QosDomain).
>
> Thank you for clarifying. To build on this, what would be best way for resctrl to interpret this?
> As I see it all values in PQR_PLZA_ASSOC apply to *all* resources yet (theoretically?) every resource
Yes. That is correct. PLZA applies to all the resources.
> can have domains that span different CPUs. There thus seem to be a built in assumption of what a "domain"
> means for PQR_PLZA_ASSOC so it sounds to me as though, instead of saying that "PQR_PLZA_ASSOC needs
> to be the same in QoS domain" it may be more accurate to, for example, say that "PQR_PLZA_ASSOC has L3 scope"?
Yes.
>
> This seems to be what this implementation does since it hardcodes PQR_PLZA_ASSOC scope to the L3
> resource but that creates dependency to the L3 resource that would make PLZA unusable if, for example,
> the user boots with "rdt=!l3cat" while wanting to use PLZA to manage MBA allocations when in kernel?
Yes. that is correct. It should not be attached to one resource. We need
to change it to global scope.
>
> ...
>
>> Yes, I agree with your concerns. The goal here is to make the interface less disruptive while still addressing the different use cases.
>
> I consider changing resctrl behavior when values are written to existing resctrl files
> to be disruptive. This is something we explicitly discussed during v1 as something to
> be avoided so this implementation that overloads the tasks file again is unexpected.
Yes. Agree. If required we need to introduce new files (kmode_cpus,
kmode_cpu_list or kmode_tasks) to handle these cases.
>
>> Background: Customers have identified an issue with the QoS
>> Bandwidth Control feature: when a CLOS is aggressively throttled
>> and execution transitions into kernel mode, kernel operations are
>> also subject to the same aggressive throttling.
>>
>>> Privilege-Level Zero Association (PLZA) allows a user to specify a
>> COS and/or RMID to be used during execution at Privilege Level Zero.
>> When PLZA is enabled on a hardware thread, any execution that enters
>> Privilege Level Zero will have its transactions associated with the
>> PLZA COS and/or RMID. Otherwise, the thread continues to use the COS
>> and RMID specified by |PQR_ASSOC|. In other words, the hardware
>> provides a dedicated COS and/or RMID specifically for kernel-mode
>> execution.
> ack.
>
>>
>> There are multiple ways this feature can be applied. For simplicity, the discussion below focuses only on CLOSID.
>>
>>
>> 1. Global PLZA enablement
>>
>> PLZA can be configured as a global feature by setting |PQR_PLZA_ASSOC.closid = CLOSID| and |PQR_PLZA_ASSOC.plza_en = 1| on all threads in the system. A dedicated CLOSID is reserved for this purpose,
>
> Also discussed during v1 is that there is no need to dedicate a CLOSID for this purpose.
> There could be an "unthrottled" CLOSID to which all high priority user space tasks as
> well as all kernel work of all tasks are assigned.
> If user space chooses to dedicate a CLOSID for kernel work then that should supported and
> interface can allow that, but there is no need for resctrl to enforce this.
>
>> and all CPU threads use its allocations whenever they enter Privilege Level Zero. This CLOSID does not need to be associated with any resctrl group.
I misspoke here.
>
> The CLOSID has to be associated with a resource group to be able to manage its
> resource allocations, no?
Yes. We need to have resource group schemata to enforce the limits.
>
>> The user can explicitly enable or disable this feature.
> ack.
>
>> There is no context switch overhead but there is no flexibility with this approach.
>
> Flexibility is subjective. As I understand this supports the only use case we learned about so far:
> https://lore.kernel.org/lkml/CABPqkBSq=cgn-am4qorA_VN0vsbpbfDePSi7gubicpROB1=djw@mail.gmail.com/
>
>> 2. Group based PLZA allocation : PLZA is managed via dedicated
>> restctrl group. A separate resctrl group can be created
>> specifically for PLZA, with a dedicated CLOSID used exclusively
>> for kernel mode execution. This approach can be further divided
>> into two association models:
>
> So far this sounds like global allocation since both need a dedicated resource group.
> Whether this group is dedicated to kernel work or shared between kernel and user space work
> is up to the user. There is no motivation why CLOSID should ever be enforced to be
> exclusive for kernel mode execution.
Yes. That is fine.
>
>>
>> i) CPU based association
>> CPUs are assigned to the PLZA group, and PLZA is enabled only on
>> those CPUs. This effectively creates a dedicated PLZA group. MSRs (|
>> PQR_PLZA_ASSOC)| are programmed only when the user changes CPU
>> assignments. This approach requires no changes to the context switch
>> code and introduces no additional context switch overhead.
>>
>> ii) Task based association
>> Tasks are explicitly assigned by the user to the PLZA group. Tasks
>> need to be updated when user adds a new task. Also, this requires
>> updates during task scheduling so that the MSRs (|PQR_PLZA_ASSOC)|
>> are programmed on each context switch, which introduces additional
>> context switch overhead.
>
> As discussed during v1 any changes needed to support per task assignment would
> need to be done with new files dedicated to this purpose. Do not overload the
> existing resctrl tasks/cpus/cpus_list files.
Yes. Sure.
>
>> I tried to fit these requirements into the interface files in /sys/
>> fs/resctrl/info/. I may have missed few things while trying to
>> achieve it. As usual, I am open for the discussion and
>> recommendations.
>
> Many of these items were already discussed as part of v1 so I think we may be
> talking past each other here. I tried to highlight the relevant points raised
> during v1 discussion that I thought there already was agreement on.
>
> The one new aspect is that I assumed this implementation will only be for
> global configuration and assignment. It looks like you want to support both
> global configuration and per-task assignment. In the original I did not consider
> configuration and assignment to occur at different scope so we may need to come up
> with new modes to distinguish. Consider the addition of two modes as below:
>
> # cat info/kernel_mode
> [inherit_ctrl_and_mon]
> global_assign_ctrl_inherit_mon_set_all
> global_assign_ctrl_assign_mon_set_all
> global_assign_ctrl_inherit_mon_set_individual
> global_assign_ctrl_assign_mon_set_individual
>
> Above introduces a "set_all" and "set_individual" suffix to the original two
> modes.
>
> global_assign_ctrl_inherit_mon_set_all
> global_assign_ctrl_assign_mon_set_all:
>
> Above are the original two modes but makes it clear that when this mode is
> activated _all_ tasks run with the assignment.
>
> global_assign_ctrl_inherit_mon_set_individual
> global_assign_ctrl_assign_mon_set_individual:
>
> Above are two new modes. In this mode user space also assigns a resource
> group globally but then needs to follow that up by activating every task
> separately to run with this assignment.
> One way in which this can be accomplished could be to have "kernel_mode_tasks",
> "kernel_mode_cpus", and "kernel_mode_cpus_list" files become visible (or be
> created) in the resource group found in info/kernel_mode_assignment. User
> space interacts with the new files to set which tasks and/or CPUs run with
> PLZA enabled.
>
> Even so, as I understand global_assign_ctrl_inherit_mon_set_all and
> global_assign_ctrl_assign_mon_set_all addresses the only known use case. Do you know
> if there are use cases for global_assign_ctrl_inherit_mon_set_individual and
> global_assign_ctrl_assign_mon_set_individual? The latter two adds significant
> complexity to resctrl while I have not heard about any use case for it.
>
Yes. I agree. The changes in context switch code is a concern.
You covered some of the cases I was thinking(xx_set_individual).
How about this idea?
I suggest splitting the PLZA into two distinct aspects:
1. How PLZA is applied within a resource group
2. How PLZA is monitored
Introduce a new file, "info/kmode_type", to describe how kmode applies
in the system.
# cat info/kmode_type
[global] <- Kernel mode applies to the entire system (all CPUs/tasks)
cpus <- Kernel mode applies only to the CPUs in the group
tasks <- Kernel mode applies only to the tasks in the group
The "global" option is the default right now and it is current common
use-case.
The "info/kmode_type -> cpus" option introduces new files "kmode_cpus"
and "kmode_cpus_list" for users to apply kmode to specific set of CPUs.
This lets users change the CPU set for PLZA. The PLZA MSR is updated
when user changes the association to the file. No context switch code
changes are needed. This will be dedicated group. The current resctrl
group files, "cpus, cpus_list and tasks" will not be accessible in this
mode. This option give some flexibility for the user without the context
switch overhead.
The "info/kmode_type -> tasks" option introduces a new file,
"kmode_tasks", for users to apply kmode to specific set of tasks. This
requires context switch changes. This will be dedicated group. The
current resctrl group files, "cpus, cpus_list and tasks" will not be
accessible in this mode. We currently have no use case for this, so it
will not be supported now.
Add a file, "info/kmode_monitor", to describe how kmode is monitored.
# cat info/kmode_monitor
[inherit_ctrl_and_mon] <- Kernel uses the same CLOSID/RMID as user.
Default option for the "global"
assign_ctrl_inherit_mon <- One CLOSID for all kernel work; RMID
inherited from user.
assign_ctrl_assign_mon <- One resource group (CLOSID+RMID) for all
kernel work. Default option for "cpu" type.
Rename “kernel_mode_assignment” to “kmode_group” to assign the specific
group to kmode. This file usage is same as before.
#cat info/kmode_groups (Renamed "kernel_mode_assignment")
//
Thoughts?
thanks
Babu
^ permalink raw reply
* Re: [PATCH v7 17/22] x86/virt/tdx: Avoid updates during update-sensitive operations
From: Sean Christopherson @ 2026-04-06 22:29 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, kas, nik.borisov, paulmck,
pbonzini, reinette.chatre, rick.p.edgecombe, sagis, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, xiaoyao.li, yan.y.zhao,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, x86,
H. Peter Anvin
In-Reply-To: <20260331124214.117808-18-chao.gao@intel.com>
On Tue, Mar 31, 2026, Chao Gao wrote:
> A runtime TDX module update can conflict with TD lifecycle operations that
> are update-sensitive.
>
> Today, update-sensitive operations include:
>
> - TD build: TD measurement is accumulated across multiple
> TDH.MEM.PAGE.ADD, TDH.MR.EXTEND, and TDH.MR.FINALIZE calls.
>
> - TD migration: intermediate crypto state is saved/restored across
> interrupted/resumed TDH.EXPORT.STATE.* and TDH.IMPORT.STATE.* flows.
>
> If an update races TD build, for example, TD measurement can become
> incorrect and attestation can fail.
>
> The TDX architecture exposes two approaches:
>
> 1) Avoid updates during update-sensitive operations.
> 2) Detect incompatibility after update and recover.
>
> Post-update detection (option #2) is not a good fit: as discussed in [1],
> future module behavior may expand update-sensitive operations in ways that
> make KVM ABIs unstable and will break userspace.
>
> "Do nothing" is also not preferred: while it keeps kernel code simple, it
> lets the issue leak into the broader stack, where both detection and
> recovery require significantly more effort.
>
> So, use option #1. Specifically, request "avoid update-sensitive" behavior
> during TDX module shutdown and map the resulting failure to -EBUSY so
> userspace can distinguish an update race from other failures.
>
> When the "avoid update-sensitive" feature isn't supported, proceed with
> updates. If a race occurs between module update and update-sensitive
> operations, failures happen at a later stage (e.g., incorrect TD
> measurements in attestation reports for TD build). Effectively, this
> means "let userspace update at their own risk". Userspace can check if
> the feature is supported or not. The alternative of blocking updates
> entirely is rejected [2] as it introduces permanent kernel complexity to
> accommodate limitations in early TDX module releases that userspace can
> handle.
>
> Note: this implementation is based on a reference patch by Vishal [3].
> Note2: moving "NO_RBP_MOD" is just to centralize bit definitions.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Link: https://lore.kernel.org/linux-coco/aQIbM5m09G0FYTzE@google.com/ # [1]
> Link: https://lore.kernel.org/kvm/699fe97dc212f_2f4a100b@dwillia2-mobl4.notmuch/ # [2]
> Link: https://lore.kernel.org/linux-coco/CAGtprH_oR44Vx9Z0cfxvq5-QbyLmy_+Gn3tWm3wzHPmC1nC0eg@mail.gmail.com/ # [3]
> ---
For the STATUS_MASK movement:
Acked-by: Sean Christopherson <seanjc@google.com>
> ---
> arch/x86/include/asm/tdx.h | 11 +++++++++--
> arch/x86/kvm/vmx/tdx_errno.h | 2 --
> arch/x86/virt/vmx/tdx/tdx.c | 25 +++++++++++++++++++++----
> arch/x86/virt/vmx/tdx/tdx.h | 3 ---
> 4 files changed, 30 insertions(+), 11 deletions(-)
>
> diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
> index 79733fdb35c6..00751506dd3c 100644
> --- a/arch/x86/include/asm/tdx.h
> +++ b/arch/x86/include/asm/tdx.h
> @@ -26,11 +26,18 @@
> #define TDX_SEAMCALL_GP (TDX_SW_ERROR | X86_TRAP_GP)
> #define TDX_SEAMCALL_UD (TDX_SW_ERROR | X86_TRAP_UD)
>
> +#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL
> +
...
> diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/kvm/vmx/tdx_errno.h
> index 6ff4672c4181..215c00d76a94 100644
> --- a/arch/x86/kvm/vmx/tdx_errno.h
> +++ b/arch/x86/kvm/vmx/tdx_errno.h
> @@ -4,8 +4,6 @@
> #ifndef __KVM_X86_TDX_ERRNO_H
> #define __KVM_X86_TDX_ERRNO_H
>
> -#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL
> -
> /*
> * TDX SEAMCALL Status Codes (returned in RAX)
> */
^ permalink raw reply
* Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
From: Jason Gunthorpe @ 2026-04-06 22:21 UTC (permalink / raw)
To: Alexey Kardashevskiy
Cc: Xu Yilun, Aneesh Kumar K.V, Dan Williams, linux-coco, linux-pci,
gregkh, bhelgaas, alistair23, lukas, Arnd Bergmann
In-Reply-To: <70912675-0737-4ebf-8ba0-ab9a2e493bbe@amd.com>
On Tue, Apr 07, 2026 at 08:08:51AM +1000, Alexey Kardashevskiy wrote:
>
>
> On 4/4/26 01:08, Jason Gunthorpe wrote:
> > On Fri, Apr 03, 2026 at 11:41:25PM +1100, Alexey Kardashevskiy wrote:
> > >
> > >
> > > On 30/3/26 22:49, Jason Gunthorpe wrote:
> > > > On Mon, Mar 30, 2026 at 04:47:44PM +1100, Alexey Kardashevskiy wrote:
> > > >
> > > > > What do I miss? Thanks,
> > > >
> > > > You can't tell where things start so there is no way to relate the
> > > > offsets to something the kernel can understand.
> > >
> > > Reported ranges have BAR indexes and start addresses (with the
> > > reported MMIO offset added), and the first reported range starts at
> > > the first 4K of that BAR.
> >
> > I was told this is not the case, the first reported range can start
> > anywhere in the BAR?
>
> This is what I am trying to clarify - if all ranges must be reported
> (as some think this is what the PCIe spec says), then no, not
> anywhere.
>
> pcie r7, Table 11-16 TDI Report Structure, MMIO_RANGE:
>
> "Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report
non-secure ranges? But I don't know, it was hashed out in some thread
for ARM and then I know our folks looked at it and nobody pushed back
to insist that every single byte of the BAR had to be covered by a
reported range.
I wouldn't take the sentance you quoted as confirmation, you need a
sentance that says every single byte of the BAR is covered by a single
reported range.
Jason
^ permalink raw reply
* Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
From: Alexey Kardashevskiy @ 2026-04-06 22:08 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Xu Yilun, Aneesh Kumar K.V, Dan Williams, linux-coco, linux-pci,
gregkh, bhelgaas, alistair23, lukas, Arnd Bergmann
In-Reply-To: <20260403140858.GJ310919@nvidia.com>
On 4/4/26 01:08, Jason Gunthorpe wrote:
> On Fri, Apr 03, 2026 at 11:41:25PM +1100, Alexey Kardashevskiy wrote:
>>
>>
>> On 30/3/26 22:49, Jason Gunthorpe wrote:
>>> On Mon, Mar 30, 2026 at 04:47:44PM +1100, Alexey Kardashevskiy wrote:
>>>
>>>> What do I miss? Thanks,
>>>
>>> You can't tell where things start so there is no way to relate the
>>> offsets to something the kernel can understand.
>>
>> Reported ranges have BAR indexes and start addresses (with the
>> reported MMIO offset added), and the first reported range starts at
>> the first 4K of that BAR.
>
> I was told this is not the case, the first reported range can start
> anywhere in the BAR?
This is what I am trying to clarify - if all ranges must be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure, MMIO_RANGE:
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
--
Alexey
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Sean Christopherson @ 2026-04-06 22:00 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Chang S. Bae, Kiryl Shutsemau, kvm, the arch/x86 maintainers,
linux-coco, Kernel Mailing List, Linux, Andrew Cooper
In-Reply-To: <CABgObfbLm3FR4f_nv5EyYJx4jwfeBaVgTLhr7P++hmhCP98e3Q@mail.gmail.com>
On Mon, Apr 06, 2026, Paolo Bonzini wrote:
> Il lun 6 apr 2026, 17:28 Sean Christopherson <seanjc@google.com> ha scritto:
> > > You're right about fast paths...
> >
> > Ya, potential fastpath usage is why I wanted to just context switch around
> > entry/exit.
> >
> > > so something like the attached patch.
> > > It is not too bad to translate into assembly, where it could use
> > > alternatives (in the same way as
> > > RESTORE_GUEST_SPEC_CTRL/RESTORE_GUEST_SPEC_CTRL_BODY) in place of
> > > static_cpu_has(). Maybe it's best to bite the bullet and do it
> > > already...
> >
> > My strong vote is to context switch in assembly, but _conditionally_ context
> > switch R16-R31.
> >
> > But that second paragraph isn't quite correct, at least not for KVM. Specifically,
> > "need a branch prior to regaining speculative safety" isn't correct, as that holds
> > true if and only if "regaining speculative safety" requires executing code that
> > might access R16-R31. If we massage __vmx_vcpu_run() to restore SPEC_CTRL in
> > assembly, same as __svm_vcpu_run(), then __{svm,vmx}_vcpu_run() can simply context
> > switch R16-R31 if and only if APX is enabled in XCR0.
>
> I might even have patches for that lying around (the SPEC_CTRL part).
>
> > KVM always intercepts XCR0 writes (when XCR0 isn't context switched by "hardware",
> > i.e. ignoring SEV-ES+ and TDX guests), and IIUC all access to R16-R31 is gated on
> > XCR0.APX=1
>
> Right, fortunately.
>
> > . So unless I'm missing something (or hardware is flawed and lets the
> > guest speculative consume R16-R31, which would be sad), it's perfectly safe to
> > run the guest with host state in R16-R31.
> >
> > That would avoid pointlessly context switching 16 registers when APX is not being
> > used by the guest, and would avoid having to write XCR0 in the fastpath.
>
> For now yes, but once/if the kernel starts using the registers there's
> no way out of writing XCR0 for APX-disabled guests in the fast path.
Why's that? So long as KVM uses vcpu->arch.regs[R16-R31] as the source of truth
when emulating anything, there's no danger of taking a #UD in the host due to
accessing R16-R31 with XCR0.APX=0. There's not even any danger of consuming stale
guest state, e.g. in case KVM screws up accesses R16-R31 instead of generating #UD,
as the value in regs[] will still be the guest's last written value.
If we wanted be paranoid, we could add sanity checks to ensure R16-R31 don't show
up in hardware-provided informational fields, but to some extent that's orthogonal
to how KVM maintains guest values.
> If we ignore that, we can keep guest XCR0 all the time for now, and
> that would be:
> - move SPEC_CTRL to assembly
> - not changing XCR0 handling at all
> - use XCR0 in addition to just static_cpu_has(X86_FEATURE_APX) to make
> r16-r31 swap conditional
>
> > > - if (vcpu->arch.xcr0 != kvm_host.xcr0)
> > > + /*
> > > + * Do not load the definitive XCR0 yet; vcpu->arch.early_xcr0 keeps
> > > + * APX enabled so that the kernel can move to and from r16...r31.
> > > + */
> > > + if (vcpu->arch.early_xcr0 != kvm_host.xcr0)
> > > xsetbv(XCR_XFEATURE_ENABLED_MASK,
> > > - load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
> > > + load_guest ? vcpu->arch.early_xcr0 : kvm_host.xcr0);
> >
> > Even _if_ we want to play XCR0 games,
>
> (which depends on whether we want to be ready for kernel usage of APX, right?)
No?
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Paolo Bonzini @ 2026-04-06 21:41 UTC (permalink / raw)
To: Sean Christopherson
Cc: Chang S. Bae, Kiryl Shutsemau, kvm, the arch/x86 maintainers,
linux-coco, Kernel Mailing List, Linux, Andrew Cooper
In-Reply-To: <adPRA4ZhnvbaXSn0@google.com>
Il lun 6 apr 2026, 17:28 Sean Christopherson <seanjc@google.com> ha scritto:
> > You're right about fast paths...
>
> Ya, potential fastpath usage is why I wanted to just context switch around
> entry/exit.
>
> > so something like the attached patch.
> > It is not too bad to translate into assembly, where it could use
> > alternatives (in the same way as
> > RESTORE_GUEST_SPEC_CTRL/RESTORE_GUEST_SPEC_CTRL_BODY) in place of
> > static_cpu_has(). Maybe it's best to bite the bullet and do it
> > already...
>
> My strong vote is to context switch in assembly, but _conditionally_ context
> switch R16-R31.
>
> But that second paragraph isn't quite correct, at least not for KVM. Specifically,
> "need a branch prior to regaining speculative safety" isn't correct, as that holds
> true if and only if "regaining speculative safety" requires executing code that
> might access R16-R31. If we massage __vmx_vcpu_run() to restore SPEC_CTRL in
> assembly, same as __svm_vcpu_run(), then __{svm,vmx}_vcpu_run() can simply context
> switch R16-R31 if and only if APX is enabled in XCR0.
I might even have patches for that lying around (the SPEC_CTRL part).
> KVM always intercepts XCR0 writes (when XCR0 isn't context switched by "hardware",
> i.e. ignoring SEV-ES+ and TDX guests), and IIUC all access to R16-R31 is gated on
> XCR0.APX=1
Right, fortunately.
> . So unless I'm missing something (or hardware is flawed and lets the
> guest speculative consume R16-R31, which would be sad), it's perfectly safe to
> run the guest with host state in R16-R31.
>
> That would avoid pointlessly context switching 16 registers when APX is not being
> used by the guest, and would avoid having to write XCR0 in the fastpath.
For now yes, but once/if the kernel starts using the registers there's
no way out of writing XCR0 for APX-disabled guests in the fast path.
If we ignore that, we can keep guest XCR0 all the time for now, and
that would be:
- move SPEC_CTRL to assembly
- not changing XCR0 handling at all
- use XCR0 in addition to just static_cpu_has(X86_FEATURE_APX) to make
r16-r31 swap conditional
> > - if (vcpu->arch.xcr0 != kvm_host.xcr0)
> > + /*
> > + * Do not load the definitive XCR0 yet; vcpu->arch.early_xcr0 keeps
> > + * APX enabled so that the kernel can move to and from r16...r31.
> > + */
> > + if (vcpu->arch.early_xcr0 != kvm_host.xcr0)
> > xsetbv(XCR_XFEATURE_ENABLED_MASK,
> > - load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
> > + load_guest ? vcpu->arch.early_xcr0 : kvm_host.xcr0);
>
> Even _if_ we want to play XCR0 games,
(which depends on whether we want to be ready for kernel usage of APX, right?)
> tracking early_xcr0 is unnecessary. This can be:
>
> /*
> * XCR0 is context switched around VM-Enter/VM-Exit if APX is enabled
> * in the host but not in the guest.
> */
> if (vcpu->arch.xcr0 != kvm_host.xcr0 &&
> (!cpu_feature_enabled(X86_FEATURE_APX) ||
> vcpu->arch.xcr0 & XFEATURE_MASK_APX))
> xsetbv(XCR_XFEATURE_ENABLED_MASK,
> load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
This is a bit more complex however, because in the end early_xcr0 is
precomputing the same conditions and optimizations. For example...
> > +void __kvm_load_guest_apx(struct kvm_vcpu *vcpu)
> > +{
> > + if (vcpu->arch.early_xcr0 != vcpu->arch.xcr0)
> > + xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
>
> This is wrong. The "real" xcr0 needs to be loaded *after* accessing R16+.
... this is actually the same optimization you mention above: the real
xcr0 only needs to be loaded if APX is off in the guest, and in that
case you don't need to load r16-r31. So you can load xcr0 first, and
then any components (for now only APX) that need to be swapped.
> > + if (!(vcpu->arch.xcr0 & XFEATURE_MASK_APX))
> > + return;
... Because the loads are conditional on APX being enabled in the real xcr0.
Paolo
> > +
> > + WARN_ON_ONCE(!irqs_disabled());
> > +
> > + asm("mov %[r16], %%r16\n"
> > + "mov %[r17], %%r17\n" // ...
> > + : : [r16] "m" (vcpu->arch.regs[16]),
> > + [r17] "m" (vcpu->arch.regs[17]));
> > +}
>
^ permalink raw reply
* Re: [PATCH] dma-buf: heaps: system: document system_cc_shared heap
From: T.J. Mercier @ 2026-04-06 20:20 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, christian.koenig,
m.szyprowski, robin.murphy, jgg, leon, sean.anderson, ptesarik,
catalin.marinas, aneesh.kumar, suzuki.poulose, steven.price,
thomas.lendacky, john.allen, ashish.kalra, suravee.suthikulpanit,
linux-coco
In-Reply-To: <20260402141103.598495-1-jiri@resnulli.us>
On Thu, Apr 2, 2026 at 7:11 AM Jiri Pirko <jiri@resnulli.us> wrote:
>
> From: Jiri Pirko <jiri@nvidia.com>
>
> Document the system_cc_shared dma-buf heap that was introduced
> recently. Describe its purpose, availability conditions and
> relation to confidential computing VMs.
>
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
> ---
> Documentation/userspace-api/dma-buf-heaps.rst | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/Documentation/userspace-api/dma-buf-heaps.rst b/Documentation/userspace-api/dma-buf-heaps.rst
> index 05445c83b79a..591732393e7d 100644
> --- a/Documentation/userspace-api/dma-buf-heaps.rst
> +++ b/Documentation/userspace-api/dma-buf-heaps.rst
> @@ -16,6 +16,14 @@ following heaps:
>
> - The ``system`` heap allocates virtually contiguous, cacheable, buffers.
>
> + - The ``system_cc_shared`` heap allocates virtually contiguous, cacheable,
> + buffers using shared (decrypted) memory. It is only present on
> + confidential computing (CoCo) VMs where memory encryption is active
> + (e.g., AMD SEV, Intel TDX). The allocated pages have the encryption
> + bit cleared, making them accessible for device DMA without TDISP
> + support. On non-CoCo VMs configurations, this heap is
"non-CoCo VM configurations"
> + not registered.
Doesn't seem like you need to wrap this line.
with that: Reviewed-by: T.J.Mercier <tjmercier@google.com>
> +
> - The ``default_cma_region`` heap allocates physically contiguous,
> cacheable, buffers. Only present if a CMA region is present. Such a
> region is usually created either through the kernel commandline
Each paragraph starting with '-' confused me for a second there. Those
aren't part of the diff. :)
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Sean Christopherson @ 2026-04-06 15:40 UTC (permalink / raw)
To: Dave Hansen
Cc: Paolo Bonzini, Kiryl Shutsemau, kvm, x86, linux-coco,
linux-kernel, Chang S . Bae
In-Reply-To: <fb9e61cc-20df-47cd-b324-1dd8d5985372@intel.com>
On Fri, Apr 03, 2026, Dave Hansen wrote:
> On 4/2/26 16:19, Sean Christopherson wrote:
> > Do we know what the compiler and/or kernel rules for using R16-R31 will be?
> > E.g. if C code is allowed to use R16-R31 at will, then KVM will either need to
> > swap R16-R31 in assembly, or annotate a pile of functions as "no_egpr" or
> > whatever.
>
> My _assumption_ is that the speedup from using the new GPRs as GPRs in
> the kernel is going to be enough for us to support it. This is even
> though those kernel binaries won't run on old hardware.
>
> If I'm right, then we're going to have to handle the new GPRs just like
> the existing ones and save them on kernel entry before we hit C code.
Ooof, one nasty wrinkle to prepare for is an NMI that arrives after VM-Exit on
Intel CPUs. Unless Intel extends VMX to context switch XCR0 at VM-Entry/VM-Exit,
and/or provides GIF-like functionality (which would be awesome!), it will be
possible for an NMI to be taken with the guest's XCR0 loaded, i.e. with XCR0.APX=0
even when APX is fully enabled in the host.
> I'm not sure I want to be messing with XSAVE there. XSAVE requires
> munging a header which means even if we used XSAVE we'd need to XSAVE
> and then copy things over to pt_regs (assuming we continue using pt_regs).
>
> That doesn't seem like loads of fun because we'll also need to copy out
> to the XSAVE UABI spots, like PKRU times 32.
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Sean Christopherson @ 2026-04-06 15:28 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Chang S. Bae, Kiryl Shutsemau, kvm, x86, linux-coco, linux-kernel,
Andrew Cooper
In-Reply-To: <CABgObfY05NU8DS82jkwpF89_p1nR7VJ30HBq_xaMg_u+-j=0Cw@mail.gmail.com>
+Andrew
On Sat, Apr 04, 2026, Paolo Bonzini wrote:
> On Sat, Apr 4, 2026 at 12:05 AM Chang S. Bae <chang.seok.bae@intel.com> wrote:
> >
> > On 4/3/2026 9:03 AM, Paolo Bonzini wrote:
> > >
> > > But until the kernel starts using APX, I would do the save/restore near
> > > kvm_load_xfeatures(), because __vmx_vcpu_run()/__svm_vcpu_run() would
> > > have to check whether xcr0.apx is set or not.
> > Right, I'd much prefer this. Then, it requires to audit whether any
> > fast-path handler could access EGPRs.
> >
> > But there are cases with the new {RD|WR}MSR (MSR_IMM) instructions that
> > appear to access GPRs. Because of this, the EGPR saving/restoring needs
> > to happen earlier.
>
> You're right about fast paths...
Ya, potential fastpath usage is why I wanted to just context switch around
entry/exit.
> so something like the attached patch.
> It is not too bad to translate into assembly, where it could use
> alternatives (in the same way as
> RESTORE_GUEST_SPEC_CTRL/RESTORE_GUEST_SPEC_CTRL_BODY) in place of
> static_cpu_has(). Maybe it's best to bite the bullet and do it
> already...
My strong vote is to context switch in assembly, but _conditionally_ context
switch R16-R31. All of this started from Andrew's comment:
: You can't unconditionally use PUSH2/POP2 in the VMExit, because at that
: point in time it's the guest's XCR0 in context. If the guest has APX
: disabled, PUSH2 in the VMExit path will #UD.
:
: You either need two VMExit handlers, one APX and one non-APX and choose
: based on the guest XCR0 value, or you need a branch prior to regaining
: speculative safety, or you need to save/restore XCR0 as the first
: action. It's horrible any way you look at it.
But that second paragraph isn't quite correct, at least not for KVM. Specifically,
"need a branch prior to regaining speculative safety" isn't correct, as that holds
true if and only if "regaining speculative safety" requires executing code that
might access R16-R31. If we massage __vmx_vcpu_run() to restore SPEC_CTRL in
assembly, same as __svm_vcpu_run(), then __{svm,vmx}_vcpu_run() can simply context
switch R16-R31 if and only if APX is enabled in XCR0.
KVM always intercepts XCR0 writes (when XCR0 isn't context switched by "harware",
i.e. ignoring SEV-ES+ and TDX guests), and IIUC all access to R16-R31 is gated on
XCR0.APX=1. So unless I'm missing something (or hardware is flawed and lets the
guest speculative consume R16-R31, which would be sad), it's perfectly safe to
run the guest with host state in R16-R31.
That would avoid pointlessly context switching 16 registers when APX is not being
used by the guest, and would avoid having to write XCR0 in the fastpath.
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 959fcc01ee0f..9a1766037b6f 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -887,6 +887,7 @@ struct kvm_vcpu_arch {
> struct fpu_guest guest_fpu;
>
> u64 xcr0;
> + u64 early_xcr0;
...
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 0757b93e528d..69abfdd946dd 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -1220,9 +1220,13 @@ static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, bool load_guest)
> if (!kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE))
> return;
>
> - if (vcpu->arch.xcr0 != kvm_host.xcr0)
> + /*
> + * Do not load the definitive XCR0 yet; vcpu->arch.early_xcr0 keeps
> + * APX enabled so that the kernel can move to and from r16...r31.
> + */
> + if (vcpu->arch.early_xcr0 != kvm_host.xcr0)
> xsetbv(XCR_XFEATURE_ENABLED_MASK,
> - load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
> + load_guest ? vcpu->arch.early_xcr0 : kvm_host.xcr0);
Even _if_ we want to play XCR0 games, tracking early_xcr0 is unnecessary. This
can be:
/*
* XCR0 is context switched around VM-Enter/VM-Exit if APX is enabled
* in the host but not in the guest.
*/
if (vcpu->arch.xcr0 != kvm_host.xcr0 &&
(!cpu_feature_enabled(X86_FEATURE_APX) ||
vcpu->arch.xcr0 & XFEATURE_MASK_APX))
xsetbv(XCR_XFEATURE_ENABLED_MASK,
load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
And then __kvm_load_guest_apx()
<context switch R16-R31>
if (cpu_feature_enabled(X86_FEATURE_APX) &&
!(vcpu->arch.xcr0 & & XFEATURE_MASK_APX))
xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
And __kvm_save_guest_apx() would reverse the order of __kvm_load_guest_apx().
> @@ -11056,6 +11061,49 @@ static void kvm_vcpu_reload_apic_access_page(struct kvm_vcpu *vcpu)
> kvm_x86_call(set_apic_access_page_addr)(vcpu);
> }
>
> +/*
> + * Assuming the kernel does not use APX for now. When
> + * the kernel starts using APX this needs to move into
> + * assembly, and KVM_GET/SET_XSAVE needs to fill in
> + * EGPRs from vcpu->arch.regs.
> + */
> +void __kvm_load_guest_apx(struct kvm_vcpu *vcpu)
> +{
> + if (vcpu->arch.early_xcr0 != vcpu->arch.xcr0)
> + xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
This is wrong. The "real" xcr0 needs to be loaded *after* accessing R16+.
> + if (!(vcpu->arch.xcr0 & XFEATURE_MASK_APX))
> + return;
> +
> + WARN_ON_ONCE(!irqs_disabled());
> +
> + asm("mov %[r16], %%r16\n"
> + "mov %[r17], %%r17\n" // ...
> + : : [r16] "m" (vcpu->arch.regs[16]),
> + [r17] "m" (vcpu->arch.regs[17]));
> +}
^ permalink raw reply
* Re: [PATCH 2/2] x86/virt/tdx: Use PFN directly for unmapping guest private memory
From: Paolo Bonzini @ 2026-04-04 6:39 UTC (permalink / raw)
To: Yan Zhao, Xiaoyao Li
Cc: seanjc, dave.hansen, tglx, mingo, bp, kas, x86, linux-kernel, kvm,
linux-coco, kai.huang, rick.p.edgecombe, yilun.xu, vannapurve,
ackerleytng, sagis, binbin.wu, isaku.yamahata
In-Reply-To: <abu6MVdXar3MmTdF@yzhao56-desk.sh.intel.com>
On 3/19/26 09:56, Yan Zhao wrote:
> On Thu, Mar 19, 2026 at 04:56:10PM +0800, Xiaoyao Li wrote:
>> So why not considering option 2?
>>
>> 2. keep tdx_quirk_reset_page() as-is for the cases of
>> tdx_reclaim_page() and tdx_reclaim_td_control_pages() that have the
>> struct page. But only change tdx_sept_remove_private_spte() to use
>> tdx_quirk_reset_paddr() directly.
>>
>> It will need export tdx_quirk_reset_paddr() for KVM. I think it will be OK?
> I don't think it's necessary. But if we have to export an extra API, IMHO,
> tdx_quirk_reset_pfn() is better than tdx_quirk_reset_paddr(). Otherwise,
> why not only expose tdx_quirk_reset_paddr()?
That works for me, it seems the cleanest.
Paolo
^ permalink raw reply
* Re: [PATCH 0/7] KVM: x86: APX reg prep work
From: Paolo Bonzini @ 2026-04-04 5:16 UTC (permalink / raw)
To: Chang S. Bae
Cc: Sean Christopherson, Kiryl Shutsemau, kvm, x86, linux-coco,
linux-kernel
In-Reply-To: <aea176c8-35de-4042-bf98-e42ce05f93fb@intel.com>
[-- Attachment #1: Type: text/plain, Size: 951 bytes --]
On Sat, Apr 4, 2026 at 12:05 AM Chang S. Bae <chang.seok.bae@intel.com> wrote:
>
> On 4/3/2026 9:03 AM, Paolo Bonzini wrote:
> >
> > But until the kernel starts using APX, I would do the save/restore near
> > kvm_load_xfeatures(), because __vmx_vcpu_run()/__svm_vcpu_run() would
> > have to check whether xcr0.apx is set or not.
> Right, I'd much prefer this. Then, it requires to audit whether any
> fast-path handler could access EGPRs.
>
> But there are cases with the new {RD|WR}MSR (MSR_IMM) instructions that
> appear to access GPRs. Because of this, the EGPR saving/restoring needs
> to happen earlier.
You're right about fast paths... so something like the attached patch.
It is not too bad to translate into assembly, where it could use
alternatives (in the same way as
RESTORE_GUEST_SPEC_CTRL/RESTORE_GUEST_SPEC_CTRL_BODY) in place of
static_cpu_has(). Maybe it's best to bite the bullet and do it
already...
Paolo
[-- Attachment #2: apx.patch --]
[-- Type: text/x-patch, Size: 6071 bytes --]
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 959fcc01ee0f..9a1766037b6f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -887,6 +887,7 @@ struct kvm_vcpu_arch {
struct fpu_guest guest_fpu;
u64 xcr0;
+ u64 early_xcr0;
u64 guest_supported_xcr0;
u64 ia32_xss;
u64 guest_supported_xss;
@@ -2101,6 +2102,20 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);
int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
+void __kvm_load_guest_apx(struct kvm_vcpu *vcpu);
+static inline void kvm_load_guest_apx(struct kvm_vcpu *vcpu)
+{
+ if (static_cpu_has(X86_FEATURE_APX))
+ __kvm_load_guest_apx(vcpu);
+}
+
+void __kvm_save_guest_apx(struct kvm_vcpu *vcpu);
+static inline void kvm_save_guest_apx(struct kvm_vcpu *vcpu)
+{
+ if (static_cpu_has(X86_FEATURE_APX))
+ __kvm_save_guest_apx(vcpu);
+}
+
int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa,
const void *val, int bytes);
diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h
index 657f5f743ed9..e44cfed94160 100644
--- a/arch/x86/kvm/reverse_cpuid.h
+++ b/arch/x86/kvm/reverse_cpuid.h
@@ -31,6 +31,7 @@
/* Intel-defined sub-features, CPUID level 0x00000007:1 (EDX) */
#define X86_FEATURE_AVX_VNNI_INT8 KVM_X86_FEATURE(CPUID_7_1_EDX, 4)
#define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5)
+#define KVM_X86_FEATURE_APX KVM_X86_FEATURE(CPUID_7_1_EDX, 7)
#define X86_FEATURE_AMX_COMPLEX KVM_X86_FEATURE(CPUID_7_1_EDX, 8)
#define X86_FEATURE_AVX_VNNI_INT16 KVM_X86_FEATURE(CPUID_7_1_EDX, 10)
#define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14)
@@ -151,6 +152,7 @@ static __always_inline u32 __feature_translate(int x86_feature)
KVM_X86_TRANSLATE_FEATURE(TSA_SQ_NO);
KVM_X86_TRANSLATE_FEATURE(TSA_L1_NO);
KVM_X86_TRANSLATE_FEATURE(MSR_IMM);
+ KVM_X86_TRANSLATE_FEATURE(APX);
default:
return x86_feature;
}
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index e6477affac9a..c0a8143f274c 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4359,6 +4359,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
vcpu->arch.host_debugctl != svm->vmcb->save.dbgctl)
update_debugctlmsr(svm->vmcb->save.dbgctl);
+ kvm_load_guest_apx(vcpu);
kvm_wait_lapic_expire(vcpu);
/*
@@ -4381,6 +4382,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
vcpu->arch.regs[VCPU_REGS_RIP] = svm->vmcb->save.rip;
}
+ kvm_save_guest_apx(vcpu);
vcpu->arch.regs_dirty = 0;
if (unlikely(svm->vmcb->control.exit_code == SVM_EXIT_NMI))
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 8b24e682535b..c4c0da9281c1 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -7693,10 +7693,12 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
else if (force_immediate_exit)
smp_send_reschedule(vcpu->cpu);
+ kvm_load_guest_apx(vcpu);
kvm_wait_lapic_expire(vcpu);
/* The actual VMENTER/EXIT is in the .noinstr.text section. */
vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx));
+ kvm_save_guest_apx(vcpu);
/* All fields are clean at this point */
if (kvm_is_using_evmcs()) {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0757b93e528d..69abfdd946dd 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1220,9 +1220,13 @@ static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, bool load_guest)
if (!kvm_is_cr4_bit_set(vcpu, X86_CR4_OSXSAVE))
return;
- if (vcpu->arch.xcr0 != kvm_host.xcr0)
+ /*
+ * Do not load the definitive XCR0 yet; vcpu->arch.early_xcr0 keeps
+ * APX enabled so that the kernel can move to and from r16...r31.
+ */
+ if (vcpu->arch.early_xcr0 != kvm_host.xcr0)
xsetbv(XCR_XFEATURE_ENABLED_MASK,
- load_guest ? vcpu->arch.xcr0 : kvm_host.xcr0);
+ load_guest ? vcpu->arch.early_xcr0 : kvm_host.xcr0);
if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
vcpu->arch.ia32_xss != kvm_host.xss)
@@ -1302,6 +1302,11 @@ int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
vcpu->arch.xcr0 = xcr0;
+ /* APX is needed to save/restore registers for fast path WRMSR. */
+ vcpu->arch.early_xcr0 = xcr0;
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_APX))
+ vcpu->arch.early_xcr0 |= kvm_host.xcr0 & XFEATURE_MASK_APX;
+
if ((xcr0 ^ old_xcr0) & XFEATURE_MASK_EXTEND)
vcpu->arch.cpuid_dynamic_bits_dirty = true;
return 0;
@@ -11056,6 +11061,49 @@ static void kvm_vcpu_reload_apic_access_page(struct kvm_vcpu *vcpu)
kvm_x86_call(set_apic_access_page_addr)(vcpu);
}
+/*
+ * Assuming the kernel does not use APX for now. When
+ * the kernel starts using APX this needs to move into
+ * assembly, and KVM_GET/SET_XSAVE needs to fill in
+ * EGPRs from vcpu->arch.regs.
+ */
+void __kvm_load_guest_apx(struct kvm_vcpu *vcpu)
+{
+ if (vcpu->arch.early_xcr0 != vcpu->arch.xcr0)
+ xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
+
+ if (!(vcpu->arch.xcr0 & XFEATURE_MASK_APX))
+ return;
+
+ WARN_ON_ONCE(!irqs_disabled());
+
+ asm("mov %[r16], %%r16\n"
+ "mov %[r17], %%r17\n" // ...
+ : : [r16] "m" (vcpu->arch.regs[16]),
+ [r17] "m" (vcpu->arch.regs[17]));
+}
+
+/*
+ * Assuming the kernel does not use APX for now. When
+ * the kernel starts using APX this needs to move into
+ * assembly and zero out APX registers for the host.
+ */
+void __kvm_save_guest_apx(struct kvm_vcpu *vcpu)
+{
+ if (vcpu->arch.early_xcr0 != vcpu->arch.xcr0)
+ xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.early_xcr0);
+
+ if (!(vcpu->arch.xcr0 & XFEATURE_MASK_APX))
+ return;
+
+ WARN_ON_ONCE(!irqs_disabled());
+
+ asm("mov %%r16, %[r16]\n"
+ "mov %%r17, %[r17]\n" // ...
+ : : [r16] "m" (vcpu->arch.regs[16]),
+ [r17] "m" (vcpu->arch.regs[17]));
+}
+
/*
* Called within kvm->srcu read side.
* Returns 1 to let vcpu_run() continue the guest execution loop without
^ permalink raw reply related
* Re: [PATCH] KVM: TDX: Fix APIC MSR ranges in tdx_has_emulated_msr()
From: Edgecombe, Rick P @ 2026-04-04 0:11 UTC (permalink / raw)
To: seanjc@google.com
Cc: kvm@vger.kernel.org, Hansen, Dave, Yamahata, Isaku,
dmaluka@chromium.org, x86@kernel.org, kas@kernel.org,
bp@alien8.de, linux-kernel@vger.kernel.org, mingo@redhat.com,
dave.hansen@linux.intel.com, binbin.wu@linux.intel.com,
linux-coco@lists.linux.dev, hpa@zytor.com, tglx@kernel.org,
pbonzini@redhat.com
In-Reply-To: <adBIPdBA-aJWcUrY@google.com>
On Fri, 2026-04-03 at 16:07 -0700, Sean Christopherson wrote:
> > I mean ones where wrmsr is handled by the TDX module instead of generating a
> > #VE that gets morphed into TDVMCALL by the guest. Actually usually called
> > "native", but I just reused your "accelerated" term from the mail.
>
> It's neither. Precision matters here, otherwise I can't follow along.
> Accelerated means the CPU virtualizes it without software involvement. Native
> would mean the guest has direct access to bare metal hardware.
Oh, sorry.
> IIUC, what's happening here is that the TDX-Module is emulating x2APIC stuff.
I'll stick to this language. The tdx docs call them differently of course.
"native" there is about #VE or not. So I can talk about it with respect to VE or
!VE.
>
> > So... "Reduced #VE" (also called "VE reduction") reduces which things cause
> > a #VE. The guest opts into it and the TDX module starts behaving
> > differently. It's kind of grab bag of changes including changing CPUID
> > behavior, which is another wrinkle. It was intended to fixup guest side TDX
> > arch issues.
>
> And KVM has no visilibity into which mode the guest has selected? That's
> awful.
Yea, on both accounts. So where we are at with this is, starting to reject
changes that build on the pattern. We haven't gone so far as to ask for a
feature to notify the host of the guest opt-ins. But I wouldn't say we have a
grand design in mind either. If you have any clarity, please feel free to drop a
quotable.
>
> If KVM has no visiblity, then I don't see an option other than for KVM to
> advertise and emulate what it can at all times, and it becomes the guest's
> responsibility to not screw up. I guess it's not really any different from
> not trying to use MMIO accesses after switching to x2APIC mode.
Like your diff? Expose any MSRs that might be emulated in the TDX paradigm. But
don't expose all MSRs that KVM supports.
^ permalink raw reply
* Re: [PATCH] KVM: TDX: Fix APIC MSR ranges in tdx_has_emulated_msr()
From: Sean Christopherson @ 2026-04-03 23:07 UTC (permalink / raw)
To: Rick P Edgecombe
Cc: linux-coco@lists.linux.dev, Dave Hansen, dmaluka@chromium.org,
bp@alien8.de, kas@kernel.org, x86@kernel.org,
binbin.wu@linux.intel.com, linux-kernel@vger.kernel.org,
dave.hansen@linux.intel.com, tglx@kernel.org, kvm@vger.kernel.org,
hpa@zytor.com, pbonzini@redhat.com, mingo@redhat.com,
Isaku Yamahata
In-Reply-To: <ec581571ab4141ed77e6979b3e03773e301a62c1.camel@intel.com>
On Fri, Apr 03, 2026, Rick P Edgecombe wrote:
> On Fri, 2026-04-03 at 12:07 -0700, Sean Christopherson wrote:
> > > > No? Don't we just want to allow access to MSRs that aren't accelerated?
> > > > What the TDX-Module supports is largely irrelevant, I think.
> > >
> > > Not sure if I might be missing the point here. As above, we don't have
> > > enough info to know which MSRs are accelerated. If the guest enabled #VE
> > > reduction, it changes which ones are accelerated and the VMM is not
> > > notified.
> >
> > What does the "accleration" in that case? Or does it reduce which ones are
> > accelerated?
>
> I mean ones where wrmsr is handled by the TDX module instead of generating a #VE
> that gets morphed into TDVMCALL by the guest. Actually usually called "native",
> but I just reused your "accelerated" term from the mail.
It's neither. Precision matters here, otherwise I can't follow along. Accelerated
means the CPU virtualizes it without software involvement. Native would mean the
guest has direct access to bare metal hardware. IIUC, what's happening here is
that the TDX-Module is emulating x2APIC stuff.
> So... "Reduced #VE" (also called "VE reduction") reduces which things cause a
> #VE. The guest opts into it and the TDX module starts behaving differently. It's
> kind of grab bag of changes including changing CPUID behavior, which is another
> wrinkle. It was intended to fixup guest side TDX arch issues.
And KVM has no visilibity into which mode the guest has selected? That's awful.
If KVM has no visiblity, then I don't see an option other than for KVM to advertise
and emulate what it can at all times, and it becomes the guest's responsibility
to not screw up. I guess it's not really any different from not trying to use
MMIO accesses after switching to x2APIC mode.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox