* Re: [PATCH v5 19/22] x86/virt/tdx: Enable TDX module runtime updates
From: Kiryl Shutsemau @ 2026-03-19 14:03 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-20-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:39AM -0700, Chao Gao wrote:
> All pieces of TDX module runtime updates are in place. Enable it if it
> is supported.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 18/22] coco/tdx-host: Don't expose P-SEAMLDR features on CPUs with erratum
From: Kiryl Shutsemau @ 2026-03-19 14:02 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, kvm, linux-coco, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-19-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:38AM -0700, Chao Gao wrote:
> Some TDX-capable CPUs have an erratum, as documented in Intel® Trust
> Domain CPU Architectural Extensions (May 2021 edition) Chapter 2.3:
>
> SEAMRET from the P-SEAMLDR clears the current VMCS structure pointed
> to by the current-VMCS pointer. A VMM that invokes the P-SEAMLDR using
> SEAMCALL must reload the current-VMCS, if required, using the VMPTRLD
> instruction.
>
> Clearing the current VMCS behind KVM's back will break KVM.
>
> This erratum is not present when IA32_VMX_BASIC[60] is set. Add a CPU
> bug bit for this erratum and refuse to expose P-SEAMLDR features (e.g.,
> TDX module updates) on affected CPUs.
>
> == Alternatives ==
> Two workarounds were considered but both were rejected:
>
> 1. Save/restore the current VMCS around P-SEAMLDR calls. This produces ugly
> assembly code [1] and doesn't play well with #MCE or #NMI if they
> need to use the current VMCS.
>
> 2. Move KVM's VMCS tracking logic to the TDX core code, which would break
> the boundary between KVM and the TDX core code [2].
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
> Link: https://lore.kernel.org/kvm/fedb3192-e68c-423c-93b2-a4dc2f964148@intel.com/ # [1]
> Link: https://lore.kernel.org/kvm/aYIXFmT-676oN6j0@google.com/ # [2]
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 17/22] x86/virt/tdx: Avoid updates during update-sensitive operations
From: Kiryl Shutsemau @ 2026-03-19 14:00 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-18-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:37AM -0700, Chao Gao wrote:
> diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
> index b3a7301e77c6..4c4f7acd4044 100644
> --- a/arch/x86/include/asm/tdx.h
> +++ b/arch/x86/include/asm/tdx.h
> @@ -26,11 +26,18 @@
> #define TDX_SEAMCALL_GP (TDX_SW_ERROR | X86_TRAP_GP)
> #define TDX_SEAMCALL_UD (TDX_SW_ERROR | X86_TRAP_UD)
>
> +#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL
> +
> /*
> * TDX module SEAMCALL leaf function error codes
> */
> -#define TDX_SUCCESS 0ULL
> -#define TDX_RND_NO_ENTROPY 0x8000020300000000ULL
> +#define TDX_SUCCESS 0ULL
> +#define TDX_RND_NO_ENTROPY 0x8000020300000000ULL
> +#define TDX_UPDATE_COMPAT_SENSITIVE 0x8000051200000000ULL
This competes with other patchset[1].
[1] https://lore.kernel.org/all/20260307010358.819645-1-rick.p.edgecombe@intel.com
> @@ -1189,9 +1192,21 @@ int tdx_module_shutdown(void)
> * modules as new modules likely have higher handoff version.
> */
> args.rcx = tdx_sysinfo.handoff.module_hv;
> - ret = seamcall_prerr(TDH_SYS_SHUTDOWN, &args);
> - if (ret)
> - return ret;
> +
> + if (tdx_supports_update_compatibility(&tdx_sysinfo))
> + args.rcx |= TDX_SYS_SHUTDOWN_AVOID_COMPAT_SENSITIVE;
Hm. So what happens if the module doesn't support it? We just ignore
problem?
Maybe we should just block updates on such modules?
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 16/22] x86/virt/tdx: Update tdx_sysinfo and check features post-update
From: Kiryl Shutsemau @ 2026-03-19 13:43 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-17-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:36AM -0700, Chao Gao wrote:
> tdx_sysinfo contains all metadata of the active TDX module, including
> versions, supported features, and TDMR/TDCS/TDVPS information etc. These
> values may change over updates. Blindly refreshing the entire tdx_sysinfo
> could disrupt running software, as it may subtly rely on the previous state
> unless proven otherwise.
>
> Adopt a conservative approach, like microcode updates, by only refreshing
> version information that does not affect functionality, while ignoring
> all other changes. This is acceptable as new modules are required to
> maintain backward compatibility.
>
> Any updates to metadata beyond versions should be justified and reviewed on
> a case-by-case basis.
>
> Note that preallocating a tdx_sys_info buffer before updates is to avoid
> having to handle -ENOMEM when updating tdx_sysinfo after a successful
> update.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
> Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 15/22] x86/virt/tdx: Restore TDX module state
From: Kiryl Shutsemau @ 2026-03-19 13:37 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-16-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:35AM -0700, Chao Gao wrote:
> TDX module state was packed as handoff data during module shutdown. After
> per-CPU initialization, the new module can restore TDX module state from
> handoff data to preserve running TDs.
>
> Once the restoration is done, the TDX module update is complete, which
> means the new module is ready to handle requests from the host and guests.
>
> Implement the new TDH.SYS.UPDATE SEAMCALL to restore TDX module state
> and invoke it on one CPU since it only needs to be called once.
>
> Note that Intel® Trust Domain Extensions (Intel® TDX) Module Base
> Architecture Specification, Revision 348549-007, Chapter 4.5.5 states:
>
> If TDH.SYS.UPDATE returns an error, then the host VMM can continue
> with the non-update sequence (TDH.SYS.CONFIG, 15 TDH.SYS.KEY.CONFIG
> etc.). In this case all existing TDs are lost. Alternatively, the host
> VMM can request the P-SEAMLDR to update to another TDX module. If that
> update is successful, existing TDs are preserved
>
> Don't implement the two alternatives due to their complexity and unclear
> benefits.
Make it clear which option you've took.
>
> Also note that the location and the format of handoff data is defined by
> the TDX module. The new module knows where to get handoff data and how
> to parse it. The kernel doesn't need to provide its location, format etc.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 14/22] x86/virt/seamldr: Do TDX per-CPU initialization after updates
From: Kiryl Shutsemau @ 2026-03-19 13:33 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-15-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:34AM -0700, Chao Gao wrote:
> After installing the new TDX module, each CPU needs to be initialized
> again to make the CPU ready to run any other SEAMCALLs. So, call
> tdx_cpu_enable() on all CPUs.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 13/22] x86/virt/seamldr: Install a new TDX module
From: Kiryl Shutsemau @ 2026-03-19 13:32 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-14-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:33AM -0700, Chao Gao wrote:
> @@ -225,6 +227,7 @@ static void ack_state(void)
> static int do_seamldr_install_module(void *seamldr_params)
> {
> enum module_update_state newstate, curstate = MODULE_UPDATE_START;
> + struct tdx_module_args args = {};
> int cpu = smp_processor_id();
> bool primary;
> int ret = 0;
> @@ -243,6 +246,10 @@ static int do_seamldr_install_module(void *seamldr_params)
> if (primary)
> ret = tdx_module_shutdown();
> break;
> + case MODULE_UPDATE_CPU_INSTALL:
> + args.rcx = __pa(seamldr_params);
> + ret = seamldr_call(P_SEAMLDR_INSTALL, &args);
I think it should be in a helper alongside with the tdx_module_args.
> + break;
> default:
> break;
> }
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 11/22] x86/virt/seamldr: Shut down the current TDX module
From: Kiryl Shutsemau @ 2026-03-19 13:24 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-12-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:31AM -0700, Chao Gao wrote:
> The first step of TDX module updates is shutting down the current TDX
> Module. This step also packs state information that needs to be
> preserved across updates as handoff data, which will be consumed by the
> updated module. The handoff data is stored internally in the SEAM range
> and is hidden from the kernel.
>
> To ensure a successful update, the new module must be able to consume
> the handoff data generated by the old module. Since handoff data layout
> may change between modules, the handoff data is versioned. Each module
> has a native handoff version and provides backward support for several
> older versions.
>
> The complete handoff versioning protocol is complex as it supports both
> module upgrades and downgrades. See details in Intel® Trust Domain
> Extensions (Intel® TDX) Module Base Architecture Specification, Revision
> 348549-007, Chapter 4.5.3 "Handoff Versioning".
>
> Ideally, the kernel needs to retrieve the handoff versions supported by
> the current module and the new module and select a version supported by
> both. But, since this implementation chooses to only support module
> upgrades, simply request the current module to generate handoff data
> using its highest supported version, expecting that the new module will
> likely support it.
>
> Note that only one CPU needs to call the TDX module's shutdown API.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Kai Huang <kai.huang@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Nit is below.
> @@ -100,6 +100,17 @@ static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf
> return ret;
> }
>
> +static int get_tdx_sys_info_handoff(struct tdx_sys_info_handoff *sysinfo_handoff)
> +{
> + int ret = 0;
> + u64 val;
> +
> + if (!ret && !(ret = read_sys_metadata_field(0x8900000100000000, &val)))
!ret check is redundant as well as the ret initialization above.
> + sysinfo_handoff->module_hv = val;
> +
> + return ret;
> +}
> +
> static int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
> {
> int ret = 0;
> @@ -116,5 +127,12 @@ static int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 10/22] x86/virt/seamldr: Abort updates if errors occurred midway
From: Kiryl Shutsemau @ 2026-03-19 13:19 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-11-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:30AM -0700, Chao Gao wrote:
> diff --git a/arch/x86/virt/vmx/tdx/seamldr.c b/arch/x86/virt/vmx/tdx/seamldr.c
> index 978fcca92128..e195703398e7 100644
> --- a/arch/x86/virt/vmx/tdx/seamldr.c
> +++ b/arch/x86/virt/vmx/tdx/seamldr.c
> @@ -190,6 +190,7 @@ enum module_update_state {
> static struct {
> enum module_update_state state;
> int thread_ack;
> + int failed;
bool is enough, right?
> /*
> * Protect update_data. Raw spinlock as it will be acquired from
> * interrupt-disabled contexts.
> @@ -237,12 +238,17 @@ static int do_seamldr_install_module(void *seamldr_params)
> break;
> }
>
> - ack_state();
> + if (ret) {
> + scoped_guard(raw_spinlock, &update_data.lock)
> + update_data.failed++;
I don't see a reason in spin lock here. Just WRITE_ONCE() would work
fine.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 09/22] x86/virt/seamldr: Introduce skeleton for TDX module updates
From: Kiryl Shutsemau @ 2026-03-19 13:15 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <20260315135920.354657-10-chao.gao@intel.com>
On Sun, Mar 15, 2026 at 06:58:29AM -0700, Chao Gao wrote:
> TDX module updates require careful synchronization with other TDX
> operations. The requirements are (#1/#2 reflect current behavior that
> must be preserved):
>
> 1. SEAMCALLs need to be callable from both process and IRQ contexts.
> 2. SEAMCALLs need to be able to run concurrently across CPUs
> 3. During updates, only update-related SEAMCALLs are permitted; all
> other SEAMCALLs shouldn't be called.
> 4. During updates, all online CPUs must participate in the update work.
>
> No single lock primitive satisfies all requirements. For instance,
> rwlock_t handles #1/#2 but fails #4: CPUs spinning with IRQs disabled
> cannot be directed to perform update work.
>
> Use stop_machine() as it is the only well-understood mechanism that can
> meet all requirements.
>
> And TDX module updates consist of several steps (See Intel® Trust Domain
> Extensions (Intel® TDX) Module Base Architecture Specification, Revision
> 348549-007, Chapter 4.5 "TD-Preserving TDX module Update"). Ordering
> requirements between steps mandate lockstep synchronization across all
> CPUs.
>
> multi_cpu_stop() is a good example of performing a multi-step task in
> lockstep. But it doesn't synchronize steps within the callback function
> it takes. So, implement one based on its pattern to establish the
> skeleton for TDX module updates. Specifically, add a global state
> machine where each state represents a step in the update flow. The state
> advances only after all CPUs acknowledge completing their work in the
> current state. This acknowledgment mechanism is what ensures lockstep
> execution.
>
> Potential alternative to stop_machine()
> =======================================
> An alternative approach is to lock all KVM entry points and kick all
> vCPUs. Here, KVM entry points refer to KVM VM/vCPU ioctl entry points,
> implemented in KVM common code (virt/kvm). Adding a locking mechanism
> there would affect all architectures KVM supports. And to lock only TDX
> vCPUs, new logic would be needed to identify TDX vCPUs, which the KVM
> common code currently lacks. This would add significant complexity and
> maintenance overhead to KVM for this TDX-specific use case.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
One question below.
> +static struct {
> + enum module_update_state state;
> + int thread_ack;
> + /*
> + * Protect update_data. Raw spinlock as it will be acquired from
> + * interrupt-disabled contexts.
> + */
> + raw_spinlock_t lock;
> +} update_data = {
> + .lock = __RAW_SPIN_LOCK_UNLOCKED(update_data.lock)
> +};
multi_stop_cpu() used atomic_t for thread_ack insead of spinlock.
Any particular reason you took different direction?
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v5 08/22] x86/virt/seamldr: Allocate and populate a module update request
From: Kiryl Shutsemau @ 2026-03-19 12:58 UTC (permalink / raw)
To: Chao Gao
Cc: linux-kernel, linux-coco, kvm, binbin.wu, dan.j.williams,
dave.hansen, ira.weiny, kai.huang, nik.borisov, paulmck, pbonzini,
reinette.chatre, rick.p.edgecombe, sagis, seanjc, tony.lindgren,
vannapurve, vishal.l.verma, yilun.xu, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, x86, H. Peter Anvin
In-Reply-To: <abukxFDD3g0sKYXP@intel.com>
On Thu, Mar 19, 2026 at 03:24:52PM +0800, Chao Gao wrote:
> On Wed, Mar 18, 2026 at 10:58:44AM +0000, Kiryl Shutsemau wrote:
> >On Wed, Mar 18, 2026 at 04:50:34PM +0800, Chao Gao wrote:
> >> >> + ptr = sig;
> >> >> + for (i = 0; i < sig_size / SZ_4K; i++) {
> >> >> + /*
> >> >> + * Don't assume @sig is page-aligned although it is 4KB-aligned.
> >> >> + * Always add the in-page offset to get the physical address.
> >> >> + */
> >> >
> >> >I don't follow this. If @sig is 4k aligned in VA, it is page aligned.
> >>
> >> Dan's concern was that PAGE_SIZE is not guaranteed to be 4096.
> >>
> >> I agree that PAGE_SIZE is 4K on x86 today. But to address that concern, I saw
> >> two options:
> >>
> >> 1. Add WARN_ON_ONCE(PAGE_SIZE != SZ_4K), or
> >> 2. Handle it as in the code above.
> >>
> >> I didn't find existing code using option 1 in x86, so I chose option 2.
> >
> >Please, make it clear in the comment that you are talking about PAGE_SIZE != SZ_4K.
>
> Sure. How about:
>
> /*
> * @sig is 4KB-aligned, but that does not imply PAGE_SIZE
> * alignment when PAGE_SIZE != SZ_4K. Always include the
> * in-page offset.
> */
Ack.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Kiryl Shutsemau @ 2026-03-19 12:57 UTC (permalink / raw)
To: Yan Zhao
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, x86, linux-kernel,
kvm, linux-coco, kai.huang, rick.p.edgecombe, yilun.xu,
vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <abvlOcnY4assk8DW@yzhao56-desk.sh.intel.com>
On Thu, Mar 19, 2026 at 07:59:53PM +0800, Yan Zhao wrote:
> On Thu, Mar 19, 2026 at 10:39:14AM +0000, Kiryl Shutsemau wrote:
> > On Thu, Mar 19, 2026 at 08:57:03AM +0800, Yan Zhao wrote:
> > > @@ -1639,16 +1644,17 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page)
> > > }
> > > EXPORT_SYMBOL_FOR_KVM(tdh_vp_addcx);
> > >
> > > -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2)
> > > +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, kvm_pfn_t pfn,
> > > + u64 *ext_err1, u64 *ext_err2)
> > > {
> > > struct tdx_module_args args = {
> > > .rcx = gpa | level,
> > > .rdx = tdx_tdr_pa(td),
> > > - .r8 = page_to_phys(page),
> > > + .r8 = PFN_PHYS(pfn),
> > > };
> > > u64 ret;
> > >
> > > - tdx_clflush_page(page);
> > > + tdx_clflush_pfn(pfn);
> >
> > This is pre-existing problem, but shouldn't we respect @level here?
> > Flush size need to take page size into account.
> Hmm, flush size is fixed to PAGE_SIZE, because this series is based on the
> upstream code where huge page is not supported, so there's
> "if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))" in KVM.
>
> Though tdh_mem_page_aug() is an API, it is currently only exported to KVM and
> uses type kvm_pfn_t. So, is it still acceptable to assume flush size to be
> PAGE_SIZE? Honoring level will soon be introduced by huge page patches.
It caught my eye because previously size to flush was passed down to
tdx_clflush_page() in the struct page (although never used there).
With switching to pfn, we give up this information and it has to be
passed separately. It would be easy to miss that in huge page patches,
if we don't pass down level here.
>
> If you think it needs to be fixed before huge page series, what about fixing it
> in a separate cleanup patch? IMO, it would be better placed after Sean's cleanup
> patch [1], so we can use page_level_size() instead of inventing the wheel.
I am okay with a separate patch.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Yan Zhao @ 2026-03-19 12:14 UTC (permalink / raw)
To: Kiryl Shutsemau, seanjc, pbonzini, dave.hansen, tglx, mingo, bp,
x86, linux-kernel, kvm, linux-coco, kai.huang, rick.p.edgecombe,
yilun.xu, vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <abvlOcnY4assk8DW@yzhao56-desk.sh.intel.com>
On Thu, Mar 19, 2026 at 07:59:53PM +0800, Yan Zhao wrote:
> On Thu, Mar 19, 2026 at 10:39:14AM +0000, Kiryl Shutsemau wrote:
> > On Thu, Mar 19, 2026 at 08:57:03AM +0800, Yan Zhao wrote:
> > > @@ -1639,16 +1644,17 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page)
> > > }
> > > EXPORT_SYMBOL_FOR_KVM(tdh_vp_addcx);
> > >
> > > -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2)
> > > +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, kvm_pfn_t pfn,
> > > + u64 *ext_err1, u64 *ext_err2)
> > > {
> > > struct tdx_module_args args = {
> > > .rcx = gpa | level,
> > > .rdx = tdx_tdr_pa(td),
> > > - .r8 = page_to_phys(page),
> > > + .r8 = PFN_PHYS(pfn),
> > > };
> > > u64 ret;
> > >
> > > - tdx_clflush_page(page);
> > > + tdx_clflush_pfn(pfn);
> >
> > This is pre-existing problem, but shouldn't we respect @level here?
> > Flush size need to take page size into account.
> Hmm, flush size is fixed to PAGE_SIZE, because this series is based on the
> upstream code where huge page is not supported, so there's
> "if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))" in KVM.
>
> Though tdh_mem_page_aug() is an API, it is currently only exported to KVM and
> uses type kvm_pfn_t. So, is it still acceptable to assume flush size to be
> PAGE_SIZE? Honoring level will soon be introduced by huge page patches.
>
> If you think it needs to be fixed before huge page series, what about fixing it
> in a separate cleanup patch? IMO, it would be better placed after Sean's cleanup
> patch [1], so we can use page_level_size() instead of inventing the wheel.
BTW, the cleanup patch would then essentially look like the one in the huge page
series [2]...
[2] https://lore.kernel.org/kvm/20260129011517.3545883-27-seanjc@google.com/
So, if a cleanup patch before the huge page series is required, maybe just
adding WARN_ON_ONCE(level != PG_LEVEL_4K) in that patch?
> [1] https://lore.kernel.org/kvm/20260129011517.3545883-2-seanjc@google.com/
^ permalink raw reply
* Re: [PATCH v13 07/48] arm64: RMI: Check for RMI support at KVM init
From: Steven Price @ 2026-03-19 12:47 UTC (permalink / raw)
To: Suzuki K Poulose, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Zenghui Yu, linux-arm-kernel, linux-kernel,
Joey Gouly, Alexandru Elisei, Christoffer Dall, Fuad Tabba,
linux-coco, Ganapatrao Kulkarni, Gavin Shan, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <8fd56e84-821a-47c7-8735-928057a7a653@arm.com>
On 19/03/2026 10:38, Suzuki K Poulose wrote:
> On 18/03/2026 15:53, Steven Price wrote:
>> Query the RMI version number and check if it is a compatible version. A
>> static key is also provided to signal that a supported RMM is available.
>>
>> Functions are provided to query if a VM or VCPU is a realm (or rec)
>> which currently will always return false.
>>
>> Later patches make use of struct realm and the states as the ioctls
>> interfaces are added to support realm and REC creation and destruction.
>>
>> Signed-off-by: Steven Price <steven.price@arm.com>
>> ---
>> Changes since v12:
>> * Drop check for 4k page size.
>> Changes since v11:
>> * Reword slightly the comments on the realm states.
>> Changes since v10:
>> * kvm_is_realm() no longer has a NULL check.
>> * Rename from "rme" to "rmi" when referring to the RMM interface.
>> * Check for RME (hardware) support before probing for RMI support.
>> Changes since v8:
>> * No need to guard kvm_init_rme() behind 'in_hyp_mode'.
>> Changes since v6:
>> * Improved message for an unsupported RMI ABI version.
>> Changes since v5:
>> * Reword "unsupported" message from "host supports" to "we want" to
>> clarify that 'we' are the 'host'.
>> Changes since v2:
>> * Drop return value from kvm_init_rme(), it was always 0.
>> * Rely on the RMM return value to identify whether the RSI ABI is
>> compatible.
>> ---
>> arch/arm64/include/asm/kvm_emulate.h | 18 +++++++++
>> arch/arm64/include/asm/kvm_host.h | 4 ++
>> arch/arm64/include/asm/kvm_rmi.h | 56 +++++++++++++++++++++++++++
>> arch/arm64/include/asm/virt.h | 1 +
>> arch/arm64/kernel/cpufeature.c | 1 +
>> arch/arm64/kvm/Makefile | 2 +-
>> arch/arm64/kvm/arm.c | 5 +++
>> arch/arm64/kvm/rmi.c | 57 ++++++++++++++++++++++++++++
>> 8 files changed, 143 insertions(+), 1 deletion(-)
>> create mode 100644 arch/arm64/include/asm/kvm_rmi.h
>> create mode 100644 arch/arm64/kvm/rmi.c
>>
>> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/
>> include/asm/kvm_emulate.h
>> index 5bf3d7e1d92c..f38b50151ce8 100644
>> --- a/arch/arm64/include/asm/kvm_emulate.h
>> +++ b/arch/arm64/include/asm/kvm_emulate.h
>> @@ -688,4 +688,22 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu
>> *vcpu)
>> vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
>> }
>> }
>
>
> --8>---
>
>> +
>> +static inline bool kvm_is_realm(struct kvm *kvm)
>> +{
>> + if (static_branch_unlikely(&kvm_rmi_is_available))
>> + return kvm->arch.is_realm;
>> + return false;
>> +}
>> +
>> +static inline enum realm_state kvm_realm_state(struct kvm *kvm)
>> +{
>> + return READ_ONCE(kvm->arch.realm.state);
>> +}
>> +
>> +static inline bool vcpu_is_rec(struct kvm_vcpu *vcpu)
>> +{
>> + return false;
>> +}
>> +
>> #endif /* __ARM64_KVM_EMULATE_H__ */
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/
>> asm/kvm_host.h
>> index 5d5a3bbdb95e..9267a2f2d65b 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -27,6 +27,7 @@
>> #include <asm/fpsimd.h>
>> #include <asm/kvm.h>
>> #include <asm/kvm_asm.h>
>> +#include <asm/kvm_rmi.h>
>> #include <asm/vncr_mapping.h>
>> #define __KVM_HAVE_ARCH_INTC_INITIALIZED
>> @@ -405,6 +406,9 @@ struct kvm_arch {
>> * the associated pKVM instance in the hypervisor.
>> */
>> struct kvm_protected_vm pkvm;
>> +
>> + bool is_realm;
>> + struct realm realm;
>> };
>> struct kvm_vcpu_fault_info {
>> diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/
>> asm/kvm_rmi.h
>> new file mode 100644
>> index 000000000000..3506f50b05cd
>> --- /dev/null
>> +++ b/arch/arm64/include/asm/kvm_rmi.h
>> @@ -0,0 +1,56 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright (C) 2023-2025 ARM Ltd.
>> + */
>> +
>> +#ifndef __ASM_KVM_RMI_H
>> +#define __ASM_KVM_RMI_H
>> +
>> +/**
>> + * enum realm_state - State of a Realm
>> + */
>> +enum realm_state {
>> + /**
>> + * @REALM_STATE_NONE:
>> + * Realm has not yet been created. rmi_realm_create() has not
>> + * yet been called.
>> + */
>> + REALM_STATE_NONE,
>> + /**
>> + * @REALM_STATE_NEW:
>> + * Realm is under construction, rmi_realm_create() has been
>> + * called, but it is not yet activated. Pages may be populated.
>> + */
>> + REALM_STATE_NEW,
>> + /**
>> + * @REALM_STATE_ACTIVE:
>> + * Realm has been created and is eligible for execution with
>> + * rmi_rec_enter(). Pages may no longer be populated with
>> + * rmi_data_create().
>> + */
>> + REALM_STATE_ACTIVE,
>> + /**
>> + * @REALM_STATE_DYING:
>> + * Realm is in the process of being destroyed or has already
>> been
>> + * destroyed.
>> + */
>> + REALM_STATE_DYING,
>> + /**
>> + * @REALM_STATE_DEAD:
>> + * Realm has been destroyed.
>> + */
>> + REALM_STATE_DEAD
>> +};
>> +
>> +/**
>> + * struct realm - Additional per VM data for a Realm
>> + *
>> + * @state: The lifetime state machine for the realm
>> + */
>> +struct realm {
>> + enum realm_state state;
>> +};
>> +
>
>
> --8<---
>
> Minor nit: The above looks out of place in this patch. Could we
> move it to where this may be actually used ?
Yes, good point. This can be moved to patch 12 quite easily. I think
originally I'd needed this earlier on, but the code's moved on.
Thanks,
Steve
> Rest looks good to me.
>
>
>> +void kvm_init_rmi(void);
>> +
>> +#endif /* __ASM_KVM_RMI_H */
>> diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/
>> virt.h
>> index b51ab6840f9c..dc9b2899e0b2 100644
>> --- a/arch/arm64/include/asm/virt.h
>> +++ b/arch/arm64/include/asm/virt.h
>> @@ -87,6 +87,7 @@ void __hyp_reset_vectors(void);
>> bool is_kvm_arm_initialised(void);
>> DECLARE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
>> +DECLARE_STATIC_KEY_FALSE(kvm_rmi_is_available);
>> static inline bool is_pkvm_initialized(void)
>> {
>> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/
>> cpufeature.c
>> index c31f8e17732a..ddf7e57f23e8 100644
>> --- a/arch/arm64/kernel/cpufeature.c
>> +++ b/arch/arm64/kernel/cpufeature.c
>> @@ -289,6 +289,7 @@ static const struct arm64_ftr_bits
>> ftr_id_aa64isar3[] = {
>> static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = {
>> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_CSV3_SHIFT, 4, 0),
>> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_CSV2_SHIFT, 4, 0),
>> + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_RME_SHIFT, 4, 0),
>> ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_DIT_SHIFT, 4, 0),
>> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_AMU_SHIFT, 4, 0),
>> ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE,
>> ID_AA64PFR0_EL1_MPAM_SHIFT, 4, 0),
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index 3ebc0570345c..e17c4077d8e7 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -16,7 +16,7 @@ CFLAGS_handle_exit.o += -Wno-override-init
>> kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
>> inject_fault.o va_layout.o handle_exit.o config.o \
>> guest.o debug.o reset.o sys_regs.o stacktrace.o \
>> - vgic-sys-reg-v3.o fpsimd.o pkvm.o \
>> + vgic-sys-reg-v3.o fpsimd.o pkvm.o rmi.o \
>> arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
>> vgic/vgic.o vgic/vgic-init.o \
>> vgic/vgic-irqfd.o vgic/vgic-v2.o \
>> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
>> index 29f0326f7e00..274d7866efdc 100644
>> --- a/arch/arm64/kvm/arm.c
>> +++ b/arch/arm64/kvm/arm.c
>> @@ -39,6 +39,7 @@
>> #include <asm/kvm_nested.h>
>> #include <asm/kvm_pkvm.h>
>> #include <asm/kvm_ptrauth.h>
>> +#include <asm/kvm_rmi.h>
>> #include <asm/sections.h>
>> #include <asm/stacktrace/nvhe.h>
>> @@ -104,6 +105,8 @@ long kvm_get_cap_for_kvm_ioctl(unsigned int
>> ioctl, long *ext)
>> return -EINVAL;
>> }
>> +DEFINE_STATIC_KEY_FALSE(kvm_rmi_is_available);
>> +
>> DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector);
>> DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
>> @@ -2921,6 +2924,8 @@ static __init int kvm_arm_init(void)
>> in_hyp_mode = is_kernel_in_hyp_mode();
>> + kvm_init_rmi();
>> +
>> if (cpus_have_final_cap(ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) ||
>> cpus_have_final_cap(ARM64_WORKAROUND_1508412))
>> kvm_info("Guests without required CPU erratum workarounds
>> can deadlock system!\n" \
>> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
>> new file mode 100644
>> index 000000000000..fac151580c01
>> --- /dev/null
>> +++ b/arch/arm64/kvm/rmi.c
>> @@ -0,0 +1,57 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (C) 2023-2025 ARM Ltd.
>> + */
>> +
>> +#include <linux/kvm_host.h>
>> +
>> +#include <asm/rmi_cmds.h>
>> +#include <asm/virt.h>
>> +
>> +static int rmi_check_version(void)
>> +{
>> + struct arm_smccc_res res;
>> + unsigned short version_major, version_minor;
>> + unsigned long host_version = RMI_ABI_VERSION(RMI_ABI_MAJOR_VERSION,
>> + RMI_ABI_MINOR_VERSION);
>> + unsigned long aa64pfr0 =
>> read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
>> +
>> + /* If RME isn't supported, then RMI can't be */
>> + if (cpuid_feature_extract_unsigned_field(aa64pfr0,
>> ID_AA64PFR0_EL1_RME_SHIFT) == 0)
>> + return -ENXIO;
>> +
>> + arm_smccc_1_1_invoke(SMC_RMI_VERSION, host_version, &res);
>> +
>> + if (res.a0 == SMCCC_RET_NOT_SUPPORTED)
>> + return -ENXIO;
>> +
>> + version_major = RMI_ABI_VERSION_GET_MAJOR(res.a1);
>> + version_minor = RMI_ABI_VERSION_GET_MINOR(res.a1);
>> +
>> + if (res.a0 != RMI_SUCCESS) {
>> + unsigned short high_version_major, high_version_minor;
>> +
>> + high_version_major = RMI_ABI_VERSION_GET_MAJOR(res.a2);
>> + high_version_minor = RMI_ABI_VERSION_GET_MINOR(res.a2);
>> +
>> + kvm_err("Unsupported RMI ABI (v%d.%d - v%d.%d) we want v%d.
>> %d\n",
>> + version_major, version_minor,
>> + high_version_major, high_version_minor,
>> + RMI_ABI_MAJOR_VERSION,
>> + RMI_ABI_MINOR_VERSION);
>> + return -ENXIO;
>> + }
>> +
>> + kvm_info("RMI ABI version %d.%d\n", version_major, version_minor);
>> +
>> + return 0;
>> +}
>> +
>> +void kvm_init_rmi(void)
>> +{
>> + /* Continue without realm support if we can't agree on a version */
>> + if (rmi_check_version())
>> + return;
>> +
>> + /* Future patch will enable static branch kvm_rmi_is_available */
>> +}
>
^ permalink raw reply
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Yan Zhao @ 2026-03-19 11:59 UTC (permalink / raw)
To: Kiryl Shutsemau
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, x86, linux-kernel,
kvm, linux-coco, kai.huang, rick.p.edgecombe, yilun.xu,
vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <abvR2A19S4ZtpAli@thinkstation>
On Thu, Mar 19, 2026 at 10:39:14AM +0000, Kiryl Shutsemau wrote:
> On Thu, Mar 19, 2026 at 08:57:03AM +0800, Yan Zhao wrote:
> > @@ -1639,16 +1644,17 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page)
> > }
> > EXPORT_SYMBOL_FOR_KVM(tdh_vp_addcx);
> >
> > -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2)
> > +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, kvm_pfn_t pfn,
> > + u64 *ext_err1, u64 *ext_err2)
> > {
> > struct tdx_module_args args = {
> > .rcx = gpa | level,
> > .rdx = tdx_tdr_pa(td),
> > - .r8 = page_to_phys(page),
> > + .r8 = PFN_PHYS(pfn),
> > };
> > u64 ret;
> >
> > - tdx_clflush_page(page);
> > + tdx_clflush_pfn(pfn);
>
> This is pre-existing problem, but shouldn't we respect @level here?
> Flush size need to take page size into account.
Hmm, flush size is fixed to PAGE_SIZE, because this series is based on the
upstream code where huge page is not supported, so there's
"if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))" in KVM.
Though tdh_mem_page_aug() is an API, it is currently only exported to KVM and
uses type kvm_pfn_t. So, is it still acceptable to assume flush size to be
PAGE_SIZE? Honoring level will soon be introduced by huge page patches.
If you think it needs to be fixed before huge page series, what about fixing it
in a separate cleanup patch? IMO, it would be better placed after Sean's cleanup
patch [1], so we can use page_level_size() instead of inventing the wheel.
[1] https://lore.kernel.org/kvm/20260129011517.3545883-2-seanjc@google.com/
^ permalink raw reply
* Re: [PATCH 2/2] x86/virt/tdx: Use PFN directly for unmapping guest private memory
From: Kiryl Shutsemau @ 2026-03-19 10:48 UTC (permalink / raw)
To: Yan Zhao
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, x86, linux-kernel,
kvm, linux-coco, kai.huang, rick.p.edgecombe, yilun.xu,
vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <20260319005808.9013-1-yan.y.zhao@intel.com>
On Thu, Mar 19, 2026 at 08:58:08AM +0800, Yan Zhao wrote:
> @@ -1817,11 +1817,11 @@ static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
> if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
> return;
>
> - err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
> + err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, pfn);
> if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
> return;
>
> - tdx_quirk_reset_page(page);
> + tdx_quirk_reset_page(pfn);
> }
>
> void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
The same problem. @level is ignored.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v13 10/48] arm64: RMI: Ensure that the RMM has GPT entries for memory
From: Suzuki K Poulose @ 2026-03-19 10:41 UTC (permalink / raw)
To: Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Zenghui Yu, linux-arm-kernel, linux-kernel,
Joey Gouly, Alexandru Elisei, Christoffer Dall, Fuad Tabba,
linux-coco, Ganapatrao Kulkarni, Gavin Shan, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-11-steven.price@arm.com>
On 18/03/2026 15:53, Steven Price wrote:
> The RMM may not be tracking all the memory of the system at boot. Create
> the necessary tracking state and GPTs within the RMM so that all boot
> memory can be delegated to the RMM as needed during runtime.
>
> Note: support is currently missing for SROs which means that if the RMM
> needs memory donating this will fail (and render CCA unusable in Linux).
>
> Signed-off-by: Steven Price <steven.price@arm.com>
> ---
> New patch for v13
> ---
> arch/arm64/kvm/rmi.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 89 insertions(+)
>
> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 9590dff9a2c1..80aedc85e94a 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c
> @@ -4,6 +4,7 @@
> */
>
> #include <linux/kvm_host.h>
> +#include <linux/memblock.h>
>
> #include <asm/kvm_pgtable.h>
> #include <asm/rmi_cmds.h>
> @@ -56,6 +57,18 @@ static int rmi_check_version(void)
> return 0;
> }
>
> +/*
> + * These are the 'default' sizes when passing 0 as the tracking_region_size.
This is a little bit vague. Should we explicitly mention :
"For now we set the tracking_region_size to 0 for RMI_RMM_CONFIG_SET()"
> + * TODO: Support other granule sizes
nit: s/granule/Tracking/
Suzuki
> + */
> +#ifdef CONFIG_PAGE_SIZE_4KB
> +#define RMM_GRANULE_TRACKING_SIZE SZ_1G
> +#elif defined(CONFIG_PAGE_SIZE_16KB)
> +#define RMM_GRANULE_TRACKING_SIZE SZ_32M
> +#elif defined(CONFIG_PAGE_SIZE_64KB)
> +#define RMM_GRANULE_TRACKING_SIZE SZ_512M
> +#endif
> +
> static int rmi_configure(void)
> {
> struct rmm_config *config __free(free_page) = NULL;
> @@ -95,6 +108,80 @@ static int rmi_configure(void)
> return 0;
> }
>
> +static int rmi_verify_memory_tracking(phys_addr_t start, phys_addr_t end)
> +{
> + start = ALIGN_DOWN(start, RMM_GRANULE_TRACKING_SIZE);
> + end = ALIGN(end, RMM_GRANULE_TRACKING_SIZE);
> +
> + while (start < end) {
> + unsigned long ret, category, state;
> +
> + ret = rmi_granule_tracking_get(start, &category, &state);
> + if (ret != RMI_SUCCESS ||
> + state != RMI_TRACKING_FINE ||
> + category != RMI_MEM_CATEGORY_CONVENTIONAL) {
> + /* TODO: Set granule tracking in this case */
> + kvm_err("Granule tracking for region isn't fine/conventional: %llx",
> + start);
> + return -ENODEV;
> + }
> + start += RMM_GRANULE_TRACKING_SIZE;
> + }
> +
> + return 0;
> +}
> +
> +static unsigned long rmi_l0gpt_size(void)
> +{
> + return 1UL << (30 + FIELD_GET(RMI_FEATURE_REGISTER_1_L0GPTSZ,
> + rmm_feat_reg1));
> +}
> +
> +static int rmi_create_gpts(phys_addr_t start, phys_addr_t end)
> +{
> + unsigned long l0gpt_sz = rmi_l0gpt_size();
> +
> + start = ALIGN_DOWN(start, l0gpt_sz);
> + end = ALIGN(end, l0gpt_sz);
> +
> + while (start < end) {
> + int ret = rmi_gpt_l1_create(start);
> +
> + if (ret && ret != RMI_ERROR_GPT) {
> + /*
> + * FIXME: Handle SRO so that memory can be donated for
> + * the tables.
> + */
> + kvm_err("GPT Level1 table missing for %llx\n", start);
> + return -ENOMEM;
> + }
> + start += l0gpt_sz;
> + }
> +
> + return 0;
> +}
> +
> +static int rmi_init_metadata(void)
> +{
> + phys_addr_t start, end;
> + const struct memblock_region *r;
> +
> + for_each_mem_region(r) {
> + int ret;
> +
> + start = memblock_region_memory_base_pfn(r) << PAGE_SHIFT;
> + end = memblock_region_memory_end_pfn(r) << PAGE_SHIFT;
> + ret = rmi_verify_memory_tracking(start, end);
> + if (ret)
> + return ret;
> + ret = rmi_create_gpts(start, end);
> + if (ret)
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> static int rmm_check_features(void)
> {
> if (kvm_lpa2_is_enabled() && !rmi_has_feature(RMI_FEATURE_REGISTER_0_LPA2)) {
> @@ -120,6 +207,8 @@ void kvm_init_rmi(void)
> return;
> if (rmi_configure())
> return;
> + if (rmi_init_metadata())
> + return;
>
> /* Future patch will enable static branch kvm_rmi_is_available */
> }
^ permalink raw reply
* Re: [PATCH v13 07/48] arm64: RMI: Check for RMI support at KVM init
From: Suzuki K Poulose @ 2026-03-19 10:38 UTC (permalink / raw)
To: Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Zenghui Yu, linux-arm-kernel, linux-kernel,
Joey Gouly, Alexandru Elisei, Christoffer Dall, Fuad Tabba,
linux-coco, Ganapatrao Kulkarni, Gavin Shan, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-8-steven.price@arm.com>
On 18/03/2026 15:53, Steven Price wrote:
> Query the RMI version number and check if it is a compatible version. A
> static key is also provided to signal that a supported RMM is available.
>
> Functions are provided to query if a VM or VCPU is a realm (or rec)
> which currently will always return false.
>
> Later patches make use of struct realm and the states as the ioctls
> interfaces are added to support realm and REC creation and destruction.
>
> Signed-off-by: Steven Price <steven.price@arm.com>
> ---
> Changes since v12:
> * Drop check for 4k page size.
> Changes since v11:
> * Reword slightly the comments on the realm states.
> Changes since v10:
> * kvm_is_realm() no longer has a NULL check.
> * Rename from "rme" to "rmi" when referring to the RMM interface.
> * Check for RME (hardware) support before probing for RMI support.
> Changes since v8:
> * No need to guard kvm_init_rme() behind 'in_hyp_mode'.
> Changes since v6:
> * Improved message for an unsupported RMI ABI version.
> Changes since v5:
> * Reword "unsupported" message from "host supports" to "we want" to
> clarify that 'we' are the 'host'.
> Changes since v2:
> * Drop return value from kvm_init_rme(), it was always 0.
> * Rely on the RMM return value to identify whether the RSI ABI is
> compatible.
> ---
> arch/arm64/include/asm/kvm_emulate.h | 18 +++++++++
> arch/arm64/include/asm/kvm_host.h | 4 ++
> arch/arm64/include/asm/kvm_rmi.h | 56 +++++++++++++++++++++++++++
> arch/arm64/include/asm/virt.h | 1 +
> arch/arm64/kernel/cpufeature.c | 1 +
> arch/arm64/kvm/Makefile | 2 +-
> arch/arm64/kvm/arm.c | 5 +++
> arch/arm64/kvm/rmi.c | 57 ++++++++++++++++++++++++++++
> 8 files changed, 143 insertions(+), 1 deletion(-)
> create mode 100644 arch/arm64/include/asm/kvm_rmi.h
> create mode 100644 arch/arm64/kvm/rmi.c
>
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index 5bf3d7e1d92c..f38b50151ce8 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -688,4 +688,22 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu)
> vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
> }
> }
--8>---
> +
> +static inline bool kvm_is_realm(struct kvm *kvm)
> +{
> + if (static_branch_unlikely(&kvm_rmi_is_available))
> + return kvm->arch.is_realm;
> + return false;
> +}
> +
> +static inline enum realm_state kvm_realm_state(struct kvm *kvm)
> +{
> + return READ_ONCE(kvm->arch.realm.state);
> +}
> +
> +static inline bool vcpu_is_rec(struct kvm_vcpu *vcpu)
> +{
> + return false;
> +}
> +
> #endif /* __ARM64_KVM_EMULATE_H__ */
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 5d5a3bbdb95e..9267a2f2d65b 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -27,6 +27,7 @@
> #include <asm/fpsimd.h>
> #include <asm/kvm.h>
> #include <asm/kvm_asm.h>
> +#include <asm/kvm_rmi.h>
> #include <asm/vncr_mapping.h>
>
> #define __KVM_HAVE_ARCH_INTC_INITIALIZED
> @@ -405,6 +406,9 @@ struct kvm_arch {
> * the associated pKVM instance in the hypervisor.
> */
> struct kvm_protected_vm pkvm;
> +
> + bool is_realm;
> + struct realm realm;
> };
>
> struct kvm_vcpu_fault_info {
> diff --git a/arch/arm64/include/asm/kvm_rmi.h b/arch/arm64/include/asm/kvm_rmi.h
> new file mode 100644
> index 000000000000..3506f50b05cd
> --- /dev/null
> +++ b/arch/arm64/include/asm/kvm_rmi.h
> @@ -0,0 +1,56 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2023-2025 ARM Ltd.
> + */
> +
> +#ifndef __ASM_KVM_RMI_H
> +#define __ASM_KVM_RMI_H
> +
> +/**
> + * enum realm_state - State of a Realm
> + */
> +enum realm_state {
> + /**
> + * @REALM_STATE_NONE:
> + * Realm has not yet been created. rmi_realm_create() has not
> + * yet been called.
> + */
> + REALM_STATE_NONE,
> + /**
> + * @REALM_STATE_NEW:
> + * Realm is under construction, rmi_realm_create() has been
> + * called, but it is not yet activated. Pages may be populated.
> + */
> + REALM_STATE_NEW,
> + /**
> + * @REALM_STATE_ACTIVE:
> + * Realm has been created and is eligible for execution with
> + * rmi_rec_enter(). Pages may no longer be populated with
> + * rmi_data_create().
> + */
> + REALM_STATE_ACTIVE,
> + /**
> + * @REALM_STATE_DYING:
> + * Realm is in the process of being destroyed or has already been
> + * destroyed.
> + */
> + REALM_STATE_DYING,
> + /**
> + * @REALM_STATE_DEAD:
> + * Realm has been destroyed.
> + */
> + REALM_STATE_DEAD
> +};
> +
> +/**
> + * struct realm - Additional per VM data for a Realm
> + *
> + * @state: The lifetime state machine for the realm
> + */
> +struct realm {
> + enum realm_state state;
> +};
> +
--8<---
Minor nit: The above looks out of place in this patch. Could we
move it to where this may be actually used ?
Rest looks good to me.
> +void kvm_init_rmi(void);
> +
> +#endif /* __ASM_KVM_RMI_H */
> diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h
> index b51ab6840f9c..dc9b2899e0b2 100644
> --- a/arch/arm64/include/asm/virt.h
> +++ b/arch/arm64/include/asm/virt.h
> @@ -87,6 +87,7 @@ void __hyp_reset_vectors(void);
> bool is_kvm_arm_initialised(void);
>
> DECLARE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
> +DECLARE_STATIC_KEY_FALSE(kvm_rmi_is_available);
>
> static inline bool is_pkvm_initialized(void)
> {
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index c31f8e17732a..ddf7e57f23e8 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -289,6 +289,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar3[] = {
> static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = {
> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_CSV3_SHIFT, 4, 0),
> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_CSV2_SHIFT, 4, 0),
> + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_RME_SHIFT, 4, 0),
> ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_DIT_SHIFT, 4, 0),
> ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_AMU_SHIFT, 4, 0),
> ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_EL1_MPAM_SHIFT, 4, 0),
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 3ebc0570345c..e17c4077d8e7 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -16,7 +16,7 @@ CFLAGS_handle_exit.o += -Wno-override-init
> kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
> inject_fault.o va_layout.o handle_exit.o config.o \
> guest.o debug.o reset.o sys_regs.o stacktrace.o \
> - vgic-sys-reg-v3.o fpsimd.o pkvm.o \
> + vgic-sys-reg-v3.o fpsimd.o pkvm.o rmi.o \
> arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \
> vgic/vgic.o vgic/vgic-init.o \
> vgic/vgic-irqfd.o vgic/vgic-v2.o \
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 29f0326f7e00..274d7866efdc 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -39,6 +39,7 @@
> #include <asm/kvm_nested.h>
> #include <asm/kvm_pkvm.h>
> #include <asm/kvm_ptrauth.h>
> +#include <asm/kvm_rmi.h>
> #include <asm/sections.h>
> #include <asm/stacktrace/nvhe.h>
>
> @@ -104,6 +105,8 @@ long kvm_get_cap_for_kvm_ioctl(unsigned int ioctl, long *ext)
> return -EINVAL;
> }
>
> +DEFINE_STATIC_KEY_FALSE(kvm_rmi_is_available);
> +
> DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector);
>
> DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
> @@ -2921,6 +2924,8 @@ static __init int kvm_arm_init(void)
>
> in_hyp_mode = is_kernel_in_hyp_mode();
>
> + kvm_init_rmi();
> +
> if (cpus_have_final_cap(ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) ||
> cpus_have_final_cap(ARM64_WORKAROUND_1508412))
> kvm_info("Guests without required CPU erratum workarounds can deadlock system!\n" \
> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> new file mode 100644
> index 000000000000..fac151580c01
> --- /dev/null
> +++ b/arch/arm64/kvm/rmi.c
> @@ -0,0 +1,57 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2023-2025 ARM Ltd.
> + */
> +
> +#include <linux/kvm_host.h>
> +
> +#include <asm/rmi_cmds.h>
> +#include <asm/virt.h>
> +
> +static int rmi_check_version(void)
> +{
> + struct arm_smccc_res res;
> + unsigned short version_major, version_minor;
> + unsigned long host_version = RMI_ABI_VERSION(RMI_ABI_MAJOR_VERSION,
> + RMI_ABI_MINOR_VERSION);
> + unsigned long aa64pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
> +
> + /* If RME isn't supported, then RMI can't be */
> + if (cpuid_feature_extract_unsigned_field(aa64pfr0, ID_AA64PFR0_EL1_RME_SHIFT) == 0)
> + return -ENXIO;
> +
> + arm_smccc_1_1_invoke(SMC_RMI_VERSION, host_version, &res);
> +
> + if (res.a0 == SMCCC_RET_NOT_SUPPORTED)
> + return -ENXIO;
> +
> + version_major = RMI_ABI_VERSION_GET_MAJOR(res.a1);
> + version_minor = RMI_ABI_VERSION_GET_MINOR(res.a1);
> +
> + if (res.a0 != RMI_SUCCESS) {
> + unsigned short high_version_major, high_version_minor;
> +
> + high_version_major = RMI_ABI_VERSION_GET_MAJOR(res.a2);
> + high_version_minor = RMI_ABI_VERSION_GET_MINOR(res.a2);
> +
> + kvm_err("Unsupported RMI ABI (v%d.%d - v%d.%d) we want v%d.%d\n",
> + version_major, version_minor,
> + high_version_major, high_version_minor,
> + RMI_ABI_MAJOR_VERSION,
> + RMI_ABI_MINOR_VERSION);
> + return -ENXIO;
> + }
> +
> + kvm_info("RMI ABI version %d.%d\n", version_major, version_minor);
> +
> + return 0;
> +}
> +
> +void kvm_init_rmi(void)
> +{
> + /* Continue without realm support if we can't agree on a version */
> + if (rmi_check_version())
> + return;
> +
> + /* Future patch will enable static branch kvm_rmi_is_available */
> +}
^ permalink raw reply
* Re: [PATCH 1/2] x86/virt/tdx: Use PFN directly for mapping guest private memory
From: Kiryl Shutsemau @ 2026-03-19 10:39 UTC (permalink / raw)
To: Yan Zhao
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, x86, linux-kernel,
kvm, linux-coco, kai.huang, rick.p.edgecombe, yilun.xu,
vannapurve, ackerleytng, sagis, binbin.wu, xiaoyao.li,
isaku.yamahata
In-Reply-To: <20260319005703.8983-1-yan.y.zhao@intel.com>
On Thu, Mar 19, 2026 at 08:57:03AM +0800, Yan Zhao wrote:
> @@ -1639,16 +1644,17 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page)
> }
> EXPORT_SYMBOL_FOR_KVM(tdh_vp_addcx);
>
> -u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2)
> +u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, kvm_pfn_t pfn,
> + u64 *ext_err1, u64 *ext_err2)
> {
> struct tdx_module_args args = {
> .rcx = gpa | level,
> .rdx = tdx_tdr_pa(td),
> - .r8 = page_to_phys(page),
> + .r8 = PFN_PHYS(pfn),
> };
> u64 ret;
>
> - tdx_clflush_page(page);
> + tdx_clflush_pfn(pfn);
This is pre-existing problem, but shouldn't we respect @level here?
Flush size need to take page size into account.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v13 10/48] arm64: RMI: Ensure that the RMM has GPT entries for memory
From: Suzuki K Poulose @ 2026-03-19 10:31 UTC (permalink / raw)
To: Steven Price, kvm, kvmarm
Cc: Catalin Marinas, Marc Zyngier, Will Deacon, James Morse,
Oliver Upton, Zenghui Yu, linux-arm-kernel, linux-kernel,
Joey Gouly, Alexandru Elisei, Christoffer Dall, Fuad Tabba,
linux-coco, Ganapatrao Kulkarni, Gavin Shan, Shanker Donthineni,
Alper Gun, Aneesh Kumar K . V, Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-11-steven.price@arm.com>
Hi Steven
On 18/03/2026 15:53, Steven Price wrote:
> The RMM may not be tracking all the memory of the system at boot. Create
Looks good to me. Please find some suggestions below.
May be add a bit more context here :
RMM maintains the state of all the granules in the System to make sure
that the host is abiding by the rules. This state can be maintained at
different granularity - per PAGE (TRACKING_FINE) or per region (COARSE),
where the "region size" depends on the underlying "RMI_GRANULE_SIZE".
The state of the "tracked area" must be the same. This implies, we may
need to have "FINE" tracking for DRAM, so that we can start delegating
PAGEs. For now, we only support RMM with statically carved out memory
for tracking FINE granularity for the tracking regions. We will extend
the support for modifying the TRACKING region in the future.
Similarly, the firmware may create L0 GPT entries describing the total
address space (think of this as Block mappings in the page tables). But
if we change the "PAS" of a granule in the block mapping, we may need
to create L1 tables to track the PAS at the finer granularity. For now
we only support a system where the L1 GPTs are created at boot time
and dynamic GPT support will be added later.
> the necessary tracking state and GPTs within the RMM so that all boot
> memory can be delegated to the RMM as needed during runtime.
>
> Note: support is currently missing for SROs which means that if the RMM
> needs memory donating this will fail (and render CCA unusable in Linux).
>
> Signed-off-by: Steven Price <steven.price@arm.com>
> ---
> New patch for v13
> ---
> arch/arm64/kvm/rmi.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 89 insertions(+)
>
> diff --git a/arch/arm64/kvm/rmi.c b/arch/arm64/kvm/rmi.c
> index 9590dff9a2c1..80aedc85e94a 100644
> --- a/arch/arm64/kvm/rmi.c
> +++ b/arch/arm64/kvm/rmi.c
> @@ -4,6 +4,7 @@
> */
>
> #include <linux/kvm_host.h>
> +#include <linux/memblock.h>
>
> #include <asm/kvm_pgtable.h>
> #include <asm/rmi_cmds.h>
> @@ -56,6 +57,18 @@ static int rmi_check_version(void)
> return 0;
> }
>
> +/*
> + * These are the 'default' sizes when passing 0 as the tracking_region_size.
> + * TODO: Support other granule sizes
> + */
> +#ifdef CONFIG_PAGE_SIZE_4KB
> +#define RMM_GRANULE_TRACKING_SIZE SZ_1G
> +#elif defined(CONFIG_PAGE_SIZE_16KB)
> +#define RMM_GRANULE_TRACKING_SIZE SZ_32M
> +#elif defined(CONFIG_PAGE_SIZE_64KB)
> +#define RMM_GRANULE_TRACKING_SIZE SZ_512M
> +#endif
> +
Probably this should be made a Kconfig option, like the VA_BITS we have
today for each page size.
> static int rmi_configure(void)
> {
> struct rmm_config *config __free(free_page) = NULL;
> @@ -95,6 +108,80 @@ static int rmi_configure(void)
> return 0;
> }
>
> +static int rmi_verify_memory_tracking(phys_addr_t start, phys_addr_t end)
Could we add a comment what we are trying to do here ?
/*
* Make sure the area is tracked by RMM at FINE granularity.
* We do not support changing the TRACKING yet. This will
* be added in the future.
*/
> +{
> + start = ALIGN_DOWN(start, RMM_GRANULE_TRACKING_SIZE);
> + end = ALIGN(end, RMM_GRANULE_TRACKING_SIZE);
> +
> + while (start < end) {
> + unsigned long ret, category, state;
> +
> + ret = rmi_granule_tracking_get(start, &category, &state);
> + if (ret != RMI_SUCCESS ||
> + state != RMI_TRACKING_FINE ||
> + category != RMI_MEM_CATEGORY_CONVENTIONAL) {
> + /* TODO: Set granule tracking in this case */
> + kvm_err("Granule tracking for region isn't fine/conventional: %llx",
> + start);
> + return -ENODEV;
> + }
> + start += RMM_GRANULE_TRACKING_SIZE;
> + }
> +
> + return 0;
> +}
> +
> +static unsigned long rmi_l0gpt_size(void)
> +{
> + return 1UL << (30 + FIELD_GET(RMI_FEATURE_REGISTER_1_L0GPTSZ,
> + rmm_feat_reg1));
> +}
> +
> +static int rmi_create_gpts(phys_addr_t start, phys_addr_t end)
> +{
> + unsigned long l0gpt_sz = rmi_l0gpt_size();
> +
> + start = ALIGN_DOWN(start, l0gpt_sz);
> + end = ALIGN(end, l0gpt_sz);
> +
> + while (start < end) {
> + int ret = rmi_gpt_l1_create(start);
How about adding a comment here explaining why we look for RMI_ERROR_GPT ?
>
/*
* Make sure the L1 GPT tables are created for the region.
* RMI_ERROR_GPT indicates the L1 table exists.
*/
+
> + if (ret && ret != RMI_ERROR_GPT) {
> + /*
> + * FIXME: Handle SRO so that memory can be donated for
> + * the tables.
> + */
> + kvm_err("GPT Level1 table missing for %llx\n", start);
> + return -ENOMEM;
> + }
> + start += l0gpt_sz;
> + }
> +
> + return 0;
> +}
> +
> +static int rmi_init_metadata(void)
> +{
> + phys_addr_t start, end;
> + const struct memblock_region *r;
> +
> + for_each_mem_region(r) {
> + int ret;
> +
> + start = memblock_region_memory_base_pfn(r) << PAGE_SHIFT;
> + end = memblock_region_memory_end_pfn(r) << PAGE_SHIFT;
> + ret = rmi_verify_memory_tracking(start, end);
> + if (ret)
> + return ret;
> + ret = rmi_create_gpts(start, end);
> + if (ret)
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> static int rmm_check_features(void)
> {
> if (kvm_lpa2_is_enabled() && !rmi_has_feature(RMI_FEATURE_REGISTER_0_LPA2)) {
> @@ -120,6 +207,8 @@ void kvm_init_rmi(void)
> return;
> if (rmi_configure())
> return;
> + if (rmi_init_metadata())
> + return;
>
> /* Future patch will enable static branch kvm_rmi_is_available */
> }
^ permalink raw reply
* Re: [PATCH v13 37/48] arm64: RMI: Prevent Device mappings for Realms
From: Joey Gouly @ 2026-03-19 10:27 UTC (permalink / raw)
To: Steven Price
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Suzuki K Poulose, Zenghui Yu,
linux-arm-kernel, linux-kernel, Alexandru Elisei,
Christoffer Dall, Fuad Tabba, linux-coco, Ganapatrao Kulkarni,
Gavin Shan, Shanker Donthineni, Alper Gun, Aneesh Kumar K . V,
Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-38-steven.price@arm.com>
On Wed, Mar 18, 2026 at 03:54:01PM +0000, Steven Price wrote:
> Physical device assignment is not supported by RMM v1.0, so it
But we're targetting 2.0 now!
I guess just change it to something about device support being a later feature.
Thanks,
Joey
> doesn't make much sense to allow device mappings within the realm.
> Prevent them when the guest is a realm.
>
> Signed-off-by: Steven Price <steven.price@arm.com>
> ---
> Changes from v6:
> * Fix the check in user_mem_abort() to prevent all pages that are not
> guest_memfd() from being mapped into the protected half of the IPA.
> Changes from v5:
> * Also prevent accesses in user_mem_abort()
> ---
> arch/arm64/kvm/mmu.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index ad1300f366df..7d7caab8f573 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1222,6 +1222,10 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
> if (is_protected_kvm_enabled())
> return -EPERM;
>
> + /* We don't support mapping special pages into a Realm */
> + if (kvm_is_realm(kvm))
> + return -EPERM;
> +
> size += offset_in_page(guest_ipa);
> guest_ipa &= PAGE_MASK;
>
> @@ -1965,6 +1969,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> return 1;
> }
>
> + /*
> + * For now we shouldn't be hitting protected addresses because they are
> + * handled in private_memslot_fault(). In the future this check may be
> + * relaxed to support e.g. protected devices.
> + */
> + if (vcpu_is_rec(vcpu) &&
> + kvm_gpa_from_fault(kvm, fault_ipa) == fault_ipa)
> + return -EINVAL;
> +
> if (nested)
> adjust_nested_fault_perms(nested, &prot, &writable);
>
> --
> 2.43.0
>
>
^ permalink raw reply
* Re: [PATCH v13 36/48] arm64: RMI: Always use 4k pages for realms
From: Joey Gouly @ 2026-03-19 10:24 UTC (permalink / raw)
To: Steven Price
Cc: kvm, kvmarm, Catalin Marinas, Marc Zyngier, Will Deacon,
James Morse, Oliver Upton, Suzuki K Poulose, Zenghui Yu,
linux-arm-kernel, linux-kernel, Alexandru Elisei,
Christoffer Dall, Fuad Tabba, linux-coco, Ganapatrao Kulkarni,
Gavin Shan, Shanker Donthineni, Alper Gun, Aneesh Kumar K . V,
Emi Kisanuki, Vishal Annapurve
In-Reply-To: <20260318155413.793430-37-steven.price@arm.com>
Hi,
On Wed, Mar 18, 2026 at 03:54:00PM +0000, Steven Price wrote:
> Guest_memfd doesn't yet natively support huge pages, and there are
> currently difficulties for a VMM to manage huge pages efficiently so for
> now always split up mappings to PTE (4k).
>
> The two issues that need progressing before supporting huge pages for
> realms are:
>
> 1. guest_memfd needs to be able to allocate from an appropriate
> allocator which can provide huge pages.
>
> 2. The VMM needs to be able to repurpose private memory for a shared
> mapping when the guest VM requests memory is transitioned. Because
> this can happen at a 4k granularity it isn't possible to
> free/reallocate while huge pages are in use. Allowing the VMM to
> mmap() the shared portion of a huge page would allow the huge page
> to be recreated when the memory is unshared and made protected again.
>
> These two issues are not specific to realms and don't affect the realm
> API, so for now just break everything down to 4k pages in the RMM
> controlled stage 2. Future work can add huge page support without
> changing the uAPI.
The commit title/message mention 4K, but should probably say PAGE_SIZE or
something now that RMM isn't fixed to 4K.
Thanks,
Joey
>
> Signed-off-by: Steven Price <steven.price@arm.com>
> Reviewed-by: Gavin Shan <gshan@redhat.com>
> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
> ---
> Changes since v7:
> * Rewritten commit message
> ---
> arch/arm64/kvm/mmu.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index 73c18c2861a2..ad1300f366df 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1761,11 +1761,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
> write_fault = kvm_is_write_fault(vcpu);
>
> /*
> - * Realms cannot map protected pages read-only
> + * Realms cannot map protected pages read-only, also force PTE mappings
> + * for Realms.
> * FIXME: It should be possible to map unprotected pages read-only
> */
> - if (vcpu_is_rec(vcpu))
> + if (vcpu_is_rec(vcpu)) {
> write_fault = true;
> + force_pte = true;
> + }
>
> exec_fault = kvm_vcpu_trap_is_exec_fault(vcpu);
> VM_WARN_ON_ONCE(write_fault && exec_fault);
> --
> 2.43.0
>
>
^ permalink raw reply
* Re: [PATCH 2/2] x86/virt/tdx: Use PFN directly for unmapping guest private memory
From: Yan Zhao @ 2026-03-19 8:56 UTC (permalink / raw)
To: Xiaoyao Li
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, kas, x86,
linux-kernel, kvm, linux-coco, kai.huang, rick.p.edgecombe,
yilun.xu, vannapurve, ackerleytng, sagis, binbin.wu,
isaku.yamahata
In-Reply-To: <b74acb4b-4658-4113-9fce-9faf972975c4@intel.com>
On Thu, Mar 19, 2026 at 04:56:10PM +0800, Xiaoyao Li wrote:
> On 3/19/2026 2:45 PM, Yan Zhao wrote:
> > On Thu, Mar 19, 2026 at 11:20:48AM +0800, Xiaoyao Li wrote:
> > > On 3/19/2026 8:58 AM, Yan Zhao wrote:
> > > > From: Sean Christopherson <seanjc@google.com>
> [...]
> > > > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> > > > index a9dd75190c67..2f9d07ad1a9a 100644
> > > > --- a/arch/x86/virt/vmx/tdx/tdx.c
> > > > +++ b/arch/x86/virt/vmx/tdx/tdx.c
> > > > @@ -730,9 +730,9 @@ static void tdx_quirk_reset_paddr(unsigned long base, unsigned long size)
> > > > mb();
> > > > }
> > > > -void tdx_quirk_reset_page(struct page *page)
> > > > +void tdx_quirk_reset_page(kvm_pfn_t pfn)
> > >
> > > So why keep the function tdx_quirk_reset_page() but expect passing in the
> > > kvm_pfn_t? It looks werid that the name indicates to reset a page but what
> > > gets passed in is a pfn.
> > I thought about introducing tdx_quirk_reset_pfn(). But considering
> > tdx_quirk_reset_pfn() has to be an exported API, I'm reluctant to do that.
> >
> > Given that even with tdx_quirk_reset_pfn(), it still expects TDX convertible
> > RAM, I think having tdx_quirk_reset_page() to take pfn is still acceptable.
> >
> > We just don't want KVM to do pfn --> struct page --> pfn conversions.
>
> Only tdx_sept_remove_private_spte() is doing such conversions. While
> tdx_reclaim_page() and tdx_reclaim_td_control_pages() already have the
> struct page natively.
Unlike requiring KVM to call pfn_to_page() before invoking guest private memory
related APIs, Having tdx_reclaim_page() and tdx_reclaim_td_control_pages() to
call page_to_pfn() does not impose unnecessary assumptions of how KVM allocates
memory. So, I think it's fine for them to invoke tdx_quirk_reset_page() which
takes PFN as input.
> So why not considering option 2?
>
> 2. keep tdx_quirk_reset_page() as-is for the cases of
> tdx_reclaim_page() and tdx_reclaim_td_control_pages() that have the
> struct page. But only change tdx_sept_remove_private_spte() to use
> tdx_quirk_reset_paddr() directly.
>
> It will need export tdx_quirk_reset_paddr() for KVM. I think it will be OK?
I don't think it's necessary. But if we have to export an extra API, IMHO,
tdx_quirk_reset_pfn() is better than tdx_quirk_reset_paddr(). Otherwise,
why not only expose tdx_quirk_reset_paddr()?
^ permalink raw reply
* Re: [PATCH 2/2] x86/virt/tdx: Use PFN directly for unmapping guest private memory
From: Xiaoyao Li @ 2026-03-19 8:56 UTC (permalink / raw)
To: Yan Zhao
Cc: seanjc, pbonzini, dave.hansen, tglx, mingo, bp, kas, x86,
linux-kernel, kvm, linux-coco, kai.huang, rick.p.edgecombe,
yilun.xu, vannapurve, ackerleytng, sagis, binbin.wu,
isaku.yamahata
In-Reply-To: <abubmi7BjMwgnrvC@yzhao56-desk.sh.intel.com>
On 3/19/2026 2:45 PM, Yan Zhao wrote:
> On Thu, Mar 19, 2026 at 11:20:48AM +0800, Xiaoyao Li wrote:
>> On 3/19/2026 8:58 AM, Yan Zhao wrote:
>>> From: Sean Christopherson <seanjc@google.com>
[...]
>>> diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
>>> index a9dd75190c67..2f9d07ad1a9a 100644
>>> --- a/arch/x86/virt/vmx/tdx/tdx.c
>>> +++ b/arch/x86/virt/vmx/tdx/tdx.c
>>> @@ -730,9 +730,9 @@ static void tdx_quirk_reset_paddr(unsigned long base, unsigned long size)
>>> mb();
>>> }
>>> -void tdx_quirk_reset_page(struct page *page)
>>> +void tdx_quirk_reset_page(kvm_pfn_t pfn)
>>
>> So why keep the function tdx_quirk_reset_page() but expect passing in the
>> kvm_pfn_t? It looks werid that the name indicates to reset a page but what
>> gets passed in is a pfn.
> I thought about introducing tdx_quirk_reset_pfn(). But considering
> tdx_quirk_reset_pfn() has to be an exported API, I'm reluctant to do that.
>
> Given that even with tdx_quirk_reset_pfn(), it still expects TDX convertible
> RAM, I think having tdx_quirk_reset_page() to take pfn is still acceptable.
>
> We just don't want KVM to do pfn --> struct page --> pfn conversions.
Only tdx_sept_remove_private_spte() is doing such conversions. While
tdx_reclaim_page() and tdx_reclaim_td_control_pages() already have the
struct page natively.
So why not considering option 2?
2. keep tdx_quirk_reset_page() as-is for the cases of
tdx_reclaim_page() and tdx_reclaim_td_control_pages() that have the
struct page. But only change tdx_sept_remove_private_spte() to use
tdx_quirk_reset_paddr() directly.
It will need export tdx_quirk_reset_paddr() for KVM. I think it will be OK?
^ permalink raw reply
* Re: [PATCH] KVM: TDX: Fix APIC MSR ranges in tdx_has_emulated_msr()
From: Binbin Wu @ 2026-03-19 7:40 UTC (permalink / raw)
To: Dave Hansen, Rick Edgecombe, Dmytro Maluka, kvm,
Sean Christopherson, Paolo Bonzini, Isaku Yamahata
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen,
maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), H. Peter Anvin,
Kiryl Shutsemau, open list:X86 ARCHITECTURE (32-BIT AND 64-BIT),
open list:X86 TRUST DOMAIN EXTENSIONS (TDX)
In-Reply-To: <93358559-5ed1-4574-8951-24d7ea9354e4@intel.com>
On 3/19/2026 9:48 AM, Dave Hansen wrote:
> On 3/18/26 18:14, Binbin Wu wrote:
>> The bug doesn't cause problems for TDs because:
>> - These x2apic MSRs (TASKPRI, PROCPRI, EOI, ISRx, TMRx, IRRx) are virtualized by CPU,
>> when a TD accesses these MSRs, it doesn't cause #VE, thus no TDVMCALL from the TD to
>> request the emulation of these MSRs.
>> - The bug make the "false" range of APIC MSRs smaller, so it doesn't impact the result
>> for the rest of the APIC MSRs.
>
> Could we fix this up so that the code that's there is actually usable
> and testable, please?
>
tdx_has_emulated_msr() is used by KVM to decide whether to emulate a MSR access from the
TDVMCALL or just return the error code.
During an off-list discussion, Rick noted that #VE reduction could change the behavior of
accessing an MSR (e.g., from #VE to #GP or to be virtualized by the TDX module) without
KVM knowing.Because KVM lacks the full context to perfectly decide if an MSR should be
emulated, the question was raised: Can we just delete tdx_has_emulated_msr() entirely?
However, these native type x2apic MSRs are a special case. Since the TDX module owns the
APICv page, KVM cannot emulate these MSRs. If we remove tdx_has_emulated_msr(), a guest
directly issuing TDVMCALLs for these native type x2apic MSRs will trigger a silent failure,
even though this is the guest's fault.
It comes down to a tradeoff. Should we prioritize code simplicity by dropping the function,
or keep it to explicitly catch this misbehaving guest corner case?
BTW, besides the bug described by this patch, according to the latest published TDX module
ABI table, MSR IA32_X2APIC_SELF_IPI is native type, but not included in the list.
There are some MSRs, which are reserved for xAPIC MSR, not included in the list, but they
can be covered by the KVM common code.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox