* Re: [PATCH v2 2/7] x86/sev: add support for enabling RMPOPT
From: Tom Lendacky @ 2026-03-06 15:33 UTC (permalink / raw)
To: Borislav Petkov, Ashish Kalra
Cc: tglx, mingo, dave.hansen, x86, hpa, seanjc, peterz, herbert,
davem, ardb, pbonzini, aik, Michael.Roth, KPrateek.Nayak,
Tycho.Andersen, Nathan.Fontenot, jackyli, pgonda, rientjes,
jacobhxu, xin, pawan.kumar.gupta, babu.moger, dyoung, nikunj,
john.allen, darwi, linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <20260306151849.GJaarwWSaWnnRh9ffB@fat_crate.local>
On 3/6/26 09:18, Borislav Petkov wrote:
> On Mon, Mar 02, 2026 at 09:35:55PM +0000, Ashish Kalra wrote:
>> From: Ashish Kalra <ashish.kalra@amd.com>
>>
>> The new RMPOPT instruction sets bits in a per-CPU RMPOPT table, which
>> indicates whether specific 1GB physical memory regions contain SEV-SNP
>
> "... which indicate... "
>
>> guest memory.
>>
>> Per-CPU RMPOPT tables support at most 2 TB of addressable memory for
>> RMP optimizations.
>>
>> Initialize the per-CPU RMPOPT table base to the starting physical
>> address. This enables RMP optimization for up to 2 TB of system RAM on
>> all CPUs.
>
> ...
>
>> +static void __configure_rmpopt(void *val)
>> +{
>> + u64 rmpopt_base = ((u64)val & PUD_MASK) | MSR_AMD64_RMPOPT_ENABLE;
>> +
>> + wrmsrq(MSR_AMD64_RMPOPT_BASE, rmpopt_base);
>> +}
>> +
>> +static __init void configure_and_enable_rmpopt(void)
>
> If the sub-helper is called __configure_rmpopt() then this should be called
> "configure_rmpopt", without the prepended underscores.
>
>> + phys_addr_t pa_start = ALIGN_DOWN(PFN_PHYS(min_low_pfn), PUD_SIZE);
>> +
>> + if (!cpu_feature_enabled(X86_FEATURE_RMPOPT)) {
>> + pr_debug("RMPOPT not supported on this platform\n");
>> + return;
>> + }
>> +
>> + if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP)) {
>> + pr_debug("RMPOPT optimizations not enabled as SNP support is not enabled\n");
>> + return;
>> + }
>
> Zap this one - snp_rmptable_init() already checked it.
>
> Also, zap those pr_debugs - you have that information elsewhere already.
>
>> +
>> + if (!(rmp_cfg & MSR_AMD64_SEG_RMP_ENABLED)) {
>
> You can't test this one - you need to test the result of
> setup_segmented_rmptable() and whether it did set up the segmented RMP
> properly. Only then you can continue here.
If the segmented RMP setup fails, then CC_ATTR_HOST_SEV_SNP gets cleared,
so it looks like the above check needs to remain then.
Thanks,
Tom
>
>> + pr_info("RMPOPT optimizations not enabled, segmented RMP required\n");
>
> This looks like pr_notice() to me.
>
>> + return;
>> + }
>
^ permalink raw reply
* [PATCH 0/4] Fuller TDX kexec support
From: Rick Edgecombe @ 2026-03-07 1:03 UTC (permalink / raw)
To: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, chao.gao, kai.huang, ackerleytng
Cc: rick.p.edgecombe, vishal.l.verma
Hi,
This series adds a couple of cool things -
1. Allow kexec and kdump on systems with the partial write errata
2. Allow using TDX in the second (kexec'ed) kernel
It has been waiting for VMXON refactor to land because the implementation
is much cleaner on top of that. The series was mostly done by Vishal,
however for scheduling reasons I'm posting it on his behalf. I can handle
all questions/comments for the time being. So it's ready for review.
KVM folks, just a few deletions on your side and the long discussed moving
of tdx_errno.h. Tip folks and reviewers, the changes here are pretty small.
Optimistically, I'm hoping we can iterate this quickly and see it off the
list in the next few weeks.
Background
==========
Some early TDX-capable platforms have an erratum where a partial write
to TDX private memory can cause a machine check on a subsequent read.
Currently, kexec and kdump are disabled on these platforms because the
new (or kdump) kernel may trip over these, causing a machine check.
Future TDX modules will support TDH.SYS.DISABLE SEAMCALL, which disables
the TDX module and reclaims all memory resources allocated to TDX, and
cleans up any poison. After this SEAMCALL succeeds, the new kernel
can also re-initialize the TDX module from scratch via the normal bring-up
sequence.
It is probably worth mentioning that this is a different kind of cleanup
than the WBINVD stuff that was the cause of all the fuss in the earlier
kexec enabling. The WBINVD is flushing private keyid cachelines so they
are not later written back over the new kernels memory. It needs to happen
after the last SEAMCALL that might have produced them. So this new
SEAMCALL is for something else, but also needs to be slotted with respect
to WBINVD.
Implementation
==============
The series adds:
1. A pre-requisite patch to move TDX error code definitions to a
shared location so that TDX_INTERRUPTED_RESUMABLE etc. are
accessible from arch/x86/virt/vmx/tdx/. This comes from the Dynamic
PAMT series [0], but is also needed by some other series, and can
benefit them all from an early merge.
2. A preparatory patch to move some straggling stuff into arch/x86 in the
wake of the VMXON series.
3. A tdx_sys_disable() helper that wraps calls TDH.SYS.DISABLE with a
retry loop to handle TDX_INTERRUPTED_RESUMABLE.
4. Integration into the kexec path: Remove the check for partial write
errata platforms as this is addressed by the SEAMCALL clearing any
poisoned memory locations. Call tdx_sys_disable() in tdx_shutdown
which is called via syscore ops in the kexec path. Call
tdx_sys_disable() in native_machine_crash_shutdown() to cover the
crash (kdump) path.
Testing
=======
The new SEAMCALL has NOT been implemented in a TDX module yet. The
implementation is based on the draft TDX module spec available at [1].
Testing was limited to the TDX CI, and a basic kexec test. The code needs
to be robust to the TDX module not containing the feature, so this
effectively serves as regression test. During development further testing
was done by mocking up the new SEAMCALL to introduce delays and exercise
the retry loops, combined with kexec, kdump, reboot and shutdown flows.
Base
====
This series is based on the vmxon branch Sean pushed to kvm_x86,
kvm-x86-vmxon-2026.03.05.
[0]: https://lore.kernel.org/kvm/20260129011517.3545883-11-seanjc@google.com/
[1]: https://cdrdv2.intel.com/v1/dl/getContent/871617
Kiryl Shutsemau (1):
x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>
Rick Edgecombe (1):
x86/virt/tdx: Pull kexec cache flush logic into arch/x86
Vishal Verma (2):
x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
KVM: x86: Disable the TDX module during kexec and kdump
arch/x86/include/asm/shared/tdx.h | 1 +
.../vmx => include/asm/shared}/tdx_errno.h | 27 +++++++++--
arch/x86/include/asm/tdx.h | 29 ++----------
arch/x86/kernel/crash.c | 2 +
arch/x86/kernel/machine_kexec_64.c | 16 -------
arch/x86/kvm/vmx/tdx.c | 10 ----
arch/x86/kvm/vmx/tdx.h | 1 -
arch/x86/virt/vmx/tdx/tdx.c | 46 +++++++++++++------
arch/x86/virt/vmx/tdx/tdx.h | 1 +
9 files changed, 62 insertions(+), 71 deletions(-)
rename arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h (65%)
--
2.53.0
^ permalink raw reply
* [PATCH 1/4] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>
From: Rick Edgecombe @ 2026-03-07 1:03 UTC (permalink / raw)
To: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, chao.gao, kai.huang, ackerleytng
Cc: rick.p.edgecombe, vishal.l.verma, Kirill A. Shutemov, Sagi Shahar,
Vishal Annapurve
In-Reply-To: <20260307010358.819645-1-rick.p.edgecombe@intel.com>
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Today there are two separate locations where TDX error codes are defined:
arch/x86/include/asm/tdx.h
arch/x86/kvm/vmx/tdx_errno.h
They have some overlap that is already defined similarly. Reduce the
duplication and prepare to introduce some helpers for these error codes in
the central place by unifying them. Join them at:
asm/shared/tdx_errno.h
...and update the headers that contained the duplicated definitions to
include the new unified header.
Place the new header in "asm/shared". While the compressed code for the
guest doesn't use these error code header definitions today, it does
make the types of calls that return the values they define. Place the
defines in "shared" location so that compressed code has the definitions
accessible, but leave cleanups to use proper error codes for future
changes.
Opportunistically massage some comments. Also, adjust
_BITUL()->_BITULL() to address 32 bit build errors after the move.
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
[enhance log]
Tested-by: Sagi Shahar <sagis@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Acked-by: Vishal Annapurve <vannapurve@google.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
arch/x86/include/asm/shared/tdx.h | 1 +
.../vmx => include/asm/shared}/tdx_errno.h | 27 +++++++++++++++----
arch/x86/include/asm/tdx.h | 20 --------------
arch/x86/kvm/vmx/tdx.h | 1 -
4 files changed, 23 insertions(+), 26 deletions(-)
rename arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h (65%)
diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h
index 8bc074c8d7c6..6a1646fc2b2f 100644
--- a/arch/x86/include/asm/shared/tdx.h
+++ b/arch/x86/include/asm/shared/tdx.h
@@ -4,6 +4,7 @@
#include <linux/bits.h>
#include <linux/types.h>
+#include <asm/shared/tdx_errno.h>
#define TDX_HYPERCALL_STANDARD 0
diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/include/asm/shared/tdx_errno.h
similarity index 65%
rename from arch/x86/kvm/vmx/tdx_errno.h
rename to arch/x86/include/asm/shared/tdx_errno.h
index 6ff4672c4181..3aa74f6a6119 100644
--- a/arch/x86/kvm/vmx/tdx_errno.h
+++ b/arch/x86/include/asm/shared/tdx_errno.h
@@ -1,14 +1,16 @@
/* SPDX-License-Identifier: GPL-2.0 */
-/* architectural status code for SEAMCALL */
+#ifndef _X86_SHARED_TDX_ERRNO_H
+#define _X86_SHARED_TDX_ERRNO_H
-#ifndef __KVM_X86_TDX_ERRNO_H
-#define __KVM_X86_TDX_ERRNO_H
+#include <asm/trapnr.h>
+/* Upper 32 bit of the TDX error code encodes the status */
#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL
/*
- * TDX SEAMCALL Status Codes (returned in RAX)
+ * TDX SEAMCALL Status Codes
*/
+#define TDX_SUCCESS 0ULL
#define TDX_NON_RECOVERABLE_VCPU 0x4000000100000000ULL
#define TDX_NON_RECOVERABLE_TD 0x4000000200000000ULL
#define TDX_NON_RECOVERABLE_TD_NON_ACCESSIBLE 0x6000000500000000ULL
@@ -17,6 +19,7 @@
#define TDX_OPERAND_INVALID 0xC000010000000000ULL
#define TDX_OPERAND_BUSY 0x8000020000000000ULL
#define TDX_PREVIOUS_TLB_EPOCH_BUSY 0x8000020100000000ULL
+#define TDX_RND_NO_ENTROPY 0x8000020300000000ULL
#define TDX_PAGE_METADATA_INCORRECT 0xC000030000000000ULL
#define TDX_VCPU_NOT_ASSOCIATED 0x8000070200000000ULL
#define TDX_KEY_GENERATION_FAILED 0x8000080000000000ULL
@@ -28,6 +31,20 @@
#define TDX_EPT_ENTRY_STATE_INCORRECT 0xC0000B0D00000000ULL
#define TDX_METADATA_FIELD_NOT_READABLE 0xC0000C0200000000ULL
+/*
+ * SW-defined error codes.
+ *
+ * Bits 47:40 == 0xFF indicate Reserved status code class that never used by
+ * TDX module.
+ */
+#define TDX_ERROR _BITULL(63)
+#define TDX_NON_RECOVERABLE _BITULL(62)
+#define TDX_SW_ERROR (TDX_ERROR | GENMASK_ULL(47, 40))
+#define TDX_SEAMCALL_VMFAILINVALID (TDX_SW_ERROR | _ULL(0xFFFF0000))
+
+#define TDX_SEAMCALL_GP (TDX_SW_ERROR | X86_TRAP_GP)
+#define TDX_SEAMCALL_UD (TDX_SW_ERROR | X86_TRAP_UD)
+
/*
* TDX module operand ID, appears in 31:0 part of error code as
* detail information
@@ -37,4 +54,4 @@
#define TDX_OPERAND_ID_SEPT 0x92
#define TDX_OPERAND_ID_TD_EPOCH 0xa9
-#endif /* __KVM_X86_TDX_ERRNO_H */
+#endif /* _X86_SHARED_TDX_ERRNO_H */
diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
index a149740b24e8..0c1ae4954f17 100644
--- a/arch/x86/include/asm/tdx.h
+++ b/arch/x86/include/asm/tdx.h
@@ -12,26 +12,6 @@
#include <asm/trapnr.h>
#include <asm/shared/tdx.h>
-/*
- * SW-defined error codes.
- *
- * Bits 47:40 == 0xFF indicate Reserved status code class that never used by
- * TDX module.
- */
-#define TDX_ERROR _BITUL(63)
-#define TDX_NON_RECOVERABLE _BITUL(62)
-#define TDX_SW_ERROR (TDX_ERROR | GENMASK_ULL(47, 40))
-#define TDX_SEAMCALL_VMFAILINVALID (TDX_SW_ERROR | _UL(0xFFFF0000))
-
-#define TDX_SEAMCALL_GP (TDX_SW_ERROR | X86_TRAP_GP)
-#define TDX_SEAMCALL_UD (TDX_SW_ERROR | X86_TRAP_UD)
-
-/*
- * TDX module SEAMCALL leaf function error codes
- */
-#define TDX_SUCCESS 0ULL
-#define TDX_RND_NO_ENTROPY 0x8000020300000000ULL
-
#ifndef __ASSEMBLER__
#include <uapi/asm/mce.h>
diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h
index b5cd2ffb303e..ac8323a68b16 100644
--- a/arch/x86/kvm/vmx/tdx.h
+++ b/arch/x86/kvm/vmx/tdx.h
@@ -3,7 +3,6 @@
#define __KVM_X86_VMX_TDX_H
#include "tdx_arch.h"
-#include "tdx_errno.h"
#ifdef CONFIG_KVM_INTEL_TDX
#include "common.h"
--
2.53.0
^ permalink raw reply related
* [PATCH 2/4] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Rick Edgecombe @ 2026-03-07 1:03 UTC (permalink / raw)
To: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, chao.gao, kai.huang, ackerleytng
Cc: rick.p.edgecombe, vishal.l.verma
In-Reply-To: <20260307010358.819645-1-rick.p.edgecombe@intel.com>
KVM tries to take care of some required cache flushing earlier in the
kexec path in order to be kind to some long standing races that can occur
later in the operation. Until recently, VMXOFF was handled within KVM.
Since VMX being enabled is required to make a SEAMCALL, it had the best
per-cpu scoped operation to plug the flushing into.
This early kexec cache flushing in KVM happens via a syscore shutdown
callback. Now that VMX enablement control has moved to arch/x86, which has
grown its own syscore shutdown callback, it no longer make sense for it to
live in KVM. It fits better with the TDX enablement managing code.
In addition, future changes will add a SEAMCALL that happens immediately
before VMXOFF, which means the cache flush in KVM will be too late to be
helpful. So move it to the newly added TDX arch/x86 syscore shutdown
handler.
Since tdx_cpu_flush_cache_for_kexec() is no longer needed by KVM, make it
static and remove the export. Since it is also not part of an operation
spread across disparate components, remove the redundant comments and
verbose naming.
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
arch/x86/include/asm/tdx.h | 6 ------
arch/x86/kvm/vmx/tdx.c | 10 ----------
arch/x86/virt/vmx/tdx/tdx.c | 39 +++++++++++++++++++------------------
3 files changed, 20 insertions(+), 35 deletions(-)
diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
index 0c1ae4954f17..f0826b0a512a 100644
--- a/arch/x86/include/asm/tdx.h
+++ b/arch/x86/include/asm/tdx.h
@@ -206,11 +206,5 @@ static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
static inline const struct tdx_sys_info *tdx_get_sysinfo(void) { return NULL; }
#endif /* CONFIG_INTEL_TDX_HOST */
-#ifdef CONFIG_KEXEC_CORE
-void tdx_cpu_flush_cache_for_kexec(void);
-#else
-static inline void tdx_cpu_flush_cache_for_kexec(void) { }
-#endif
-
#endif /* !__ASSEMBLER__ */
#endif /* _ASM_X86_TDX_H */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b7264b533feb..50a5cfdbd33e 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -440,16 +440,6 @@ void tdx_disable_virtualization_cpu(void)
tdx_flush_vp(&arg);
}
local_irq_restore(flags);
-
- /*
- * Flush cache now if kexec is possible: this is necessary to avoid
- * having dirty private memory cachelines when the new kernel boots,
- * but WBINVD is a relatively expensive operation and doing it during
- * kexec can exacerbate races in native_stop_other_cpus(). Do it
- * now, since this is a safe moment and there is going to be no more
- * TDX activity on this CPU from this point on.
- */
- tdx_cpu_flush_cache_for_kexec();
}
#define TDX_SEAMCALL_RETRIES 10000
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index cb9b3210ab71..0802d0fd18a4 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -224,8 +224,28 @@ static int tdx_offline_cpu(unsigned int cpu)
return 0;
}
+static void tdx_cpu_flush_cache(void)
+{
+ lockdep_assert_preemption_disabled();
+
+ if (!this_cpu_read(cache_state_incoherent))
+ return;
+
+ wbinvd();
+ this_cpu_write(cache_state_incoherent, false);
+}
+
static void tdx_shutdown_cpu(void *ign)
{
+ /*
+ * Flush cache now if kexec is possible: this is necessary to avoid
+ * having dirty private memory cachelines when the new kernel boots,
+ * but WBINVD is a relatively expensive operation and doing it during
+ * kexec can exacerbate races in native_stop_other_cpus(). Do it
+ * now, since this is a safe moment and there is going to be no more
+ * TDX activity on this CPU from this point on.
+ */
+ tdx_cpu_flush_cache();
x86_virt_put_ref(X86_FEATURE_VMX);
}
@@ -1920,22 +1940,3 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
}
EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
-
-#ifdef CONFIG_KEXEC_CORE
-void tdx_cpu_flush_cache_for_kexec(void)
-{
- lockdep_assert_preemption_disabled();
-
- if (!this_cpu_read(cache_state_incoherent))
- return;
-
- /*
- * Private memory cachelines need to be clean at the time of
- * kexec. Write them back now, as the caller promises that
- * there should be no more SEAMCALLs on this CPU.
- */
- wbinvd();
- this_cpu_write(cache_state_incoherent, false);
-}
-EXPORT_SYMBOL_FOR_KVM(tdx_cpu_flush_cache_for_kexec);
-#endif
--
2.53.0
^ permalink raw reply related
* [PATCH 3/4] x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
From: Rick Edgecombe @ 2026-03-07 1:03 UTC (permalink / raw)
To: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, chao.gao, kai.huang, ackerleytng
Cc: rick.p.edgecombe, vishal.l.verma
In-Reply-To: <20260307010358.819645-1-rick.p.edgecombe@intel.com>
From: Vishal Verma <vishal.l.verma@intel.com>
Some early TDX-capable platforms have an erratum where a partial write
to TDX private memory can cause a machine check on a subsequent read.
On these platforms, kexec and kdump have been disabled in these cases,
because the old kernel cannot safely hand off TDX state to the new
kernel. Later TDX modules support the TDH.SYS.DISABLE SEAMCALL, which
provides a way to cleanly disable TDX and allow kexec to proceed.
This can be a long running operation, and the time needed largely
depends on the amount of memory that has been allocated to TDs. If all
TDs have been destroyed prior to the sys_disable call, then it is fast,
with only needing to override the TDX module memory.
After the SEAMCALL completes, the TDX module is disabled and all memory
resources allocated to TDX are freed and reset. The next kernel can then
re-initialize the TDX module from scratch via the normal TDX bring-up
sequence.
The SEAMCALL may be interrupted by an interrupt. In this case, it
returns TDX_INTERRUPTED_RESUMABLE, and it must be retried in a loop
until the operation completes successfully.
Add a tdx_sys_disable() helper, which implements the retry loop around
the SEAMCALL to provide this functionality.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
arch/x86/include/asm/tdx.h | 3 +++
arch/x86/virt/vmx/tdx/tdx.c | 18 ++++++++++++++++++
arch/x86/virt/vmx/tdx/tdx.h | 1 +
3 files changed, 22 insertions(+)
diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
index f0826b0a512a..baaf43a09e99 100644
--- a/arch/x86/include/asm/tdx.h
+++ b/arch/x86/include/asm/tdx.h
@@ -173,6 +173,8 @@ static inline int pg_level_to_tdx_sept_level(enum pg_level level)
return level - 1;
}
+void tdx_sys_disable(void);
+
u64 tdh_vp_enter(struct tdx_vp *vp, struct tdx_module_args *args);
u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page);
u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2);
@@ -204,6 +206,7 @@ static inline void tdx_init(void) { }
static inline u32 tdx_get_nr_guest_keyids(void) { return 0; }
static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
static inline const struct tdx_sys_info *tdx_get_sysinfo(void) { return NULL; }
+static inline void tdx_sys_disable(void) { }
#endif /* CONFIG_INTEL_TDX_HOST */
#endif /* !__ASSEMBLER__ */
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index 0802d0fd18a4..68bd2618dde4 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -37,6 +37,7 @@
#include <asm/msr.h>
#include <asm/cpufeature.h>
#include <asm/tdx.h>
+#include <asm/shared/tdx_errno.h>
#include <asm/cpu_device_id.h>
#include <asm/processor.h>
#include <asm/mce.h>
@@ -1940,3 +1941,20 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
}
EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
+
+void tdx_sys_disable(void)
+{
+ struct tdx_module_args args = {};
+
+ /*
+ * SEAMCALLs that can return TDX_INTERRUPTED_RESUMABLE are guaranteed
+ * to make forward progress between interrupts, so it is safe to loop
+ * unconditionally here.
+ *
+ * This is a 'destructive' SEAMCALL, in that no other SEAMCALL can be
+ * run after this until a full reinitialization is done.
+ */
+ while (seamcall(TDH_SYS_DISABLE, &args) == TDX_INTERRUPTED_RESUMABLE)
+ ;
+}
+
diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h
index dde219c823b4..e2cf2dd48755 100644
--- a/arch/x86/virt/vmx/tdx/tdx.h
+++ b/arch/x86/virt/vmx/tdx/tdx.h
@@ -46,6 +46,7 @@
#define TDH_PHYMEM_PAGE_WBINVD 41
#define TDH_VP_WR 43
#define TDH_SYS_CONFIG 45
+#define TDH_SYS_DISABLE 69
/*
* SEAMCALL leaf:
--
2.53.0
^ permalink raw reply related
* [PATCH 4/4] KVM: x86: Disable the TDX module during kexec and kdump
From: Rick Edgecombe @ 2026-03-07 1:03 UTC (permalink / raw)
To: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, chao.gao, kai.huang, ackerleytng
Cc: rick.p.edgecombe, vishal.l.verma
In-Reply-To: <20260307010358.819645-1-rick.p.edgecombe@intel.com>
From: Vishal Verma <vishal.l.verma@intel.com>
Use the TDH.SYS.DISABLE SEAMCALL, which disables the TDX module,
reclaims all memory resources assigned to TDX, and clears any
partial-write induced poison, to allow kexec and kdump on platforms with
the partial write errata.
On TDX-capable platforms with the partial write erratum, kexec has been
disabled because the new kernel could hit a machine check reading a
previously poisoned memory location.
Later TDX modules support TDH.SYS.DISABLE, which disables the module and
reclaims all TDX memory resources, allowing the new kernel to re-initialize
TDX from scratch. This operation also clears the old memory, cleaning up
any poison.
Add tdx_sys_disable() to tdx_shutdown(), which is called in the
syscore_shutdown path for kexec. This is done just before tdx_shutdown()
disables VMX on all CPUs.
For kdump, call tdx_sys_disable() in the crash path before
x86_virt_emergency_disable_virtualization_cpu() does VMXOFF.
Since this clears any poison on TDX-managed memory, the
X86_BUG_TDX_PW_MCE check in machine_kexec() that blocked kexec on
partial write errata platforms can be removed.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
---
arch/x86/kernel/crash.c | 2 ++
arch/x86/kernel/machine_kexec_64.c | 16 ----------------
arch/x86/virt/vmx/tdx/tdx.c | 1 +
3 files changed, 3 insertions(+), 16 deletions(-)
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index cd796818d94d..623d4474631a 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -38,6 +38,7 @@
#include <linux/kdebug.h>
#include <asm/cpu.h>
#include <asm/reboot.h>
+#include <asm/tdx.h>
#include <asm/intel_pt.h>
#include <asm/crash.h>
#include <asm/cmdline.h>
@@ -112,6 +113,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
crash_smp_send_stop();
+ tdx_sys_disable();
x86_virt_emergency_disable_virtualization_cpu();
/*
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 0590d399d4f1..c3f4a389992d 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -347,22 +347,6 @@ int machine_kexec_prepare(struct kimage *image)
unsigned long reloc_end = (unsigned long)__relocate_kernel_end;
int result;
- /*
- * Some early TDX-capable platforms have an erratum. A kernel
- * partial write (a write transaction of less than cacheline
- * lands at memory controller) to TDX private memory poisons that
- * memory, and a subsequent read triggers a machine check.
- *
- * On those platforms the old kernel must reset TDX private
- * memory before jumping to the new kernel otherwise the new
- * kernel may see unexpected machine check. For simplicity
- * just fail kexec/kdump on those platforms.
- */
- if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE)) {
- pr_info_once("Not allowed on platform with tdx_pw_mce bug\n");
- return -EOPNOTSUPP;
- }
-
/* Setup the identity mapped 64bit page table */
result = init_pgtable(image, __pa(control_page));
if (result)
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index 68bd2618dde4..b388fbce5d76 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -252,6 +252,7 @@ static void tdx_shutdown_cpu(void *ign)
static void tdx_shutdown(void *ign)
{
+ tdx_sys_disable();
on_each_cpu(tdx_shutdown_cpu, NULL, 1);
}
--
2.53.0
^ permalink raw reply related
* [PATCH 2/4] x86/vmware: Log kmsg dump on panic
From: Alexey Makhalov @ 2026-03-07 0:42 UTC (permalink / raw)
To: x86, virtualization, bp, hpa, dave.hansen, mingo, tglx
Cc: ajay.kaher, brennan.lamoreaux, bo.gan, bcm-kernel-feedback-list,
linux-kernel, kas, rick.p.edgecombe, linux-coco, Alexey Makhalov
In-Reply-To: <20260307004238.1181299-1-alexey.makhalov@broadcom.com>
Improve debugability of VMware Linux guests by dumping
kernel messages during a panic to VM log file (vmware.log).
Co-developed-by: Bo Gan <bo.gan@broadcom.com>
Signed-off-by: Bo Gan <bo.gan@broadcom.com>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
---
arch/x86/include/asm/vmware.h | 1 +
arch/x86/kernel/cpu/vmware.c | 132 ++++++++++++++++++++++++++++++++++
2 files changed, 133 insertions(+)
diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h
index 6a084e088b30..c23164503e54 100644
--- a/arch/x86/include/asm/vmware.h
+++ b/arch/x86/include/asm/vmware.h
@@ -93,6 +93,7 @@
#define VMWARE_HYPERVISOR_MAGIC 0x564d5868U
#define VMWARE_CMD_GETVERSION 10
+#define VMWARE_CMD_MESSAGE 30
#define VMWARE_CMD_GETHZ 45
#define VMWARE_CMD_GETVCPU_INFO 68
#define VMWARE_CMD_STEALCLOCK 91
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 93acd3414e37..d9753b1aba58 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -30,6 +30,7 @@
#include <linux/reboot.h>
#include <linux/static_call.h>
#include <linux/sched/cputime.h>
+#include <linux/kmsg_dump.h>
#include <asm/div64.h>
#include <asm/x86_init.h>
#include <asm/hypervisor.h>
@@ -211,6 +212,13 @@ static unsigned long vmware_get_tsc_khz(void)
return vmware_tsc_khz;
}
+static void kmsg_dumper_vmware_log(struct kmsg_dumper *dumper,
+ struct kmsg_dump_detail *detail);
+
+static struct kmsg_dumper kmsg_dumper = {
+ .dump = kmsg_dumper_vmware_log
+};
+
#ifdef CONFIG_PARAVIRT
static struct cyc2ns_data vmware_cyc2ns __ro_after_init;
static bool vmw_sched_clock __initdata = true;
@@ -535,6 +543,8 @@ static void __init vmware_platform_setup(void)
#endif
vmware_set_capabilities();
+
+ kmsg_dump_register(&kmsg_dumper);
}
static u8 __init get_hypercall_mode(void)
@@ -630,3 +640,125 @@ const __initconst struct hypervisor_x86 x86_hyper_vmware = {
.runtime.sev_es_hcall_finish = vmware_sev_es_hcall_finish,
#endif
};
+
+#define VMWARE_HB_CMD_MESSAGE 0
+#define MESSAGE_STATUS_SUCCESS (0x01 << 16)
+#define MESSAGE_STATUS_CPT (0x10 << 16)
+#define MESSAGE_STATUS_HB (0x80 << 16)
+
+#define RPCI_PROTOCOL_NUM 0x49435052 /* 'RPCI' */
+#define GUESTMSG_FLAG_COOKIE 0x80000000
+
+#define MESSAGE_TYPE_OPEN (0 << 16)
+#define MESSAGE_TYPE_SENDSIZE (1 << 16)
+#define MESSAGE_TYPE_SEND (2 << 16)
+#define MESSAGE_TYPE_CLOSE (6 << 16)
+
+struct vmw_msg {
+ u32 id;
+ u32 cookie_high;
+ u32 cookie_low;
+};
+
+static int
+vmware_log_open(struct vmw_msg *msg)
+{
+ u32 info;
+
+ vmware_hypercall6(VMWARE_CMD_MESSAGE | MESSAGE_TYPE_OPEN,
+ RPCI_PROTOCOL_NUM | GUESTMSG_FLAG_COOKIE,
+ 0, &info, &msg->id, &msg->cookie_high,
+ &msg->cookie_low);
+
+ if ((info & MESSAGE_STATUS_SUCCESS) == 0)
+ return 1;
+
+ msg->id &= 0xffff0000UL;
+ return 0;
+}
+
+static int
+vmware_log_close(struct vmw_msg *msg)
+{
+ u32 info;
+
+ vmware_hypercall5(VMWARE_CMD_MESSAGE | MESSAGE_TYPE_CLOSE, 0, msg->id,
+ msg->cookie_high, msg->cookie_low, &info);
+
+ if ((info & MESSAGE_STATUS_SUCCESS) == 0)
+ return 1;
+ return 0;
+}
+
+static int
+vmware_log_send(struct vmw_msg *msg, const char *string)
+{
+ u32 info;
+ u32 len = strlen(string);
+
+retry:
+ vmware_hypercall5(VMWARE_CMD_MESSAGE | MESSAGE_TYPE_SENDSIZE, len,
+ msg->id, msg->cookie_high, msg->cookie_low, &info);
+
+ if (!(info & MESSAGE_STATUS_SUCCESS))
+ return 1;
+
+ /* HB port can't access encrypted memory. */
+ if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT) && (info & MESSAGE_STATUS_HB)) {
+ vmware_hypercall_hb_out(
+ VMWARE_HB_CMD_MESSAGE | MESSAGE_STATUS_SUCCESS,
+ len, msg->id, (uintptr_t) string, msg->cookie_low,
+ msg->cookie_high, &info);
+ } else {
+ do {
+ u32 word;
+ size_t s = min_t(u32, len, sizeof(word));
+
+ memcpy(&word, string, s);
+ len -= s;
+ string += s;
+
+ vmware_hypercall5(VMWARE_CMD_MESSAGE | MESSAGE_TYPE_SEND,
+ word, msg->id, msg->cookie_high,
+ msg->cookie_low, &info);
+ } while (len && (info & MESSAGE_STATUS_SUCCESS));
+ }
+
+ if ((info & MESSAGE_STATUS_SUCCESS) == 0) {
+ if (info & MESSAGE_STATUS_CPT)
+ /* A checkpoint occurred. Retry. */
+ goto retry;
+ return 1;
+ }
+ return 0;
+}
+STACK_FRAME_NON_STANDARD(vmware_log_send);
+
+/**
+ * kmsg_dumper_vmware_log - dumps kmsg to vmware.log file on the host
+ */
+static void kmsg_dumper_vmware_log(struct kmsg_dumper *dumper,
+ struct kmsg_dump_detail *detail)
+{
+ struct vmw_msg msg;
+ struct kmsg_dump_iter iter;
+ static char line[1024];
+ size_t len = 0;
+
+ /* Line prefix to send to VM log file. */
+ line[0] = 'l';
+ line[1] = 'o';
+ line[2] = 'g';
+ line[3] = ' ';
+
+ kmsg_dump_rewind(&iter);
+ while (kmsg_dump_get_line(&iter, true, line + 4, sizeof(line) - 4,
+ &len)) {
+ line[len + 4] = '\0';
+ if (vmware_log_open(&msg))
+ return;
+ if (vmware_log_send(&msg, line))
+ return;
+ vmware_log_close(&msg);
+ }
+}
--
2.43.7
^ permalink raw reply related
* [PATCH 3/4] x86/vmware: Report guest crash to the hypervisor
From: Alexey Makhalov @ 2026-03-07 0:42 UTC (permalink / raw)
To: x86, virtualization, bp, hpa, dave.hansen, mingo, tglx
Cc: ajay.kaher, brennan.lamoreaux, bo.gan, bcm-kernel-feedback-list,
linux-kernel, kas, rick.p.edgecombe, linux-coco, Alexey Makhalov
In-Reply-To: <20260307004238.1181299-1-alexey.makhalov@broadcom.com>
Register the guest crash reporter to panic_notifier_list,
which will be called at panic time. Guest crash reporter
will report the crash to the hypervisor through
a hypercall.
Co-developed-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
Signed-off-by: Brennan Lamoreaux <brennan.lamoreaux@broadcom.com>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
---
arch/x86/include/asm/vmware.h | 1 +
arch/x86/kernel/cpu/vmware.c | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h
index c23164503e54..bf6141353774 100644
--- a/arch/x86/include/asm/vmware.h
+++ b/arch/x86/include/asm/vmware.h
@@ -97,6 +97,7 @@
#define VMWARE_CMD_GETHZ 45
#define VMWARE_CMD_GETVCPU_INFO 68
#define VMWARE_CMD_STEALCLOCK 91
+#define VMWARE_CMD_REPORTGUESTCRASH 102
/*
* Hypercall command mask:
* bits [6:0] command, range [0, 127]
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index d9753b1aba58..8997295a5a5c 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -31,6 +31,7 @@
#include <linux/static_call.h>
#include <linux/sched/cputime.h>
#include <linux/kmsg_dump.h>
+#include <linux/panic_notifier.h>
#include <asm/div64.h>
#include <asm/x86_init.h>
#include <asm/hypervisor.h>
@@ -451,6 +452,24 @@ static void __init vmware_paravirt_ops_setup(void)
#define vmware_paravirt_ops_setup() do {} while (0)
#endif
+static int vmware_report_guest_crash(struct notifier_block *self,
+ unsigned long action, void *data)
+{
+ vmware_hypercall1(VMWARE_CMD_REPORTGUESTCRASH, 0);
+ return 0;
+}
+
+static struct notifier_block guest_crash_reporter = {
+ .notifier_call = vmware_report_guest_crash
+};
+
+static int __init register_guest_crash_reporter(void)
+{
+ atomic_notifier_chain_register(&panic_notifier_list,
+ &guest_crash_reporter);
+
+ return 0;
+}
/*
* VMware hypervisor takes care of exporting a reliable TSC to the guest.
* Still, due to timing difference when running on virtual cpus, the TSC can
@@ -545,6 +564,8 @@ static void __init vmware_platform_setup(void)
vmware_set_capabilities();
kmsg_dump_register(&kmsg_dumper);
+
+ register_guest_crash_reporter();
}
static u8 __init get_hypercall_mode(void)
--
2.43.7
^ permalink raw reply related
* [PATCH 0/4] x86/vmware: Hypercall refactoring and improved guest support
From: Alexey Makhalov @ 2026-03-07 0:42 UTC (permalink / raw)
To: x86, virtualization, bp, hpa, dave.hansen, mingo, tglx
Cc: ajay.kaher, brennan.lamoreaux, bo.gan, bcm-kernel-feedback-list,
linux-kernel, kas, rick.p.edgecombe, linux-coco, Alexey Makhalov
This series improves VMware guest support on x86 by refactoring the
hypercall infrastructure and adding better crash diagnostics, along
with encrypted guest support for the steal time clock.
The first patch introduces a common vmware_hypercall() backend selected
via static calls. It consolidates the existing hypercall mechanisms
(backdoor, VMCALL/VMMCALL, and TDX) behind a single interface and
selects the optimal implementation at boot. This reduces duplication
and simplifies future extensions.
Building on top of the new hypercall infrastructure, the next two
patches improve post-mortem debugging of VMware guests. They export
panic information to the hypervisor by dumping kernel messages to the
VM vmware.log on the host and explicitly reporting guest crash event
to the hypervisor.
The final patch adds support for encrypted guests by ensuring that the
shared memory used for the steal time clock is mapped as decrypted
before being shared with the hypervisor. This enables steal time
accounting to function correctly when guest memory encryption is
enabled.
Patch overview:
1. x86/vmware: Introduce common vmware_hypercall
* Consolidate hypercall implementations behind a common API
* Select backend via static_call at boot
2. x86/vmware: Log kmsg dump on panic
* Register a kmsg dumper
* Export panic logs to the host
3. x86/vmware: Report guest crash to the hypervisor
* Register a panic notifier
* Notify the hypervisor about guest crashes
4. x86/vmware: Support steal time clock for encrypted guests
* Mark shared steal time memory as decrypted early in boot
Alexey Makhalov (4):
x86/vmware: Introduce common vmware_hypercall()
x86/vmware: Log kmsg dump on panic
x86/vmware: Report guest crash to the hypervisor
x86/vmware: Support steal time clock for encrypted guests
arch/x86/include/asm/vmware.h | 276 ++++++++------------
arch/x86/kernel/cpu/vmware.c | 470 +++++++++++++++++++++++++---------
2 files changed, 463 insertions(+), 283 deletions(-)
--
2.43.7
^ permalink raw reply
* [PATCH 1/4] x86/vmware: Introduce common vmware_hypercall()
From: Alexey Makhalov @ 2026-03-07 0:42 UTC (permalink / raw)
To: x86, virtualization, bp, hpa, dave.hansen, mingo, tglx
Cc: ajay.kaher, brennan.lamoreaux, bo.gan, bcm-kernel-feedback-list,
linux-kernel, kas, rick.p.edgecombe, linux-coco, Alexey Makhalov,
Linus Torvalds
In-Reply-To: <20260307004238.1181299-1-alexey.makhalov@broadcom.com>
Introduce vmware_hypercall(), a unified low-bandwidth VMware hypercall
API, and convert the static inlines vmware_hypercallX() family into thin
wrappers on top of it.
vmware_hypercall() is implemented as a static call with four backend
implementations: backdoor, vmcall, vmmcall, and tdxcall. All share the
same logical API but differ in their underlying register mappings.
By updating the static call target early during boot, before the first
hypercall is issued, the !alternatives_patched case no longer needs to
be handled. This allows removal of vmware_hypercall_slow().
The new API implements the widest practical hypercall use case: up to
six input and six output arguments. While this may be slightly less
efficient due to clobbering all six registers and moving unused
arguments - it avoids subtle ABI issues, including cases where other
hypervisors implementing VMware hypercalls corrupt registers.
See QEMU issue #3293 ("vmmouse driver corrupts upper 32 bits of
registers on x86-64") for an example of such behavior.
Additionally, enhance the VMware hypercall ABI documentation in
<asm/vmware.h>.
Link: https://gitlab.com/qemu-project/qemu/-/issues/3293
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
---
arch/x86/include/asm/vmware.h | 274 ++++++++++++++-------------------
arch/x86/kernel/cpu/vmware.c | 276 +++++++++++++++++++---------------
2 files changed, 267 insertions(+), 283 deletions(-)
diff --git a/arch/x86/include/asm/vmware.h b/arch/x86/include/asm/vmware.h
index 4220dae14a2d..6a084e088b30 100644
--- a/arch/x86/include/asm/vmware.h
+++ b/arch/x86/include/asm/vmware.h
@@ -3,48 +3,84 @@
#define _ASM_X86_VMWARE_H
#include <asm/cpufeatures.h>
-#include <asm/alternative.h>
#include <linux/stringify.h>
+#include <linux/static_call.h>
/*
* VMware hypercall ABI.
*
- * - Low bandwidth (LB) hypercalls (I/O port based, vmcall and vmmcall)
- * have up to 6 input and 6 output arguments passed and returned using
- * registers: %eax (arg0), %ebx (arg1), %ecx (arg2), %edx (arg3),
- * %esi (arg4), %edi (arg5).
- * The following input arguments must be initialized by the caller:
- * arg0 - VMWARE_HYPERVISOR_MAGIC
- * arg2 - Hypercall command
- * arg3 bits [15:0] - Port number, LB and direction flags
+ * - Low bandwidth (LB) hypercalls: I/O port based (aka backdoor), vmcall and
+ * vmmcall have up to 6 input and 6 output on registers arguments, with the
+ * register mapping:
+ * +------+----------------------------------------+-----------------+
+ * | Reg | Input argument | Output argument |
+ * +======+========================================+=================+
+ * | %eax | VMWARE_HYPERVISOR_MAGIC | out0 |
+ * +------+----------------------------------------+-----------------+
+ * | %ebx | (in1) | out1 |
+ * +------+----------------------------------------+-----------------+
+ * | %ecx | (cmd) - Hypercall command | out2 |
+ * +------+----------------------------------------+-----------------+
+ * | %edx | Bits [15:0] - Port number for backdoor | out3 |
+ * | | Zero for vmcall/vmmcall | |
+ * | | Bits [31:16] - (in3) | |
+ * +------+----------------------------------------+-----------------+
+ * | %esi | (in4) | out4 |
+ * +------+----------------------------------------+-----------------+
+ * | %edi | (in5) | out5 |
+ * +------+----------------------------------------+-----------------+
*
- * - Low bandwidth TDX hypercalls (x86_64 only) are similar to LB
- * hypercalls. They also have up to 6 input and 6 output on registers
- * arguments, with different argument to register mapping:
- * %r12 (arg0), %rbx (arg1), %r13 (arg2), %rdx (arg3),
- * %rsi (arg4), %rdi (arg5).
+ * - Low bandwidth TDX hypercalls (x86_64 only) are similar to LB hypercalls.
+ * They also have up to 6 input and 6 output on registers arguments, with
+ * different argument to register mapping:
+ * +------+----------------------------------------+-----------------+
+ * | Reg | Input argument | Output argument |
+ * +======+========================================+=================+
+ * | %r12 | VMWARE_HYPERVISOR_MAGIC | out0 |
+ * +------+----------------------------------------+-----------------+
+ * | %ebx | (in1) | out1 |
+ * +------+----------------------------------------+-----------------+
+ * | %r13 | (cmd) - Hypercall command | out2 |
+ * +------+----------------------------------------+-----------------+
+ * | %edx | Bits [15:0] - Must be zero | out3 |
+ * | | Bits [31:16] - (in3) | |
+ * +------+----------------------------------------+-----------------+
+ * | %esi | (in4) | out4 |
+ * +------+----------------------------------------+-----------------+
+ * | %edi | (in5) | out5 |
+ * +------+----------------------------------------+-----------------+
*
- * - High bandwidth (HB) hypercalls are I/O port based only. They have
- * up to 7 input and 7 output arguments passed and returned using
- * registers: %eax (arg0), %ebx (arg1), %ecx (arg2), %edx (arg3),
- * %esi (arg4), %edi (arg5), %ebp (arg6).
- * The following input arguments must be initialized by the caller:
- * arg0 - VMWARE_HYPERVISOR_MAGIC
- * arg1 - Hypercall command
- * arg3 bits [15:0] - Port number, HB and direction flags
+ * - High bandwidth (HB) hypercalls are I/O port based only. They have up to 7
+ * input and 7 output on reegister arguments with the following mapping:
+ * +------+----------------------------------------+-----------------+
+ * | Reg | Input argument | Output argument |
+ * +======+========================================+=================+
+ * | %eax | VMWARE_HYPERVISOR_MAGIC | out0 |
+ * +------+----------------------------------------+-----------------+
+ * | %ebx | (cmd) - Hypercall command | out1 |
+ * +------+----------------------------------------+-----------------+
+ * | %ebx | (in2) | out2 |
+ * +------+----------------------------------------+-----------------+
+ * | %edx | Bits [15:0] - Port number and HB flag | out3 |
+ * | | Bits [31:16] - (in3) | |
+ * +------+----------------------------------------+-----------------+
+ * | %esi | (in4) | out4 |
+ * +------+----------------------------------------+-----------------+
+ * | %edi | (in5) | out5 |
+ * +------+----------------------------------------+-----------------+
+ * | %ebp | (in6) | out6 |
+ * +------+----------------------------------------+-----------------+
*
- * For compatibility purposes, x86_64 systems use only lower 32 bits
- * for input and output arguments.
+ * For compatibility purposes, x86_64 systems use only lower 32 bits for input
+ * and output arguments.
*
- * The hypercall definitions differ in the low word of the %edx (arg3)
- * in the following way: the old I/O port based interface uses the port
- * number to distinguish between high- and low bandwidth versions, and
- * uses IN/OUT instructions to define transfer direction.
+ * The hypercall definitions differ in the low word of the %edx (arg3) in the
+ * following way: the old I/O port based interface uses the port number, the
+ * bandwidth mode flag, and uses IN/OUT instructions to define transfer
+ * direction.
*
- * The new vmcall interface instead uses a set of flags to select
- * bandwidth mode and transfer direction. The flags should be loaded
- * into arg3 by any user and are automatically replaced by the port
- * number if the I/O port method is used.
+ * The new vmcall interface instead uses a set of flags to select bandwidth
+ * mode and transfer direction.
*/
#define VMWARE_HYPERVISOR_HB BIT(0)
@@ -70,103 +106,64 @@
#define CPUID_VMWARE_FEATURES_ECX_VMMCALL BIT(0)
#define CPUID_VMWARE_FEATURES_ECX_VMCALL BIT(1)
-extern unsigned long vmware_hypercall_slow(unsigned long cmd,
- unsigned long in1, unsigned long in3,
- unsigned long in4, unsigned long in5,
- u32 *out1, u32 *out2, u32 *out3,
- u32 *out4, u32 *out5);
-
#define VMWARE_TDX_VENDOR_LEAF 0x1af7e4909ULL
#define VMWARE_TDX_HCALL_FUNC 1
-extern unsigned long vmware_tdx_hypercall(unsigned long cmd,
- unsigned long in1, unsigned long in3,
- unsigned long in4, unsigned long in5,
- u32 *out1, u32 *out2, u32 *out3,
- u32 *out4, u32 *out5);
+unsigned long dummy_vmware_hypercall(unsigned long cmd,
+ unsigned long in1, unsigned long in3,
+ unsigned long in4, unsigned long in5,
+ u32 *out1, u32 *out2, u32 *out3,
+ u32 *out4, u32 *out5);
/*
- * The low bandwidth call. The low word of %edx is presumed to have OUT bit
- * set. The high word of %edx may contain input data from the caller.
+ * Low bandwidth (LB) VMware hypercall.
+ *
+ * It is backed by the backdoor, vmcall, vmmcall or tdx call implementation.
+ *
+ * Use inX/outX arguments naming as the register mappings vary between
+ * different implementations. See VMware hypercall ABI above.
+ * These 10 arguments could be nicely wrapped in in/out structures, but it
+ * will introduce unnecessary structs copy in vmware_tdx_hypercall().
+ *
+ * NOTE:
+ * Do not merge vmware_{backdoor,vmcall,vmmcall}_hypercall implementations
+ * using alternative instructions. Such patching mechanism can not be used
+ * in vmware_hypercall path, as the first hypercall will be called much
+ * before the apply_alternatives(). See vmware_platform_setup().
*/
-#define VMWARE_HYPERCALL \
- ALTERNATIVE_2("movw %[port], %%dx\n\t" \
- "inl (%%dx), %%eax", \
- "vmcall", X86_FEATURE_VMCALL, \
- "vmmcall", X86_FEATURE_VMW_VMMCALL)
+DECLARE_STATIC_CALL(vmware_hypercall, dummy_vmware_hypercall);
+/*
+ * Set of commonly used vmware_hypercallX functions - wrappers on top of the
+ * vmware_hypercall.
+ */
static inline
unsigned long vmware_hypercall1(unsigned long cmd, unsigned long in1)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, 0, 0, 0,
- NULL, NULL, NULL, NULL, NULL);
-
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, 0, 0, 0,
- NULL, NULL, NULL, NULL, NULL);
+ u32 out1, out2, out3, out4, out5;
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (0)
- : "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0,
+ &out1, &out2, &out3, &out4, &out5);
}
static inline
unsigned long vmware_hypercall3(unsigned long cmd, unsigned long in1,
u32 *out1, u32 *out2)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, 0, 0, 0,
- out1, out2, NULL, NULL, NULL);
-
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, 0, 0, 0,
- out1, out2, NULL, NULL, NULL);
+ u32 out3, out4, out5;
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0), "=b" (*out1), "=c" (*out2)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (0)
- : "di", "si", "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0,
+ out1, out2, &out3, &out4, &out5);
}
static inline
unsigned long vmware_hypercall4(unsigned long cmd, unsigned long in1,
u32 *out1, u32 *out2, u32 *out3)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, 0, 0, 0,
- out1, out2, out3, NULL, NULL);
-
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, 0, 0, 0,
- out1, out2, out3, NULL, NULL);
+ u32 out4, out5;
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0), "=b" (*out1), "=c" (*out2), "=d" (*out3)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (0)
- : "di", "si", "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, 0, 0, 0,
+ out1, out2, out3, &out4, &out5);
}
static inline
@@ -174,27 +171,10 @@ unsigned long vmware_hypercall5(unsigned long cmd, unsigned long in1,
unsigned long in3, unsigned long in4,
unsigned long in5, u32 *out2)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, in3, in4, in5,
- NULL, out2, NULL, NULL, NULL);
+ u32 out1, out3, out4, out5;
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, in3, in4, in5,
- NULL, out2, NULL, NULL, NULL);
-
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0), "=c" (*out2)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3),
- "S" (in4),
- "D" (in5)
- : "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, in3, in4, in5,
+ &out1, out2, &out3, &out4, &out5);
}
static inline
@@ -202,26 +182,10 @@ unsigned long vmware_hypercall6(unsigned long cmd, unsigned long in1,
unsigned long in3, u32 *out2,
u32 *out3, u32 *out4, u32 *out5)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, in3, 0, 0,
- NULL, out2, out3, out4, out5);
+ u32 out1;
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, in3, 0, 0,
- NULL, out2, out3, out4, out5);
-
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0), "=c" (*out2), "=d" (*out3), "=S" (*out4),
- "=D" (*out5)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3)
- : "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, in3, 0, 0,
+ &out1, out2, out3, out4, out5);
}
static inline
@@ -230,27 +194,10 @@ unsigned long vmware_hypercall7(unsigned long cmd, unsigned long in1,
unsigned long in5, u32 *out1,
u32 *out2, u32 *out3)
{
- unsigned long out0;
-
- if (cpu_feature_enabled(X86_FEATURE_TDX_GUEST))
- return vmware_tdx_hypercall(cmd, in1, in3, in4, in5,
- out1, out2, out3, NULL, NULL);
+ u32 out4, out5;
- if (unlikely(!alternatives_patched) && !__is_defined(MODULE))
- return vmware_hypercall_slow(cmd, in1, in3, in4, in5,
- out1, out2, out3, NULL, NULL);
-
- asm_inline volatile (VMWARE_HYPERCALL
- : "=a" (out0), "=b" (*out1), "=c" (*out2), "=d" (*out3)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3),
- "S" (in4),
- "D" (in5)
- : "cc", "memory");
- return out0;
+ return static_call_mod(vmware_hypercall)(cmd, in1, in3, in4, in5,
+ out1, out2, out3, &out4, &out5);
}
#ifdef CONFIG_X86_64
@@ -322,6 +269,5 @@ unsigned long vmware_hypercall_hb_in(unsigned long cmd, unsigned long in2,
return out0;
}
#undef VMW_BP_CONSTRAINT
-#undef VMWARE_HYPERCALL
#endif
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index a3e6936839b1..93acd3414e37 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -64,70 +64,140 @@ struct vmware_steal_time {
};
static unsigned long vmware_tsc_khz __ro_after_init;
-static u8 vmware_hypercall_mode __ro_after_init;
-
-unsigned long vmware_hypercall_slow(unsigned long cmd,
- unsigned long in1, unsigned long in3,
- unsigned long in4, unsigned long in5,
- u32 *out1, u32 *out2, u32 *out3,
- u32 *out4, u32 *out5)
-{
- unsigned long out0, rbx, rcx, rdx, rsi, rdi;
-
- switch (vmware_hypercall_mode) {
- case CPUID_VMWARE_FEATURES_ECX_VMCALL:
- asm_inline volatile ("vmcall"
- : "=a" (out0), "=b" (rbx), "=c" (rcx),
- "=d" (rdx), "=S" (rsi), "=D" (rdi)
- : "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3),
- "S" (in4),
- "D" (in5)
- : "cc", "memory");
- break;
- case CPUID_VMWARE_FEATURES_ECX_VMMCALL:
- asm_inline volatile ("vmmcall"
- : "=a" (out0), "=b" (rbx), "=c" (rcx),
- "=d" (rdx), "=S" (rsi), "=D" (rdi)
- : "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3),
- "S" (in4),
- "D" (in5)
- : "cc", "memory");
- break;
- default:
- asm_inline volatile ("movw %[port], %%dx; inl (%%dx), %%eax"
- : "=a" (out0), "=b" (rbx), "=c" (rcx),
- "=d" (rdx), "=S" (rsi), "=D" (rdi)
- : [port] "i" (VMWARE_HYPERVISOR_PORT),
- "a" (VMWARE_HYPERVISOR_MAGIC),
- "b" (in1),
- "c" (cmd),
- "d" (in3),
- "S" (in4),
- "D" (in5)
- : "cc", "memory");
- break;
- }
+static u8 vmware_hypercall_mode __initdata;
+
+static unsigned long vmware_backdoor_hypercall(unsigned long cmd,
+ unsigned long in1, unsigned long in3,
+ unsigned long in4, unsigned long in5,
+ u32 *out1, u32 *out2, u32 *out3,
+ u32 *out4, u32 *out5)
+{
+ unsigned long out0;
+
+ /* The low word of in3(%edx) must have the backdoor port number */
+ in3 = (in3 & ~0xffff) | VMWARE_HYPERVISOR_PORT;
+
+ asm_inline volatile ("inl (%%dx), %%eax"
+ : "=a" (out0), "=b" (*out1), "=c" (*out2),
+ "=d" (*out3), "=S" (*out4), "=D" (*out5)
+ : "a" (VMWARE_HYPERVISOR_MAGIC),
+ "b" (in1),
+ "c" (cmd),
+ "d" (in3),
+ "S" (in4),
+ "D" (in5)
+ : "cc", "memory");
- if (out1)
- *out1 = rbx;
- if (out2)
- *out2 = rcx;
- if (out3)
- *out3 = rdx;
- if (out4)
- *out4 = rsi;
- if (out5)
- *out5 = rdi;
+ return out0;
+}
+
+static unsigned long vmware_vmcall_hypercall(unsigned long cmd,
+ unsigned long in1, unsigned long in3,
+ unsigned long in4, unsigned long in5,
+ u32 *out1, u32 *out2, u32 *out3,
+ u32 *out4, u32 *out5)
+{
+ unsigned long out0;
+
+ /* The low word of in3(%edx) must be zero: LB, IN */
+ in3 &= ~0xffff;
+
+ asm_inline volatile ("vmcall"
+ : "=a" (out0), "=b" (*out1), "=c" (*out2),
+ "=d" (*out3), "=S" (*out4), "=D" (*out5)
+ : "a" (VMWARE_HYPERVISOR_MAGIC),
+ "b" (in1),
+ "c" (cmd),
+ "d" (in3),
+ "S" (in4),
+ "D" (in5)
+ : "cc", "memory");
return out0;
}
+static unsigned long vmware_vmmcall_hypercall(unsigned long cmd,
+ unsigned long in1, unsigned long in3,
+ unsigned long in4, unsigned long in5,
+ u32 *out1, u32 *out2, u32 *out3,
+ u32 *out4, u32 *out5)
+{
+ unsigned long out0;
+
+ /* The low word of in3(%edx) must be zero: LB, IN */
+ in3 &= ~0xffff;
+
+ asm_inline volatile ("vmmcall"
+ : "=a" (out0), "=b" (*out1), "=c" (*out2),
+ "=d" (*out3), "=S" (*out4), "=D" (*out5)
+ : "a" (VMWARE_HYPERVISOR_MAGIC),
+ "b" (in1),
+ "c" (cmd),
+ "d" (in3),
+ "S" (in4),
+ "D" (in5)
+ : "cc", "memory");
+
+ return out0;
+}
+
+/*
+ * TDCALL[TDG.VP.VMCALL] uses %rax (arg0) and %rcx (arg2). Therefore,
+ * we remap those registers to %r12 and %r13, respectively.
+ */
+static unsigned long vmware_tdx_hypercall(unsigned long cmd,
+ unsigned long in1, unsigned long in3,
+ unsigned long in4, unsigned long in5,
+ u32 *out1, u32 *out2, u32 *out3,
+ u32 *out4, u32 *out5)
+{
+#ifdef CONFIG_INTEL_TDX_GUEST
+ struct tdx_module_args args = {};
+
+ if (!hypervisor_is_type(X86_HYPER_VMWARE)) {
+ pr_warn_once("Incorrect usage\n");
+ return ULONG_MAX;
+ }
+
+ if (cmd & ~VMWARE_CMD_MASK) {
+ pr_warn_once("Out of range command %lx\n", cmd);
+ return ULONG_MAX;
+ }
+
+ args.rbx = in1;
+ /* The low word of in3(%rdx) must be zero: LB, IN */
+ args.rdx = in3 & ~0xffff;
+ args.rsi = in4;
+ args.rdi = in5;
+ args.r10 = VMWARE_TDX_VENDOR_LEAF;
+ args.r11 = VMWARE_TDX_HCALL_FUNC;
+ args.r12 = VMWARE_HYPERVISOR_MAGIC;
+ args.r13 = cmd;
+ /* CPL */
+ args.r15 = 0;
+
+ __tdx_hypercall(&args);
+
+ *out1 = args.rbx;
+ *out2 = args.r13;
+ *out3 = args.rdx;
+ *out4 = args.rsi;
+ *out5 = args.rdi;
+
+ return args.r12;
+#else
+ return ULONG_MAX;
+#endif
+}
+
+
+DEFINE_STATIC_CALL(vmware_hypercall, vmware_backdoor_hypercall);
+EXPORT_STATIC_CALL_GPL(vmware_hypercall);
+
+/*
+ * Perform backdoor probbing of the hypervisor when
+ * X86_FEATURE_HYPERVISOR bit is not set.
+ */
static inline int __vmware_platform(void)
{
u32 eax, ebx, ecx;
@@ -397,11 +467,35 @@ static void __init vmware_set_capabilities(void)
setup_force_cpu_cap(X86_FEATURE_VMW_VMMCALL);
}
+static void __init vmware_select_hypercall(void)
+{
+ char *mode;
+
+ if (IS_ENABLED(CONFIG_INTEL_TDX_GUEST) &&
+ cpu_feature_enabled(X86_FEATURE_TDX_GUEST)) {
+ static_call_update(vmware_hypercall, vmware_tdx_hypercall);
+ mode = "tdcall";
+ } else if (vmware_hypercall_mode == CPUID_VMWARE_FEATURES_ECX_VMCALL) {
+ static_call_update(vmware_hypercall, vmware_vmcall_hypercall);
+ mode = "vmcall";
+ } else if (vmware_hypercall_mode == CPUID_VMWARE_FEATURES_ECX_VMMCALL) {
+ static_call_update(vmware_hypercall, vmware_vmmcall_hypercall);
+ mode = "vmmcall";
+ } else {
+ mode = "backdoor";
+ }
+
+ pr_info("hypercall mode: %s\n", mode);
+}
+
static void __init vmware_platform_setup(void)
{
u32 eax, ebx, ecx;
u64 lpj, tsc_khz;
+ /* Update vmware_hypercall() before the first use. */
+ vmware_select_hypercall();
+
eax = vmware_hypercall3(VMWARE_CMD_GETHZ, UINT_MAX, &ebx, &ecx);
if (ebx != UINT_MAX) {
@@ -443,7 +537,7 @@ static void __init vmware_platform_setup(void)
vmware_set_capabilities();
}
-static u8 __init vmware_select_hypercall(void)
+static u8 __init get_hypercall_mode(void)
{
int eax, ebx, ecx, edx;
@@ -456,8 +550,8 @@ static u8 __init vmware_select_hypercall(void)
* While checking the dmi string information, just checking the product
* serial key should be enough, as this will always have a VMware
* specific string when running under VMware hypervisor.
- * If !boot_cpu_has(X86_FEATURE_HYPERVISOR), vmware_hypercall_mode
- * intentionally defaults to 0.
+ * If !boot_cpu_has(X86_FEATURE_HYPERVISOR), __vmware_platform()
+ * intentionally defaults to backdoor hypercall.
*/
static u32 __init vmware_platform(void)
{
@@ -470,11 +564,7 @@ static u32 __init vmware_platform(void)
if (!memcmp(hyper_vendor_id, "VMwareVMware", 12)) {
if (eax >= CPUID_VMWARE_FEATURES_LEAF)
vmware_hypercall_mode =
- vmware_select_hypercall();
-
- pr_info("hypercall mode: 0x%02x\n",
- (unsigned int) vmware_hypercall_mode);
-
+ get_hypercall_mode();
return CPUID_VMWARE_INFO_LEAF;
}
} else if (dmi_available && dmi_name_in_serial("VMware") &&
@@ -494,58 +584,6 @@ static bool __init vmware_legacy_x2apic_available(void)
(eax & GETVCPU_INFO_LEGACY_X2APIC);
}
-#ifdef CONFIG_INTEL_TDX_GUEST
-/*
- * TDCALL[TDG.VP.VMCALL] uses %rax (arg0) and %rcx (arg2). Therefore,
- * we remap those registers to %r12 and %r13, respectively.
- */
-unsigned long vmware_tdx_hypercall(unsigned long cmd,
- unsigned long in1, unsigned long in3,
- unsigned long in4, unsigned long in5,
- u32 *out1, u32 *out2, u32 *out3,
- u32 *out4, u32 *out5)
-{
- struct tdx_module_args args = {};
-
- if (!hypervisor_is_type(X86_HYPER_VMWARE)) {
- pr_warn_once("Incorrect usage\n");
- return ULONG_MAX;
- }
-
- if (cmd & ~VMWARE_CMD_MASK) {
- pr_warn_once("Out of range command %lx\n", cmd);
- return ULONG_MAX;
- }
-
- args.rbx = in1;
- args.rdx = in3;
- args.rsi = in4;
- args.rdi = in5;
- args.r10 = VMWARE_TDX_VENDOR_LEAF;
- args.r11 = VMWARE_TDX_HCALL_FUNC;
- args.r12 = VMWARE_HYPERVISOR_MAGIC;
- args.r13 = cmd;
- /* CPL */
- args.r15 = 0;
-
- __tdx_hypercall(&args);
-
- if (out1)
- *out1 = args.rbx;
- if (out2)
- *out2 = args.r13;
- if (out3)
- *out3 = args.rdx;
- if (out4)
- *out4 = args.rsi;
- if (out5)
- *out5 = args.rdi;
-
- return args.r12;
-}
-EXPORT_SYMBOL_GPL(vmware_tdx_hypercall);
-#endif
-
#ifdef CONFIG_AMD_MEM_ENCRYPT
static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb,
struct pt_regs *regs)
--
2.43.7
^ permalink raw reply related
* [PATCH 4/4] x86/vmware: Support steal time clock for encrypted guests
From: Alexey Makhalov @ 2026-03-07 0:42 UTC (permalink / raw)
To: x86, virtualization, bp, hpa, dave.hansen, mingo, tglx
Cc: ajay.kaher, brennan.lamoreaux, bo.gan, bcm-kernel-feedback-list,
linux-kernel, kas, rick.p.edgecombe, linux-coco, Alexey Makhalov
In-Reply-To: <20260307004238.1181299-1-alexey.makhalov@broadcom.com>
Shared memory containing steal time counter should be set to
decrypted when guest memory is encrypted.
Co-developed-by: Bo Gan <bo.gan@broadcom.com>
Signed-off-by: Bo Gan <bo.gan@broadcom.com>
Signed-off-by: Alexey Makhalov <alexey.makhalov@broadcom.com>
---
arch/x86/kernel/cpu/vmware.c | 41 ++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 8997295a5a5c..e33400d4f2c1 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -32,6 +32,7 @@
#include <linux/sched/cputime.h>
#include <linux/kmsg_dump.h>
#include <linux/panic_notifier.h>
+#include <linux/set_memory.h>
#include <asm/div64.h>
#include <asm/x86_init.h>
#include <asm/hypervisor.h>
@@ -39,6 +40,7 @@
#include <asm/apic.h>
#include <asm/vmware.h>
#include <asm/svm.h>
+#include <asm/coco.h>
#undef pr_fmt
#define pr_fmt(fmt) "vmware: " fmt
@@ -379,9 +381,47 @@ static struct notifier_block vmware_pv_reboot_nb = {
.notifier_call = vmware_pv_reboot_notify,
};
+/*
+ * Map per-CPU variables for all possible CPUs as decrypted.
+ * Do this early in boot, before sharing the corresponding
+ * guest physical addresses with the hypervisor.
+ */
+static void __init set_shared_memory_decrypted(void)
+{
+ int cpu;
+
+ if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ return;
+
+ for_each_possible_cpu(cpu) {
+ unsigned long size = sizeof(vmw_steal_time);
+ unsigned long addr = (unsigned long)&per_cpu(vmw_steal_time,
+ cpu);
+
+ /*
+ * There is no generic high-level API to mark memory as
+ * decrypted. Intel's set_memory_decrypted() depends on the
+ * buddy allocator and can fail early in boot if a page split
+ * is required and allocation is not possible. Use AMD's
+ * early_set_memory_decrypted() instead, which can perform
+ * the split during early boot.
+ */
+ early_set_memory_decrypted(addr, size);
+
+ /* That's it for AMD */
+ if (cc_vendor == CC_VENDOR_AMD)
+ continue;
+
+ set_memory_decrypted(addr & PAGE_MASK, 1UL <<
+ get_order((addr & ~PAGE_MASK) + size));
+
+ }
+}
+
#ifdef CONFIG_SMP
static void __init vmware_smp_prepare_boot_cpu(void)
{
+ set_shared_memory_decrypted();
vmware_guest_cpu_init();
native_smp_prepare_boot_cpu();
}
@@ -444,6 +484,7 @@ static void __init vmware_paravirt_ops_setup(void)
vmware_cpu_down_prepare) < 0)
pr_err("vmware_guest: Failed to install cpu hotplug callbacks\n");
#else
+ set_shared_memory_decrypted();
vmware_guest_cpu_init();
#endif
}
--
2.43.7
^ permalink raw reply related
* Re: [PATCH] KVM: SEV: Track SNP launch state and disallow invalid userspace interactions
From: Sean Christopherson @ 2026-03-07 1:51 UTC (permalink / raw)
To: Jethro Beekman
Cc: Paolo Bonzini, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, kvm, linux-kernel, linux-coco
In-Reply-To: <aaCfcwdA1E4V5qgE@google.com>
On Thu, Feb 26, 2026, Sean Christopherson wrote:
> On Wed, Feb 25, 2026, Jethro Beekman wrote:
> > On 2026-02-25 12:21, Sean Christopherson wrote:
> > > On Wed, Feb 25, 2026, Jethro Beekman wrote:
> > >> On 2026-02-25 12:05, Sean Christopherson wrote:
> > >>> On Mon, Jan 19, 2026, Jethro Beekman wrote:
> > >>>> Calling any of the SNP_LAUNCH_ ioctls after SNP_LAUNCH_FINISH results in a
> > >>>> kernel page fault due to RMP violation. Track SNP launch state and exit early.
> > >>>
> > >>> What exactly trips the RMP #PF? A backtrace would be especially helpful for
> > >>> posterity.
> > >>
> > >> Here's a backtrace for calling ioctl(KVM_SEV_SNP_LAUNCH_FINISH) twice. Note this is with a modified version of QEMU.
> > >
> > >> RIP: 0010:sev_es_sync_vmsa+0x54/0x4c0 [kvm_amd]
> > >> snp_launch_update_vmsa+0x19d/0x290 [kvm_amd]
> > >> snp_launch_finish+0xb6/0x380 [kvm_amd]
> > >> sev_mem_enc_ioctl+0x14e/0x720 [kvm_amd]
> > >> kvm_arch_vm_ioctl+0x837/0xcf0 [kvm]
> > >
> > > Ah, it's the VMSA that's being accessed. Can't we just do?
> > >
> > > diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> > > index 723f4452302a..1e40ae592c93 100644
> > > --- a/arch/x86/kvm/svm/sev.c
> > > +++ b/arch/x86/kvm/svm/sev.c
> > > @@ -882,6 +882,9 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm)
> > > u8 *d;
> > > int i;
> > >
> > > + if (vcpu->arch.guest_state_protected)
> > > + return -EINVAL;
> > > +
> > > /* Check some debug related fields before encrypting the VMSA */
> > > if (svm->vcpu.guest_debug || (svm->vmcb->save.dr7 & ~DR7_FIXED_1))
> > > return -EINVAL;
> >
> > I tried relying on guest_state_protected instead of creating new state but I
> > don't think it's sufficient. In particular, your proposal may fix
> > snp_launch_finish()
>
> But it does fix that case, correct? I don't want to complicate one fix just
> because there are other bugs that are similar but yet distinct.
>
> > but I don't believe this addresses the issues in snp_launch_update() and
>
> Do you mean snp_launch_update_vmsa() here? Or am I missing an interaction with
> vCPUs in snp_launch_update()?
>
> > sev_vcpu_create().
>
> There are a pile of SEV lifecycle and locking issues, i.e. this is just one of
> several flaws. Fixing the locking has been on my todo list for a few months (we
> found some "fun" bugs with an internal run of syzkaller), and I'm finally getting
> to it. Hopefully I'll post a series early next week.
>
> Somewhat off the cuff, but I think the easiest way to close the race between
> KVM_CREATE_VCPU and KVM_SEV_SNP_LAUNCH_FINISH is to reject KVM_SEV_SNP_LAUNCH_FINISH
> if a vCPU is being created. Or did I misunderstand the race you're pointing out?
>
> Though unless there's a strong reason not to, I'd prefer to get greedy and block
> all of sev_mem_enc_ioctl(), e.g.
Circling back to this (writing changelogs), I don't think there's actually a
novel bug with respect to KVM_SEV_SNP_LAUNCH_FINISH racing with KVM_CREATE_VCPU.
kvm_for_each_vcpu() operates on online_vcpus, LAUNCH_FINISH (all SEV+ sub-ioctls)
holds kvm->mutex, and fully onlining a vCPU in kvm_vm_ioctl_create_vcpu() is done
under kvm->mutex. So AFAICT, there's no difference between an in-progress vCPU
and a vCPU that is created entirely after LAUNCH_FINISH.
It's probably worth preventing as a hardening measure, but I don't think there's
an actual bug to be fixed.
^ permalink raw reply
* Re: [PATCH 2/4] x86/vmware: Log kmsg dump on panic
From: kernel test robot @ 2026-03-07 12:03 UTC (permalink / raw)
To: Alexey Makhalov, x86, virtualization, bp, hpa, dave.hansen, mingo,
tglx
Cc: oe-kbuild-all, ajay.kaher, brennan.lamoreaux, bo.gan,
bcm-kernel-feedback-list, linux-kernel, kas, rick.p.edgecombe,
linux-coco, Alexey Makhalov
In-Reply-To: <20260307004238.1181299-3-alexey.makhalov@broadcom.com>
Hi Alexey,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/master]
[also build test WARNING on linus/master v7.0-rc2 next-20260306]
[cannot apply to tip/x86/vmware tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexey-Makhalov/x86-vmware-Introduce-common-vmware_hypercall/20260307-091038
base: tip/master
patch link: https://lore.kernel.org/r/20260307004238.1181299-3-alexey.makhalov%40broadcom.com
patch subject: [PATCH 2/4] x86/vmware: Log kmsg dump on panic
config: x86_64-rhel-9.4-ltp (https://download.01.org/0day-ci/archive/20260307/202603071246.JbbF0Qpv-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260307/202603071246.JbbF0Qpv-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603071246.JbbF0Qpv-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'dumper' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'detail' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'dumper' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'detail' not described in 'kmsg_dumper_vmware_log'
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH 2/4] x86/vmware: Log kmsg dump on panic
From: kernel test robot @ 2026-03-07 13:26 UTC (permalink / raw)
To: Alexey Makhalov, x86, virtualization, bp, hpa, dave.hansen, mingo,
tglx
Cc: llvm, oe-kbuild-all, ajay.kaher, brennan.lamoreaux, bo.gan,
bcm-kernel-feedback-list, linux-kernel, kas, rick.p.edgecombe,
linux-coco, Alexey Makhalov
In-Reply-To: <20260307004238.1181299-3-alexey.makhalov@broadcom.com>
Hi Alexey,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/master]
[also build test WARNING on linus/master v7.0-rc2 next-20260305]
[cannot apply to tip/x86/vmware tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexey-Makhalov/x86-vmware-Introduce-common-vmware_hypercall/20260307-091038
base: tip/master
patch link: https://lore.kernel.org/r/20260307004238.1181299-3-alexey.makhalov%40broadcom.com
patch subject: [PATCH 2/4] x86/vmware: Log kmsg dump on panic
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20260307/202603072123.02mytKqA-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260307/202603072123.02mytKqA-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202603072123.02mytKqA-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'dumper' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'detail' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'dumper' not described in 'kmsg_dumper_vmware_log'
>> Warning: arch/x86/kernel/cpu/vmware.c:741 function parameter 'detail' not described in 'kmsg_dumper_vmware_log'
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH net-next v3 1/2] dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
From: Leon Romanovsky @ 2026-03-08 10:19 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, tjmercier,
christian.koenig, m.szyprowski, robin.murphy, jgg, sean.anderson,
ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco
In-Reply-To: <20260305123641.164164-2-jiri@resnulli.us>
On Thu, Mar 05, 2026 at 01:36:40PM +0100, Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
>
> Current CC designs don't place a vIOMMU in front of untrusted devices.
> Instead, the DMA API forces all untrusted device DMA through swiotlb
> bounce buffers (is_swiotlb_force_bounce()) which copies data into
> decrypted memory on behalf of the device.
>
> When a caller has already arranged for the memory to be decrypted
> via set_memory_decrypted(), the DMA API needs to know so it can map
> directly using the unencrypted physical address rather than bounce
> buffering. Following the pattern of DMA_ATTR_MMIO, add
> DMA_ATTR_CC_DECRYPTED for this purpose. Like the MMIO case, only the
> caller knows what kind of memory it has and must inform the DMA API
> for it to work correctly.
>
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
> ---
> v1->v2:
> - rebased on top of recent dma-mapping-fixes
> ---
> include/linux/dma-mapping.h | 6 ++++++
> include/trace/events/dma.h | 3 ++-
> kernel/dma/direct.h | 14 +++++++++++---
> 3 files changed, 19 insertions(+), 4 deletions(-)
>
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 29973baa0581..ae3d85e494ec 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -85,6 +85,12 @@
> * a cacheline must have this attribute for this to be considered safe.
> */
> #define DMA_ATTR_CPU_CACHE_CLEAN (1UL << 11)
> +/*
> + * DMA_ATTR_CC_DECRYPTED: Indicates memory that has been explicitly decrypted
> + * (shared) for confidential computing guests. The caller must have
> + * called set_memory_decrypted(). A struct page is required.
> + */
> +#define DMA_ATTR_CC_DECRYPTED (1UL << 12)
While adding the new attribute is fine, I would expect additional checks in
dma_map_phys() to ensure the attribute cannot be misused. For example,
WARN_ON(attrs & (DMA_ATTR_CC_DECRYPTED | DMA_ATTR_MMIO)), along with a check
that we are taking the direct path only.
Thanks
^ permalink raw reply
* Re: [PATCH 1/4] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>
From: Huang, Kai @ 2026-03-08 23:47 UTC (permalink / raw)
To: tglx@kernel.org, Hansen, Dave, Edgecombe, Rick P, bp@alien8.de,
kas@kernel.org, seanjc@google.com, hpa@zytor.com,
linux-kernel@vger.kernel.org, mingo@redhat.com, x86@kernel.org,
kvm@vger.kernel.org, linux-coco@lists.linux.dev,
pbonzini@redhat.com, ackerleytng@google.com, Gao, Chao
Cc: kirill.shutemov@linux.intel.com, sagis@google.com,
Verma, Vishal L, Annapurve, Vishal
In-Reply-To: <20260307010358.819645-2-rick.p.edgecombe@intel.com>
On Fri, 2026-03-06 at 17:03 -0800, Rick Edgecombe wrote:
> From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
>
> Today there are two separate locations where TDX error codes are defined:
> arch/x86/include/asm/tdx.h
> arch/x86/kvm/vmx/tdx_errno.h
>
> They have some overlap that is already defined similarly. Reduce the
> duplication and prepare to introduce some helpers for these error codes in
> the central place by unifying them. Join them at:
> asm/shared/tdx_errno.h
> ...and update the headers that contained the duplicated definitions to
> include the new unified header.
>
> Place the new header in "asm/shared". While the compressed code for the
> guest doesn't use these error code header definitions today, it does
> make the types of calls that return the values they define. Place the
> defines in "shared" location so that compressed code has the definitions
> accessible, but leave cleanups to use proper error codes for future
> changes.
>
> Opportunistically massage some comments. Also, adjust
> _BITUL()->_BITULL() to address 32 bit build errors after the move.
>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> [enhance log]
> Tested-by: Sagi Shahar <sagis@google.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> Acked-by: Vishal Annapurve <vannapurve@google.com>
> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
> Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>
Seems this patch was from your DPAMT v4.
I made couple of small comments to that:
https://lore.kernel.org/kvm/6968dcb446fb857b3f254030e487d889b464d7ce.camel@intel.com/
https://lore.kernel.org/kvm/af7c8f3ec86688709cce550a2fc17110e3fd12b7.camel@intel.com/
.. and seems you agreed to address them.
If you plan to address in the next version, free free to add:
Reviewed-by: Kai Huang <kai.huang@intel.com>
^ permalink raw reply
* Re: [PATCH 2/4] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Huang, Kai @ 2026-03-09 0:23 UTC (permalink / raw)
To: tglx@kernel.org, Hansen, Dave, Edgecombe, Rick P, bp@alien8.de,
kas@kernel.org, seanjc@google.com, hpa@zytor.com,
linux-kernel@vger.kernel.org, mingo@redhat.com, x86@kernel.org,
kvm@vger.kernel.org, linux-coco@lists.linux.dev,
pbonzini@redhat.com, ackerleytng@google.com, Gao, Chao
Cc: Verma, Vishal L
In-Reply-To: <20260307010358.819645-3-rick.p.edgecombe@intel.com>
On Fri, 2026-03-06 at 17:03 -0800, Rick Edgecombe wrote:
> KVM tries to take care of some required cache flushing earlier in the
> kexec path in order to be kind to some long standing races that can occur
> later in the operation. Until recently, VMXOFF was handled within KVM.
> Since VMX being enabled is required to make a SEAMCALL, it had the best
> per-cpu scoped operation to plug the flushing into.
>
> This early kexec cache flushing in KVM happens via a syscore shutdown
> callback. Now that VMX enablement control has moved to arch/x86, which has
> grown its own syscore shutdown callback, it no longer make sense for it to
> live in KVM. It fits better with the TDX enablement managing code.
[...]
>
> In addition, future changes will add a SEAMCALL that happens immediately
> before VMXOFF, which means the cache flush in KVM will be too late to be
> helpful. So move it to the newly added TDX arch/x86 syscore shutdown
> handler.
Nit: I am not sure how to interpret "too late to be helpful". I think we
can just get rid of this paragraph.
>
> Since tdx_cpu_flush_cache_for_kexec() is no longer needed by KVM, make it
> static and remove the export. Since it is also not part of an operation
> spread across disparate components, remove the redundant comments and
> verbose naming.
>
> Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Feel free to add:
Acked-by: Kai Huang <kai.huang@intel.com>
Btw, there's a functional change here, and perhaps we should call out in
changelog:
- Currently tdx_cpu_flush_cache_for_kexec() is done in
kvm_disable_virtualization_cpu(), which is also called by KVM's CPUHP
offline() callback. So tdx_cpu_flush_cache_for_kexec() is explicitly done
in TDX code in CPU offline.
- With this change, tdx_cpu_flush_cache_for_kexec() is not explicitly done
in TDX code in CPU offline.
But AFAICT this is fine, since IIUC the WBINVD is always done when kernel
offlines one CPU (see [*]), i.e., the current
tdx_cpu_flush_cache_for_kexec() done in KVM's CPUHP is actually superfluous.
[*] See:
native_play_dead() ->
cpuidle_play_dead();
hlt_play_dead();
cpuidle_play_dead() can invoke different enter_dead() callbacks depending on
what idle driver is being used, but AFAICT eventually it ends up calling
either acpi_idle_play_dead() or mwait_play_dead(), both of which does WBINVD
before going to idle.
If cpuidle_play_dead() doesn't idle successfully, the hlt_play_dead() will
then WBINVD and hlt.
Actually, after looking at multiple commits around here, e.g.,
ea53069231f93 ("x86, hotplug: Use mwait to offline a processor, fix the
legacy case")
dfbba2518aac4 ("Revert "ACPI: processor: idle: Only flush cache on
entering C3")
... I believe it's a kernel policy to make sure cache is flushed when it
offlines a CPU (which makes sense anyway of course), I just couldn't find
the exact commit saying this (or I am not sure whether there's such commit).
Btw2, kinda related to this, could you help review:
https://lore.kernel.org/lkml/20260302102226.7459-1-kai.huang@intel.com/
^ permalink raw reply
* Re: [PATCH 4/4] KVM: x86: Disable the TDX module during kexec and kdump
From: Chao Gao @ 2026-03-09 8:15 UTC (permalink / raw)
To: Rick Edgecombe
Cc: bp, dave.hansen, hpa, kas, kvm, linux-coco, linux-kernel, mingo,
pbonzini, seanjc, tglx, x86, kai.huang, ackerleytng,
vishal.l.verma
In-Reply-To: <20260307010358.819645-5-rick.p.edgecombe@intel.com>
A few nits below:
The scope "KVM: x86" is wrong as this doesn't touch any KVM code.
On Fri, Mar 06, 2026 at 05:03:58PM -0800, Rick Edgecombe wrote:
>From: Vishal Verma <vishal.l.verma@intel.com>
>
>Use the TDH.SYS.DISABLE SEAMCALL, which disables the TDX module,
>reclaims all memory resources assigned to TDX, and clears any
>partial-write induced poison, to allow kexec and kdump on platforms with
>the partial write errata.
>
>On TDX-capable platforms with the partial write erratum, kexec has been
>disabled because the new kernel could hit a machine check reading a
>previously poisoned memory location.
>
>Later TDX modules support TDH.SYS.DISABLE, which disables the module and
>reclaims all TDX memory resources, allowing the new kernel to re-initialize
>TDX from scratch. This operation also clears the old memory, cleaning up
>any poison.
>
>Add tdx_sys_disable() to tdx_shutdown(), which is called in the
>syscore_shutdown path for kexec. This is done just before tdx_shutdown()
>disables VMX on all CPUs.
>
>For kdump, call tdx_sys_disable() in the crash path before
>x86_virt_emergency_disable_virtualization_cpu() does VMXOFF.
>
>Since this clears any poison on TDX-managed memory, the
>X86_BUG_TDX_PW_MCE check in machine_kexec() that blocked kexec on
>partial write errata platforms can be removed.
Use imperative mood here: "Since ..., remove the X86_BUG_TDX_PW_MCE check..."
>
>Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
>Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>---
> arch/x86/kernel/crash.c | 2 ++
> arch/x86/kernel/machine_kexec_64.c | 16 ----------------
> arch/x86/virt/vmx/tdx/tdx.c | 1 +
> 3 files changed, 3 insertions(+), 16 deletions(-)
>
>diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
>index cd796818d94d..623d4474631a 100644
>--- a/arch/x86/kernel/crash.c
>+++ b/arch/x86/kernel/crash.c
>@@ -38,6 +38,7 @@
> #include <linux/kdebug.h>
> #include <asm/cpu.h>
> #include <asm/reboot.h>
>+#include <asm/tdx.h>
> #include <asm/intel_pt.h>
> #include <asm/crash.h>
> #include <asm/cmdline.h>
>@@ -112,6 +113,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
>
> crash_smp_send_stop();
>
>+ tdx_sys_disable();
> x86_virt_emergency_disable_virtualization_cpu();
>
> /*
>diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
>index 0590d399d4f1..c3f4a389992d 100644
>--- a/arch/x86/kernel/machine_kexec_64.c
>+++ b/arch/x86/kernel/machine_kexec_64.c
>@@ -347,22 +347,6 @@ int machine_kexec_prepare(struct kimage *image)
> unsigned long reloc_end = (unsigned long)__relocate_kernel_end;
> int result;
>
>- /*
>- * Some early TDX-capable platforms have an erratum. A kernel
>- * partial write (a write transaction of less than cacheline
>- * lands at memory controller) to TDX private memory poisons that
>- * memory, and a subsequent read triggers a machine check.
>- *
>- * On those platforms the old kernel must reset TDX private
>- * memory before jumping to the new kernel otherwise the new
>- * kernel may see unexpected machine check. For simplicity
>- * just fail kexec/kdump on those platforms.
>- */
>- if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE)) {
>- pr_info_once("Not allowed on platform with tdx_pw_mce bug\n");
>- return -EOPNOTSUPP;
>- }
With this series, we need to update the "Kexec" section in tdx.rst.
>-
> /* Setup the identity mapped 64bit page table */
> result = init_pgtable(image, __pa(control_page));
> if (result)
>diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
>index 68bd2618dde4..b388fbce5d76 100644
>--- a/arch/x86/virt/vmx/tdx/tdx.c
>+++ b/arch/x86/virt/vmx/tdx/tdx.c
>@@ -252,6 +252,7 @@ static void tdx_shutdown_cpu(void *ign)
>
> static void tdx_shutdown(void *ign)
> {
>+ tdx_sys_disable();
> on_each_cpu(tdx_shutdown_cpu, NULL, 1);
> }
>
>--
>2.53.0
>
^ permalink raw reply
* Re: [PATCH net-next v3 1/2] dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
From: Jiri Pirko @ 2026-03-09 8:57 UTC (permalink / raw)
To: Leon Romanovsky
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, tjmercier,
christian.koenig, m.szyprowski, robin.murphy, jgg, sean.anderson,
ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco
In-Reply-To: <20260308101948.GO12611@unreal>
Sun, Mar 08, 2026 at 11:19:48AM +0100, leon@kernel.org wrote:
>On Thu, Mar 05, 2026 at 01:36:40PM +0100, Jiri Pirko wrote:
>> From: Jiri Pirko <jiri@nvidia.com>
>>
>> Current CC designs don't place a vIOMMU in front of untrusted devices.
>> Instead, the DMA API forces all untrusted device DMA through swiotlb
>> bounce buffers (is_swiotlb_force_bounce()) which copies data into
>> decrypted memory on behalf of the device.
>>
>> When a caller has already arranged for the memory to be decrypted
>> via set_memory_decrypted(), the DMA API needs to know so it can map
>> directly using the unencrypted physical address rather than bounce
>> buffering. Following the pattern of DMA_ATTR_MMIO, add
>> DMA_ATTR_CC_DECRYPTED for this purpose. Like the MMIO case, only the
>> caller knows what kind of memory it has and must inform the DMA API
>> for it to work correctly.
>>
>> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
>> ---
>> v1->v2:
>> - rebased on top of recent dma-mapping-fixes
>> ---
>> include/linux/dma-mapping.h | 6 ++++++
>> include/trace/events/dma.h | 3 ++-
>> kernel/dma/direct.h | 14 +++++++++++---
>> 3 files changed, 19 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
>> index 29973baa0581..ae3d85e494ec 100644
>> --- a/include/linux/dma-mapping.h
>> +++ b/include/linux/dma-mapping.h
>> @@ -85,6 +85,12 @@
>> * a cacheline must have this attribute for this to be considered safe.
>> */
>> #define DMA_ATTR_CPU_CACHE_CLEAN (1UL << 11)
>> +/*
>> + * DMA_ATTR_CC_DECRYPTED: Indicates memory that has been explicitly decrypted
>> + * (shared) for confidential computing guests. The caller must have
>> + * called set_memory_decrypted(). A struct page is required.
>> + */
>> +#define DMA_ATTR_CC_DECRYPTED (1UL << 12)
>
>While adding the new attribute is fine, I would expect additional checks in
>dma_map_phys() to ensure the attribute cannot be misused. For example,
>WARN_ON(attrs & (DMA_ATTR_CC_DECRYPTED | DMA_ATTR_MMIO)), along with a check
>that we are taking the direct path only.
Okay, I will add the check.
^ permalink raw reply
* Re: [PATCH v2 5/7] KVM: guest_memfd: Add cleanup interface for guest teardown
From: Ackerley Tng @ 2026-03-09 9:01 UTC (permalink / raw)
To: Ashish Kalra, tglx, mingo, bp, dave.hansen, x86, hpa, seanjc,
peterz, thomas.lendacky, herbert, davem, ardb
Cc: pbonzini, aik, Michael.Roth, KPrateek.Nayak, Tycho.Andersen,
Nathan.Fontenot, jackyli, pgonda, rientjes, jacobhxu, xin,
pawan.kumar.gupta, babu.moger, dyoung, nikunj, john.allen, darwi,
linux-kernel, linux-crypto, kvm, linux-coco
In-Reply-To: <ce99dc548000b5a1f4486cdd3efe510b3874684b.1772486459.git.ashish.kalra@amd.com>
Ashish Kalra <Ashish.Kalra@amd.com> writes:
> From: Ashish Kalra <ashish.kalra@amd.com>
>
> Introduce kvm_arch_gmem_cleanup() to perform architecture-specific
> cleanups when the last file descriptor for the guest_memfd inode is
> closed. This typically occurs during guest shutdown and termination
> and allows for final resource release.
>
> Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
> ---
>
> [...snip...]
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 017d84a7adf3..2724dd1099f2 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -955,6 +955,14 @@ static void kvm_gmem_destroy_inode(struct inode *inode)
>
> static void kvm_gmem_free_inode(struct inode *inode)
> {
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CLEANUP
> + /*
> + * Finalize cleanup for the inode once the last guest_memfd
> + * reference is released. This usually occurs after guest
> + * termination.
> + */
> + kvm_arch_gmem_cleanup();
> +#endif
Folks have already talked about the performance implications of doing
the scan and rmpopt, I just want to call out that one VM could have more
than one associated guest_memfd too.
I think the cleanup function should be thought of as cleanup for the
inode (even if it doesn't take an inode pointer since it's not (yet)
required).
So, the gmem cleanup function should not handle deduplicating cleanup
requests, but the arch function should, if the cleanup needs
deduplicating.
Also, .free_inode() is called through RCU, so it could be called after
some delay. Could it be possible that .free_inode() ends up being called
way after the associated VM gets torn down, or after KVM the module gets
unloaded? Does rmpopt still work fine if KVM the module got unloaded?
IIUC the current kmem_cache_free(kvm_gmem_inode_cachep, GMEM_I(inode));
is fine because in kvm_gmem_exit(), there is a rcu_barrier() before
kmem_cache_destroy(kvm_gmem_inode_cachep);.
> kmem_cache_free(kvm_gmem_inode_cachep, GMEM_I(inode));
> }
>
> --
> 2.43.0
^ permalink raw reply
* [PATCH v3 0/3] Enforce host page-size alignment for shared buffers
From: Aneesh Kumar K.V (Arm) @ 2026-03-09 10:26 UTC (permalink / raw)
To: linux-kernel, iommu, linux-coco, linux-arm-kernel, kvmarm
Cc: Aneesh Kumar K.V (Arm)
Hi all,
This patch series addresses alignment requirements for buffers shared between
private-memory guests and the host.
When running private-memory guests, the guest kernel must apply additional
constraints when allocating buffers that are shared with the hypervisor. These
shared buffers are also accessed by the host kernel and therefore must be
aligned to the host’s page size.
Architectures such as Arm can tolerate realm physical address space PFNs being
mapped as shared memory, as incorrect accesses are detected and reported as GPC
faults. However, relying on this mechanism alone is unsafe and can still lead to
kernel crashes.
This is particularly likely when guest_memfd allocations are mmapped and
accessed from userspace. Once exposed to userspace, it is not possible to
guarantee that applications will only access the intended 4K shared region
rather than the full 64K page mapped into their address space. Such userspace
addresses may also be passed back into the kernel and accessed via the linear
map, potentially resulting in a GPC fault and a kernel crash.
To address this, the series introduces a new helpers,
mem_decrypt_granule_size() and mem_decrypt_align(), which allows callers to
enforce the required alignment for shared buffers.
Changes from v2:
* Rebase to latest kernel
* Consider swiotlb always decrypted and don't align when allocating from swiotlb.
Changes from v1:
* Rename the helper to mem_encrypt_align
* Improve the commit message
* Handle DMA allocations from contiguous memory
* Handle DMA allocations from the pool
* swiotlb is still considered unencrypted. Support for an encrypted swiotlb pool
is left as TODO and is independent of this series.
Aneesh Kumar K.V (Arm) (3):
dma-direct: swiotlb: handle swiotlb alloc/free outside
__dma_direct_alloc_pages
swiotlb: dma: its: Enforce host page-size alignment for shared buffers
coco: guest: arm64: Add Realm Host Interface and hostconf RHI
arch/arm64/include/asm/mem_encrypt.h | 3 ++
arch/arm64/include/asm/rhi.h | 24 ++++++++++++
arch/arm64/include/asm/rsi.h | 2 +
arch/arm64/include/asm/rsi_cmds.h | 10 +++++
arch/arm64/include/asm/rsi_smc.h | 7 ++++
arch/arm64/kernel/Makefile | 2 +-
arch/arm64/kernel/rhi.c | 53 +++++++++++++++++++++++++
arch/arm64/kernel/rsi.c | 13 +++++++
arch/arm64/mm/mem_encrypt.c | 27 +++++++++++--
drivers/irqchip/irq-gic-v3-its.c | 20 ++++++----
include/linux/mem_encrypt.h | 12 ++++++
kernel/dma/contiguous.c | 10 +++++
kernel/dma/direct.c | 58 ++++++++++++++++++++++++----
kernel/dma/pool.c | 4 +-
kernel/dma/swiotlb.c | 21 ++++++----
15 files changed, 237 insertions(+), 29 deletions(-)
create mode 100644 arch/arm64/include/asm/rhi.h
create mode 100644 arch/arm64/kernel/rhi.c
--
2.43.0
^ permalink raw reply
* [PATCH v3 1/3] dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages
From: Aneesh Kumar K.V (Arm) @ 2026-03-09 10:26 UTC (permalink / raw)
To: linux-kernel, iommu, linux-coco, linux-arm-kernel, kvmarm
Cc: Aneesh Kumar K.V (Arm), Marc Zyngier, Thomas Gleixner,
Catalin Marinas, Will Deacon, Jason Gunthorpe, Marek Szyprowski,
Robin Murphy, Steven Price, Suzuki K Poulose
In-Reply-To: <20260309102625.2315725-1-aneesh.kumar@kernel.org>
Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in
dma_direct_alloc() / dma_direct_alloc_pages().
This is needed for follow-up changes that align shared decrypted buffers to
hypervisor page size. swiotlb pool memory is decrypted as a whole and does
not need per-allocation alignment handling.
swiotlb backing pages are already mapped decrypted by
swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so
dma-direct should not call dma_set_decrypted() on allocation nor
dma_set_encrypted() on free for swiotlb-backed memory.
Update alloc/free paths to detect swiotlb-backed pages and skip
encrypt/decrypt transitions for those paths. Keep the existing highmem
rejection in dma_direct_alloc_pages() for swiotlb allocations.
Only for "restricted-dma-pool", we currently set `for_alloc = true`, while
rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is
typically used together with "shared-dma-pool", where the shared region is
accessed after remap/ioremap and the returned address is suitable for
decrypted memory access. So existing code paths remain valid.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
kernel/dma/direct.c | 44 +++++++++++++++++++++++++++++++++++++-------
1 file changed, 37 insertions(+), 7 deletions(-)
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 8f43a930716d..c2a43e4ef902 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -125,9 +125,6 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size,
WARN_ON_ONCE(!PAGE_ALIGNED(size));
- if (is_swiotlb_for_alloc(dev))
- return dma_direct_alloc_swiotlb(dev, size);
-
gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit);
page = dma_alloc_contiguous(dev, size, gfp);
if (page) {
@@ -204,6 +201,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
{
bool remap = false, set_uncached = false;
+ bool mark_mem_decrypt = true;
struct page *page;
void *ret;
@@ -250,11 +248,21 @@ void *dma_direct_alloc(struct device *dev, size_t size,
dma_direct_use_pool(dev, gfp))
return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
+ if (is_swiotlb_for_alloc(dev)) {
+ page = dma_direct_alloc_swiotlb(dev, size);
+ if (page) {
+ mark_mem_decrypt = false;
+ goto setup_page;
+ }
+ return NULL;
+ }
+
/* we always manually zero the memory once we are done */
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true);
if (!page)
return NULL;
+setup_page:
/*
* dma_alloc_contiguous can return highmem pages depending on a
* combination the cma= arguments and per-arch setup. These need to be
@@ -281,7 +289,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,
goto out_free_pages;
} else {
ret = page_address(page);
- if (dma_set_decrypted(dev, ret, size))
+ if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size))
goto out_leak_pages;
}
@@ -298,7 +306,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,
return ret;
out_encrypt_pages:
- if (dma_set_encrypted(dev, page_address(page), size))
+ if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size))
return NULL;
out_free_pages:
__dma_direct_free_pages(dev, page, size);
@@ -310,6 +318,7 @@ void *dma_direct_alloc(struct device *dev, size_t size,
void dma_direct_free(struct device *dev, size_t size,
void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs)
{
+ bool mark_mem_encrypted = true;
unsigned int page_order = get_order(size);
if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) &&
@@ -338,12 +347,15 @@ void dma_direct_free(struct device *dev, size_t size,
dma_free_from_pool(dev, cpu_addr, PAGE_ALIGN(size)))
return;
+ if (swiotlb_find_pool(dev, dma_to_phys(dev, dma_addr)))
+ mark_mem_encrypted = false;
+
if (is_vmalloc_addr(cpu_addr)) {
vunmap(cpu_addr);
} else {
if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED))
arch_dma_clear_uncached(cpu_addr, size);
- if (dma_set_encrypted(dev, cpu_addr, size))
+ if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size))
return;
}
@@ -359,6 +371,19 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp))
return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
+ if (is_swiotlb_for_alloc(dev)) {
+ page = dma_direct_alloc_swiotlb(dev, size);
+ if (!page)
+ return NULL;
+
+ if (PageHighMem(page)) {
+ swiotlb_free(dev, page, size);
+ return NULL;
+ }
+ ret = page_address(page);
+ goto setup_page;
+ }
+
page = __dma_direct_alloc_pages(dev, size, gfp, false);
if (!page)
return NULL;
@@ -366,6 +391,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
ret = page_address(page);
if (dma_set_decrypted(dev, ret, size))
goto out_leak_pages;
+setup_page:
memset(ret, 0, size);
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
return page;
@@ -378,13 +404,17 @@ void dma_direct_free_pages(struct device *dev, size_t size,
enum dma_data_direction dir)
{
void *vaddr = page_address(page);
+ bool mark_mem_encrypted = true;
/* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */
if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) &&
dma_free_from_pool(dev, vaddr, size))
return;
- if (dma_set_encrypted(dev, vaddr, size))
+ if (swiotlb_find_pool(dev, page_to_phys(page)))
+ mark_mem_encrypted = false;
+
+ if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size))
return;
__dma_direct_free_pages(dev, page, size);
}
--
2.43.0
^ permalink raw reply related
* [PATCH v3 2/3] swiotlb: dma: its: Enforce host page-size alignment for shared buffers
From: Aneesh Kumar K.V (Arm) @ 2026-03-09 10:26 UTC (permalink / raw)
To: linux-kernel, iommu, linux-coco, linux-arm-kernel, kvmarm
Cc: Aneesh Kumar K.V (Arm), Marc Zyngier, Thomas Gleixner,
Catalin Marinas, Will Deacon, Jason Gunthorpe, Marek Szyprowski,
Robin Murphy, Steven Price, Suzuki K Poulose
In-Reply-To: <20260309102625.2315725-1-aneesh.kumar@kernel.org>
When running private-memory guests, the guest kernel must apply additional
constraints when allocating buffers that are shared with the hypervisor.
These shared buffers are also accessed by the host kernel and therefore
must be aligned to the host’s page size, and have a size that is a multiple
of the host page size.
On non-secure hosts, set_guest_memory_attributes() tracks memory at the
host PAGE_SIZE granularity. This creates a mismatch when the guest applies
attributes at 4K boundaries while the host uses 64K pages. In such cases,
set_guest_memory_attributes() call returns -EINVAL, preventing the
conversion of memory regions from private to shared.
Architectures such as Arm can tolerate realm physical address space
(protected memory) PFNs being mapped as shared memory, as incorrect
accesses are detected and reported as GPC faults. However, relying on this
mechanism is unsafe and can still lead to kernel crashes.
This is particularly likely when guest_memfd allocations are mmapped and
accessed from userspace. Once exposed to userspace, we cannot guarantee
that applications will only access the intended 4K shared region rather
than the full 64K page mapped into their address space. Such userspace
addresses may also be passed back into the kernel and accessed via the
linear map, resulting in a GPC fault and a kernel crash.
With CCA, although Stage-2 mappings managed by the RMM still operate at a
4K granularity, shared pages must nonetheless be aligned to the
host-managed page size and sized as whole host pages to avoid the issues
described above.
Introduce a new helper, mem_decrypt_align(), to allow callers to enforce
the required alignment and size constraints for shared buffers.
The architecture-specific implementation of mem_decrypt_align() will be
provided in a follow-up patch.
Note on restricted-dma-pool:
rmem_swiotlb_device_init() uses reserved-memory regions described by
firmware. Those regions are not changed in-kernel to satisfy host granule
alignment. This is intentional: we do not expect restricted-dma-pool
allocations to be used with CCA. If restricted-dma-pool is intended for CCA
shared use, firmware must provide base/size aligned to the host IPA-change
granule.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
arch/arm64/mm/mem_encrypt.c | 19 +++++++++++++++----
drivers/irqchip/irq-gic-v3-its.c | 20 +++++++++++++-------
include/linux/mem_encrypt.h | 12 ++++++++++++
kernel/dma/contiguous.c | 10 ++++++++++
kernel/dma/direct.c | 16 ++++++++++++++--
kernel/dma/pool.c | 4 +++-
kernel/dma/swiotlb.c | 21 +++++++++++++--------
7 files changed, 80 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/mm/mem_encrypt.c b/arch/arm64/mm/mem_encrypt.c
index ee3c0ab04384..38c62c9e4e74 100644
--- a/arch/arm64/mm/mem_encrypt.c
+++ b/arch/arm64/mm/mem_encrypt.c
@@ -17,8 +17,7 @@
#include <linux/compiler.h>
#include <linux/err.h>
#include <linux/mm.h>
-
-#include <asm/mem_encrypt.h>
+#include <linux/mem_encrypt.h>
static const struct arm64_mem_crypt_ops *crypt_ops;
@@ -33,18 +32,30 @@ int arm64_mem_crypt_ops_register(const struct arm64_mem_crypt_ops *ops)
int set_memory_encrypted(unsigned long addr, int numpages)
{
- if (likely(!crypt_ops) || WARN_ON(!PAGE_ALIGNED(addr)))
+ if (likely(!crypt_ops))
return 0;
+ if (WARN_ON(!IS_ALIGNED(addr, mem_decrypt_granule_size())))
+ return -EINVAL;
+
+ if (WARN_ON(!IS_ALIGNED(numpages << PAGE_SHIFT, mem_decrypt_granule_size())))
+ return -EINVAL;
+
return crypt_ops->encrypt(addr, numpages);
}
EXPORT_SYMBOL_GPL(set_memory_encrypted);
int set_memory_decrypted(unsigned long addr, int numpages)
{
- if (likely(!crypt_ops) || WARN_ON(!PAGE_ALIGNED(addr)))
+ if (likely(!crypt_ops))
return 0;
+ if (WARN_ON(!IS_ALIGNED(addr, mem_decrypt_granule_size())))
+ return -EINVAL;
+
+ if (WARN_ON(!IS_ALIGNED(numpages << PAGE_SHIFT, mem_decrypt_granule_size())))
+ return -EINVAL;
+
return crypt_ops->decrypt(addr, numpages);
}
EXPORT_SYMBOL_GPL(set_memory_decrypted);
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 291d7668cc8d..239d7e3bc16f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -213,16 +213,17 @@ static gfp_t gfp_flags_quirk;
static struct page *its_alloc_pages_node(int node, gfp_t gfp,
unsigned int order)
{
+ unsigned int new_order;
struct page *page;
int ret = 0;
- page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
-
+ new_order = get_order(mem_decrypt_align((PAGE_SIZE << order)));
+ page = alloc_pages_node(node, gfp | gfp_flags_quirk, new_order);
if (!page)
return NULL;
ret = set_memory_decrypted((unsigned long)page_address(page),
- 1 << order);
+ 1 << new_order);
/*
* If set_memory_decrypted() fails then we don't know what state the
* page is in, so we can't free it. Instead we leak it.
@@ -241,13 +242,16 @@ static struct page *its_alloc_pages(gfp_t gfp, unsigned int order)
static void its_free_pages(void *addr, unsigned int order)
{
+ int new_order;
+
+ new_order = get_order(mem_decrypt_align((PAGE_SIZE << order)));
/*
* If the memory cannot be encrypted again then we must leak the pages.
* set_memory_encrypted() will already have WARNed.
*/
- if (set_memory_encrypted((unsigned long)addr, 1 << order))
+ if (set_memory_encrypted((unsigned long)addr, 1 << new_order))
return;
- free_pages((unsigned long)addr, order);
+ free_pages((unsigned long)addr, new_order);
}
static struct gen_pool *itt_pool;
@@ -268,11 +272,13 @@ static void *itt_alloc_pool(int node, int size)
if (addr)
break;
- page = its_alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
+ page = its_alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO,
+ get_order(mem_decrypt_granule_size()));
if (!page)
break;
- gen_pool_add(itt_pool, (unsigned long)page_address(page), PAGE_SIZE, node);
+ gen_pool_add(itt_pool, (unsigned long)page_address(page),
+ mem_decrypt_granule_size(), node);
} while (!addr);
return (void *)addr;
diff --git a/include/linux/mem_encrypt.h b/include/linux/mem_encrypt.h
index 07584c5e36fb..6cf39845058e 100644
--- a/include/linux/mem_encrypt.h
+++ b/include/linux/mem_encrypt.h
@@ -54,6 +54,18 @@
#define dma_addr_canonical(x) (x)
#endif
+#ifndef mem_decrypt_granule_size
+static inline size_t mem_decrypt_granule_size(void)
+{
+ return PAGE_SIZE;
+}
+#endif
+
+static inline size_t mem_decrypt_align(size_t size)
+{
+ return ALIGN(size, mem_decrypt_granule_size());
+}
+
#endif /* __ASSEMBLY__ */
#endif /* __MEM_ENCRYPT_H__ */
diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
index c56004d314dc..2b7ff68be0c4 100644
--- a/kernel/dma/contiguous.c
+++ b/kernel/dma/contiguous.c
@@ -46,6 +46,7 @@
#include <linux/dma-map-ops.h>
#include <linux/cma.h>
#include <linux/nospec.h>
+#include <linux/dma-direct.h>
#ifdef CONFIG_CMA_SIZE_MBYTES
#define CMA_SIZE_MBYTES CONFIG_CMA_SIZE_MBYTES
@@ -374,6 +375,15 @@ struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp)
#ifdef CONFIG_DMA_NUMA_CMA
int nid = dev_to_node(dev);
#endif
+ /*
+ * for untrusted device, we require the dma buffers to be aligned to
+ * the mem_decrypt_align(PAGE_SIZE) so that we can set the memory
+ * attributes correctly.
+ */
+ if (force_dma_unencrypted(dev)) {
+ if (get_order(mem_decrypt_granule_size()) > CONFIG_CMA_ALIGNMENT)
+ return NULL;
+ }
/* CMA can be used only in the context which permits sleeping */
if (!gfpflags_allow_blocking(gfp))
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index c2a43e4ef902..34eccd047e9b 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -257,6 +257,9 @@ void *dma_direct_alloc(struct device *dev, size_t size,
return NULL;
}
+ if (force_dma_unencrypted(dev))
+ size = mem_decrypt_align(size);
+
/* we always manually zero the memory once we are done */
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true);
if (!page)
@@ -350,6 +353,9 @@ void dma_direct_free(struct device *dev, size_t size,
if (swiotlb_find_pool(dev, dma_to_phys(dev, dma_addr)))
mark_mem_encrypted = false;
+ if (mark_mem_encrypted && force_dma_unencrypted(dev))
+ size = mem_decrypt_align(size);
+
if (is_vmalloc_addr(cpu_addr)) {
vunmap(cpu_addr);
} else {
@@ -384,6 +390,9 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
goto setup_page;
}
+ if (force_dma_unencrypted(dev))
+ size = mem_decrypt_align(size);
+
page = __dma_direct_alloc_pages(dev, size, gfp, false);
if (!page)
return NULL;
@@ -414,8 +423,11 @@ void dma_direct_free_pages(struct device *dev, size_t size,
if (swiotlb_find_pool(dev, page_to_phys(page)))
mark_mem_encrypted = false;
- if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size))
- return;
+ if (mark_mem_encrypted && force_dma_unencrypted(dev)) {
+ size = mem_decrypt_align(size);
+ if (dma_set_encrypted(dev, vaddr, size))
+ return;
+ }
__dma_direct_free_pages(dev, page, size);
}
diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c
index 2b2fbb709242..b5f10ba3e855 100644
--- a/kernel/dma/pool.c
+++ b/kernel/dma/pool.c
@@ -83,7 +83,9 @@ static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
struct page *page = NULL;
void *addr;
int ret = -ENOMEM;
+ unsigned int min_encrypt_order = get_order(mem_decrypt_granule_size());
+ pool_size = mem_decrypt_align(pool_size);
/* Cannot allocate larger than MAX_PAGE_ORDER */
order = min(get_order(pool_size), MAX_PAGE_ORDER);
@@ -94,7 +96,7 @@ static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
order, false);
if (!page)
page = alloc_pages(gfp | __GFP_NOWARN, order);
- } while (!page && order-- > 0);
+ } while (!page && order-- > min_encrypt_order);
if (!page)
goto out;
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index d8e6f1d889d5..a9e6e4775ec6 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -260,7 +260,7 @@ void __init swiotlb_update_mem_attributes(void)
if (!mem->nslabs || mem->late_alloc)
return;
- bytes = PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT);
+ bytes = mem_decrypt_align(mem->nslabs << IO_TLB_SHIFT);
set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT);
}
@@ -317,8 +317,8 @@ static void __init *swiotlb_memblock_alloc(unsigned long nslabs,
unsigned int flags,
int (*remap)(void *tlb, unsigned long nslabs))
{
- size_t bytes = PAGE_ALIGN(nslabs << IO_TLB_SHIFT);
void *tlb;
+ size_t bytes = mem_decrypt_align(nslabs << IO_TLB_SHIFT);
/*
* By default allocate the bounce buffer memory from low memory, but
@@ -326,9 +326,9 @@ static void __init *swiotlb_memblock_alloc(unsigned long nslabs,
* memory encryption.
*/
if (flags & SWIOTLB_ANY)
- tlb = memblock_alloc(bytes, PAGE_SIZE);
+ tlb = memblock_alloc(bytes, mem_decrypt_granule_size());
else
- tlb = memblock_alloc_low(bytes, PAGE_SIZE);
+ tlb = memblock_alloc_low(bytes, mem_decrypt_granule_size());
if (!tlb) {
pr_warn("%s: Failed to allocate %zu bytes tlb structure\n",
@@ -337,7 +337,7 @@ static void __init *swiotlb_memblock_alloc(unsigned long nslabs,
}
if (remap && remap(tlb, nslabs) < 0) {
- memblock_free(tlb, PAGE_ALIGN(bytes));
+ memblock_free(tlb, bytes);
pr_warn("%s: Failed to remap %zu bytes\n", __func__, bytes);
return NULL;
}
@@ -459,7 +459,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
swiotlb_adjust_nareas(num_possible_cpus());
retry:
- order = get_order(nslabs << IO_TLB_SHIFT);
+ order = get_order(mem_decrypt_align(nslabs << IO_TLB_SHIFT));
nslabs = SLABS_PER_PAGE << order;
while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
@@ -468,6 +468,8 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
if (vstart)
break;
order--;
+ if (order < get_order(mem_decrypt_granule_size()))
+ break;
nslabs = SLABS_PER_PAGE << order;
retried = true;
}
@@ -535,7 +537,7 @@ void __init swiotlb_exit(void)
pr_info("tearing down default memory pool\n");
tbl_vaddr = (unsigned long)phys_to_virt(mem->start);
- tbl_size = PAGE_ALIGN(mem->end - mem->start);
+ tbl_size = mem_decrypt_align(mem->end - mem->start);
slots_size = PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs));
set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT);
@@ -571,11 +573,13 @@ void __init swiotlb_exit(void)
*/
static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limit)
{
- unsigned int order = get_order(bytes);
+ unsigned int order;
struct page *page;
phys_addr_t paddr;
void *vaddr;
+ bytes = mem_decrypt_align(bytes);
+ order = get_order(bytes);
page = alloc_pages(gfp, order);
if (!page)
return NULL;
@@ -658,6 +662,7 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes)
dma_free_from_pool(NULL, vaddr, bytes))
return;
+ bytes = mem_decrypt_align(bytes);
/* Intentional leak if pages cannot be encrypted again. */
if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes)))
__free_pages(virt_to_page(vaddr), get_order(bytes));
--
2.43.0
^ permalink raw reply related
* [PATCH v3 3/3] coco: guest: arm64: Add Realm Host Interface and hostconf RHI
From: Aneesh Kumar K.V (Arm) @ 2026-03-09 10:26 UTC (permalink / raw)
To: linux-kernel, iommu, linux-coco, linux-arm-kernel, kvmarm
Cc: Aneesh Kumar K.V (Arm), Marc Zyngier, Thomas Gleixner,
Catalin Marinas, Will Deacon, Jason Gunthorpe, Marek Szyprowski,
Robin Murphy, Steven Price, Suzuki K Poulose
In-Reply-To: <20260309102625.2315725-1-aneesh.kumar@kernel.org>
- describe the Realm Host Interface SMC IDs and result codes in a new
asm/rhi.h header
- expose struct rsi_host_call plus an rsi_host_call() helper so we can
invoke SMC_RSI_HOST_CALL from C code
- add RHI hostconf SMC IDs and helper to query version, features, and IPA
change alignment
- derive the realm hypervisor page size during init and abort realm setup
on invalid alignment
This provides the host page-size discovery needed by previous patch that
align shared buffer allocation/decryption to host requirements.
Cc: Marc Zyngier <maz@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
arch/arm64/include/asm/mem_encrypt.h | 3 ++
arch/arm64/include/asm/rhi.h | 24 +++++++++++++
arch/arm64/include/asm/rsi.h | 2 ++
arch/arm64/include/asm/rsi_cmds.h | 10 ++++++
arch/arm64/include/asm/rsi_smc.h | 7 ++++
arch/arm64/kernel/Makefile | 2 +-
arch/arm64/kernel/rhi.c | 53 ++++++++++++++++++++++++++++
arch/arm64/kernel/rsi.c | 13 +++++++
arch/arm64/mm/mem_encrypt.c | 8 +++++
9 files changed, 121 insertions(+), 1 deletion(-)
create mode 100644 arch/arm64/include/asm/rhi.h
create mode 100644 arch/arm64/kernel/rhi.c
diff --git a/arch/arm64/include/asm/mem_encrypt.h b/arch/arm64/include/asm/mem_encrypt.h
index 314b2b52025f..5541911eb028 100644
--- a/arch/arm64/include/asm/mem_encrypt.h
+++ b/arch/arm64/include/asm/mem_encrypt.h
@@ -16,6 +16,9 @@ int arm64_mem_crypt_ops_register(const struct arm64_mem_crypt_ops *ops);
int set_memory_encrypted(unsigned long addr, int numpages);
int set_memory_decrypted(unsigned long addr, int numpages);
+#define mem_decrypt_granule_size mem_decrypt_granule_size
+size_t mem_decrypt_granule_size(void);
+
int realm_register_memory_enc_ops(void);
static inline bool force_dma_unencrypted(struct device *dev)
diff --git a/arch/arm64/include/asm/rhi.h b/arch/arm64/include/asm/rhi.h
new file mode 100644
index 000000000000..0895dd92ea1d
--- /dev/null
+++ b/arch/arm64/include/asm/rhi.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ */
+
+#ifndef __ASM_RHI_H_
+#define __ASM_RHI_H_
+
+#include <linux/types.h>
+
+#define SMC_RHI_CALL(func) \
+ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
+ ARM_SMCCC_SMC_64, \
+ ARM_SMCCC_OWNER_STANDARD_HYP,\
+ (func))
+
+unsigned long rhi_get_ipa_change_alignment(void);
+#define RHI_HOSTCONF_VER_1_0 0x10000
+#define RHI_HOSTCONF_VERSION SMC_RHI_CALL(0x004E)
+
+#define __RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT BIT(0)
+#define RHI_HOSTCONF_FEATURES SMC_RHI_CALL(0x004F)
+#define RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT SMC_RHI_CALL(0x0050)
+#endif
diff --git a/arch/arm64/include/asm/rsi.h b/arch/arm64/include/asm/rsi.h
index 88b50d660e85..ae54fb3b1429 100644
--- a/arch/arm64/include/asm/rsi.h
+++ b/arch/arm64/include/asm/rsi.h
@@ -67,4 +67,6 @@ static inline int rsi_set_memory_range_shared(phys_addr_t start,
return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY,
RSI_CHANGE_DESTROYED);
}
+
+unsigned long realm_get_hyp_pagesize(void);
#endif /* __ASM_RSI_H_ */
diff --git a/arch/arm64/include/asm/rsi_cmds.h b/arch/arm64/include/asm/rsi_cmds.h
index 2c8763876dfb..a341ce0eeda1 100644
--- a/arch/arm64/include/asm/rsi_cmds.h
+++ b/arch/arm64/include/asm/rsi_cmds.h
@@ -159,4 +159,14 @@ static inline unsigned long rsi_attestation_token_continue(phys_addr_t granule,
return res.a0;
}
+static inline unsigned long rsi_host_call(struct rsi_host_call *rhi_call)
+{
+ phys_addr_t addr = virt_to_phys(rhi_call);
+ struct arm_smccc_res res;
+
+ arm_smccc_1_1_invoke(SMC_RSI_HOST_CALL, addr, &res);
+
+ return res.a0;
+}
+
#endif /* __ASM_RSI_CMDS_H */
diff --git a/arch/arm64/include/asm/rsi_smc.h b/arch/arm64/include/asm/rsi_smc.h
index e19253f96c94..9ee8b5c7612e 100644
--- a/arch/arm64/include/asm/rsi_smc.h
+++ b/arch/arm64/include/asm/rsi_smc.h
@@ -182,6 +182,13 @@ struct realm_config {
*/
#define SMC_RSI_IPA_STATE_GET SMC_RSI_FID(0x198)
+struct rsi_host_call {
+ union {
+ u16 imm;
+ u64 padding0;
+ };
+ u64 gprs[31];
+} __aligned(0x100);
/*
* Make a Host call.
*
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 76f32e424065..fcb67f50ea89 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -34,7 +34,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
cpufeature.o alternative.o cacheinfo.o \
smp.o smp_spin_table.o topology.o smccc-call.o \
syscall.o proton-pack.o idle.o patching.o pi/ \
- rsi.o jump_label.o
+ rsi.o jump_label.o rhi.o
obj-$(CONFIG_COMPAT) += sys32.o signal32.o \
sys_compat.o
diff --git a/arch/arm64/kernel/rhi.c b/arch/arm64/kernel/rhi.c
new file mode 100644
index 000000000000..d2141b5283e1
--- /dev/null
+++ b/arch/arm64/kernel/rhi.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026 ARM Ltd.
+ */
+
+#include <asm/rsi.h>
+#include <asm/rhi.h>
+
+/* we need an aligned rhicall for rsi_host_call. slab is not yet ready */
+static struct rsi_host_call hyp_pagesize_rhicall;
+unsigned long rhi_get_ipa_change_alignment(void)
+{
+ long ret;
+ unsigned long ipa_change_align;
+
+ hyp_pagesize_rhicall.imm = 0;
+ hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_VERSION;
+ ret = rsi_host_call(&hyp_pagesize_rhicall);
+ if (ret != RSI_SUCCESS)
+ goto err_out;
+
+ if (hyp_pagesize_rhicall.gprs[0] != RHI_HOSTCONF_VER_1_0)
+ goto err_out;
+
+ hyp_pagesize_rhicall.imm = 0;
+ hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_FEATURES;
+ ret = rsi_host_call(&hyp_pagesize_rhicall);
+ if (ret != RSI_SUCCESS)
+ goto err_out;
+
+ if (!(hyp_pagesize_rhicall.gprs[0] & __RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT))
+ goto err_out;
+
+ hyp_pagesize_rhicall.imm = 0;
+ hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT;
+ ret = rsi_host_call(&hyp_pagesize_rhicall);
+ if (ret != RSI_SUCCESS)
+ goto err_out;
+
+ ipa_change_align = hyp_pagesize_rhicall.gprs[0];
+ /* This error needs special handling in the caller */
+ if (ipa_change_align & (SZ_4K - 1))
+ return 0;
+
+ return ipa_change_align;
+
+err_out:
+ /*
+ * For failure condition assume host is built with 4K page size
+ * and hence ipa change alignment can be guest PAGE_SIZE.
+ */
+ return PAGE_SIZE;
+}
diff --git a/arch/arm64/kernel/rsi.c b/arch/arm64/kernel/rsi.c
index c64a06f58c0b..6e35cb947745 100644
--- a/arch/arm64/kernel/rsi.c
+++ b/arch/arm64/kernel/rsi.c
@@ -13,8 +13,10 @@
#include <asm/io.h>
#include <asm/mem_encrypt.h>
#include <asm/rsi.h>
+#include <asm/rhi.h>
static struct realm_config config;
+static unsigned long ipa_change_alignment = PAGE_SIZE;
unsigned long prot_ns_shared;
EXPORT_SYMBOL(prot_ns_shared);
@@ -138,6 +140,11 @@ static int realm_ioremap_hook(phys_addr_t phys, size_t size, pgprot_t *prot)
return 0;
}
+unsigned long realm_get_hyp_pagesize(void)
+{
+ return ipa_change_alignment;
+}
+
void __init arm64_rsi_init(void)
{
if (arm_smccc_1_1_get_conduit() != SMCCC_CONDUIT_SMC)
@@ -146,6 +153,12 @@ void __init arm64_rsi_init(void)
return;
if (WARN_ON(rsi_get_realm_config(&config)))
return;
+
+ ipa_change_alignment = rhi_get_ipa_change_alignment();
+ /* If we don't get a correct alignment response, don't enable realm */
+ if (!ipa_change_alignment)
+ return;
+
prot_ns_shared = BIT(config.ipa_bits - 1);
if (arm64_ioremap_prot_hook_register(realm_ioremap_hook))
diff --git a/arch/arm64/mm/mem_encrypt.c b/arch/arm64/mm/mem_encrypt.c
index 38c62c9e4e74..f5d64bc29c20 100644
--- a/arch/arm64/mm/mem_encrypt.c
+++ b/arch/arm64/mm/mem_encrypt.c
@@ -59,3 +59,11 @@ int set_memory_decrypted(unsigned long addr, int numpages)
return crypt_ops->decrypt(addr, numpages);
}
EXPORT_SYMBOL_GPL(set_memory_decrypted);
+
+size_t mem_decrypt_granule_size(void)
+{
+ if (is_realm_world())
+ return max(PAGE_SIZE, realm_get_hyp_pagesize());
+ return PAGE_SIZE;
+}
+EXPORT_SYMBOL_GPL(mem_decrypt_granule_size);
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v3 3/3] coco: guest: arm64: Add Realm Host Interface and hostconf RHI
From: Suzuki K Poulose @ 2026-03-09 10:50 UTC (permalink / raw)
To: Aneesh Kumar K.V (Arm), linux-kernel, iommu, linux-coco,
linux-arm-kernel, kvmarm
Cc: Marc Zyngier, Thomas Gleixner, Catalin Marinas, Will Deacon,
Jason Gunthorpe, Marek Szyprowski, Robin Murphy, Steven Price
In-Reply-To: <20260309102625.2315725-4-aneesh.kumar@kernel.org>
Hi Aneesh
On 09/03/2026 10:26, Aneesh Kumar K.V (Arm) wrote:
> - describe the Realm Host Interface SMC IDs and result codes in a new
> asm/rhi.h header
> - expose struct rsi_host_call plus an rsi_host_call() helper so we can
> invoke SMC_RSI_HOST_CALL from C code
> - add RHI hostconf SMC IDs and helper to query version, features, and IPA
> change alignment
> - derive the realm hypervisor page size during init and abort realm setup
> on invalid alignment
>
> This provides the host page-size discovery needed by previous patch that
> align shared buffer allocation/decryption to host requirements.
>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Thomas Gleixner <tglx@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: Marek Szyprowski <m.szyprowski@samsung.com>
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Steven Price <steven.price@arm.com>
> Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
> ---
> arch/arm64/include/asm/mem_encrypt.h | 3 ++
> arch/arm64/include/asm/rhi.h | 24 +++++++++++++
> arch/arm64/include/asm/rsi.h | 2 ++
> arch/arm64/include/asm/rsi_cmds.h | 10 ++++++
> arch/arm64/include/asm/rsi_smc.h | 7 ++++
> arch/arm64/kernel/Makefile | 2 +-
> arch/arm64/kernel/rhi.c | 53 ++++++++++++++++++++++++++++
> arch/arm64/kernel/rsi.c | 13 +++++++
> arch/arm64/mm/mem_encrypt.c | 8 +++++
> 9 files changed, 121 insertions(+), 1 deletion(-)
> create mode 100644 arch/arm64/include/asm/rhi.h
> create mode 100644 arch/arm64/kernel/rhi.c
>
> diff --git a/arch/arm64/include/asm/mem_encrypt.h b/arch/arm64/include/asm/mem_encrypt.h
> index 314b2b52025f..5541911eb028 100644
> --- a/arch/arm64/include/asm/mem_encrypt.h
> +++ b/arch/arm64/include/asm/mem_encrypt.h
> @@ -16,6 +16,9 @@ int arm64_mem_crypt_ops_register(const struct arm64_mem_crypt_ops *ops);
> int set_memory_encrypted(unsigned long addr, int numpages);
> int set_memory_decrypted(unsigned long addr, int numpages);
>
> +#define mem_decrypt_granule_size mem_decrypt_granule_size
> +size_t mem_decrypt_granule_size(void);
> +
> int realm_register_memory_enc_ops(void);
>
> static inline bool force_dma_unencrypted(struct device *dev)
> diff --git a/arch/arm64/include/asm/rhi.h b/arch/arm64/include/asm/rhi.h
> new file mode 100644
> index 000000000000..0895dd92ea1d
> --- /dev/null
> +++ b/arch/arm64/include/asm/rhi.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2026 ARM Ltd.
> + */
> +
> +#ifndef __ASM_RHI_H_
> +#define __ASM_RHI_H_
> +
> +#include <linux/types.h>
> +
> +#define SMC_RHI_CALL(func) \
> + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
> + ARM_SMCCC_SMC_64, \
> + ARM_SMCCC_OWNER_STANDARD_HYP,\
> + (func))
> +
> +unsigned long rhi_get_ipa_change_alignment(void);
> +#define RHI_HOSTCONF_VER_1_0 0x10000
> +#define RHI_HOSTCONF_VERSION SMC_RHI_CALL(0x004E)
> +
> +#define __RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT BIT(0)
> +#define RHI_HOSTCONF_FEATURES SMC_RHI_CALL(0x004F)
> +#define RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT SMC_RHI_CALL(0x0050)
> +#endif
> diff --git a/arch/arm64/include/asm/rsi.h b/arch/arm64/include/asm/rsi.h
> index 88b50d660e85..ae54fb3b1429 100644
> --- a/arch/arm64/include/asm/rsi.h
> +++ b/arch/arm64/include/asm/rsi.h
> @@ -67,4 +67,6 @@ static inline int rsi_set_memory_range_shared(phys_addr_t start,
> return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY,
> RSI_CHANGE_DESTROYED);
> }
> +
> +unsigned long realm_get_hyp_pagesize(void);
> #endif /* __ASM_RSI_H_ */
> diff --git a/arch/arm64/include/asm/rsi_cmds.h b/arch/arm64/include/asm/rsi_cmds.h
> index 2c8763876dfb..a341ce0eeda1 100644
> --- a/arch/arm64/include/asm/rsi_cmds.h
> +++ b/arch/arm64/include/asm/rsi_cmds.h
> @@ -159,4 +159,14 @@ static inline unsigned long rsi_attestation_token_continue(phys_addr_t granule,
> return res.a0;
> }
>
> +static inline unsigned long rsi_host_call(struct rsi_host_call *rhi_call)
> +{
> + phys_addr_t addr = virt_to_phys(rhi_call);
> + struct arm_smccc_res res;
> +
> + arm_smccc_1_1_invoke(SMC_RSI_HOST_CALL, addr, &res);
> +
> + return res.a0;
> +}
> +
> #endif /* __ASM_RSI_CMDS_H */
> diff --git a/arch/arm64/include/asm/rsi_smc.h b/arch/arm64/include/asm/rsi_smc.h
> index e19253f96c94..9ee8b5c7612e 100644
> --- a/arch/arm64/include/asm/rsi_smc.h
> +++ b/arch/arm64/include/asm/rsi_smc.h
> @@ -182,6 +182,13 @@ struct realm_config {
> */
> #define SMC_RSI_IPA_STATE_GET SMC_RSI_FID(0x198)
>
> +struct rsi_host_call {
> + union {
> + u16 imm;
> + u64 padding0;
> + };
> + u64 gprs[31];
> +} __aligned(0x100);
> /*
> * Make a Host call.
> *
> diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
> index 76f32e424065..fcb67f50ea89 100644
> --- a/arch/arm64/kernel/Makefile
> +++ b/arch/arm64/kernel/Makefile
> @@ -34,7 +34,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
> cpufeature.o alternative.o cacheinfo.o \
> smp.o smp_spin_table.o topology.o smccc-call.o \
> syscall.o proton-pack.o idle.o patching.o pi/ \
> - rsi.o jump_label.o
> + rsi.o jump_label.o rhi.o
>
> obj-$(CONFIG_COMPAT) += sys32.o signal32.o \
> sys_compat.o
> diff --git a/arch/arm64/kernel/rhi.c b/arch/arm64/kernel/rhi.c
> new file mode 100644
> index 000000000000..d2141b5283e1
> --- /dev/null
> +++ b/arch/arm64/kernel/rhi.c
> @@ -0,0 +1,53 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (C) 2026 ARM Ltd.
> + */
> +
> +#include <asm/rsi.h>
> +#include <asm/rhi.h>
> +
> +/* we need an aligned rhicall for rsi_host_call. slab is not yet ready */
> +static struct rsi_host_call hyp_pagesize_rhicall;
> +unsigned long rhi_get_ipa_change_alignment(void)
> +{
> + long ret;
> + unsigned long ipa_change_align;
> +
> + hyp_pagesize_rhicall.imm = 0;
> + hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_VERSION;
> + ret = rsi_host_call(&hyp_pagesize_rhicall);
> + if (ret != RSI_SUCCESS)
> + goto err_out;
> +
> + if (hyp_pagesize_rhicall.gprs[0] != RHI_HOSTCONF_VER_1_0)
> + goto err_out;
> +
> + hyp_pagesize_rhicall.imm = 0;
> + hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_FEATURES;
> + ret = rsi_host_call(&hyp_pagesize_rhicall);
> + if (ret != RSI_SUCCESS)
> + goto err_out;
> +
> + if (!(hyp_pagesize_rhicall.gprs[0] & __RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT))
> + goto err_out;
> +
> + hyp_pagesize_rhicall.imm = 0;
> + hyp_pagesize_rhicall.gprs[0] = RHI_HOSTCONF_GET_IPA_CHANGE_ALIGNMENT;
> + ret = rsi_host_call(&hyp_pagesize_rhicall);
> + if (ret != RSI_SUCCESS)
> + goto err_out;
> +
> + ipa_change_align = hyp_pagesize_rhicall.gprs[0];
> + /* This error needs special handling in the caller */
> + if (ipa_change_align & (SZ_4K - 1))
> + return 0;
> +
> + return ipa_change_align;
> +
> +err_out:
> + /*
> + * For failure condition assume host is built with 4K page size
> + * and hence ipa change alignment can be guest PAGE_SIZE.
> + */
> + return PAGE_SIZE;
> +}
> diff --git a/arch/arm64/kernel/rsi.c b/arch/arm64/kernel/rsi.c
> index c64a06f58c0b..6e35cb947745 100644
> --- a/arch/arm64/kernel/rsi.c
> +++ b/arch/arm64/kernel/rsi.c
> @@ -13,8 +13,10 @@
> #include <asm/io.h>
> #include <asm/mem_encrypt.h>
> #include <asm/rsi.h>
> +#include <asm/rhi.h>
>
> static struct realm_config config;
> +static unsigned long ipa_change_alignment = PAGE_SIZE;
>
> unsigned long prot_ns_shared;
> EXPORT_SYMBOL(prot_ns_shared);
> @@ -138,6 +140,11 @@ static int realm_ioremap_hook(phys_addr_t phys, size_t size, pgprot_t *prot)
> return 0;
> }
>
> +unsigned long realm_get_hyp_pagesize(void)
> +{
> + return ipa_change_alignment;
> +}
> +
> void __init arm64_rsi_init(void)
> {
> if (arm_smccc_1_1_get_conduit() != SMCCC_CONDUIT_SMC)
> @@ -146,6 +153,12 @@ void __init arm64_rsi_init(void)
> return;
> if (WARN_ON(rsi_get_realm_config(&config)))
> return;
> +
> + ipa_change_alignment = rhi_get_ipa_change_alignment();
minor nit: We reset the alignment to 0 if this fails. see below.
> + /* If we don't get a correct alignment response, don't enable realm */
Do we need to enforce this ? If the Host doesn't implement this, we
could proceed and the guest might encounter failures in "sharing" in
the worst case. Otherwise, it could proceed. Eitherway, RMM guarantees
that the "state" of the PFNs are stable as reported by the RSI
calls and the Guest knows exactly what has happened.
Suzuki
> + if (!ipa_change_alignment)
> + return;
> +
> prot_ns_shared = BIT(config.ipa_bits - 1);
>
> if (arm64_ioremap_prot_hook_register(realm_ioremap_hook))
> diff --git a/arch/arm64/mm/mem_encrypt.c b/arch/arm64/mm/mem_encrypt.c
> index 38c62c9e4e74..f5d64bc29c20 100644
> --- a/arch/arm64/mm/mem_encrypt.c
> +++ b/arch/arm64/mm/mem_encrypt.c
> @@ -59,3 +59,11 @@ int set_memory_decrypted(unsigned long addr, int numpages)
> return crypt_ops->decrypt(addr, numpages);
> }
> EXPORT_SYMBOL_GPL(set_memory_decrypted);
> +
> +size_t mem_decrypt_granule_size(void)
> +{
> + if (is_realm_world())
> + return max(PAGE_SIZE, realm_get_hyp_pagesize());
> + return PAGE_SIZE;
> +}
> +EXPORT_SYMBOL_GPL(mem_decrypt_granule_size);
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox