* Re: [PATCH v5 2/2] dma-buf: heaps: system: add system_cc_shared heap for explicitly shared memory
From: Maxime Ripard @ 2026-04-02 12:23 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, jstultz, tjmercier,
christian.koenig, m.szyprowski, robin.murphy, jgg, leon,
sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260325192352.437608-3-jiri@resnulli.us>
[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]
Hi Jiri,
On Wed, Mar 25, 2026 at 08:23:52PM +0100, Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
>
> Add a new "system_cc_shared" dma-buf heap to allow userspace to
> allocate shared (decrypted) memory for confidential computing (CoCo)
> VMs.
>
> On CoCo VMs, guest memory is private by default. The hardware uses an
> encryption bit in page table entries (C-bit on AMD SEV, "shared" bit on
> Intel TDX) to control whether a given memory access is private or
> shared. The kernel's direct map is set up as private,
> so pages returned by alloc_pages() are private in the direct map
> by default. To make this memory usable for devices that do not support
> DMA to private memory (no TDISP support), it has to be explicitly
> shared. A couple of things are needed to properly handle
> shared memory for the dma-buf use case:
>
> - set_memory_decrypted() on the direct map after allocation:
> Besides clearing the encryption bit in the direct map PTEs, this
> also notifies the hypervisor about the page state change. On free,
> the inverse set_memory_encrypted() must be called before returning
> pages to the allocator. If re-encryption fails, pages
> are intentionally leaked to prevent shared memory from being
> reused as private.
>
> - pgprot_decrypted() for userspace and kernel virtual mappings:
> Any new mapping of the shared pages, be it to userspace via
> mmap or to kernel vmalloc space via vmap, creates PTEs independent
> of the direct map. These must also have the encryption bit cleared,
> otherwise accesses through them would see encrypted (garbage) data.
>
> - DMA_ATTR_CC_SHARED for DMA mapping:
> Since the pages are already shared, the DMA API needs to be
> informed via DMA_ATTR_CC_SHARED so it can map them correctly
> as unencrypted for device access.
>
> On non-CoCo VMs, the system_cc_shared heap is not registered
> to prevent misuse by userspace that does not understand
> the security implications of explicitly shared memory.
>
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
I'm a bit late to the party, sorry.
This new heap must be documented in
Documentation/userspace-api/dma-buf-heaps.rst, but (and especially since
it seems like it was merged already) it can be done as a follow-up
patch.
Maxime
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 273 bytes --]
^ permalink raw reply
* [Invitation] bi-weekly guest_memfd upstream call on 2026-04-02
From: David Hildenbrand @ 2026-04-02 12:11 UTC (permalink / raw)
To: KVM, linux-mm@kvack.org, linux-coco@lists.linux.dev
Hi,
Late reminder :)
Our next guest_memfd upstream call is scheduled for today, Thursday,
2026-04-02 at 8:00 - 9:00am (GMT-07:00) Pacific Time - Vancouver.
We'll be using the following Google meet:
http://meet.google.com/wxp-wtju-jzw
The meeting notes can be found at [1], where we also link recordings and
collect current guest_memfd upstream proposals. If you want an google
calendar invitation that also covers all future meetings, just write me
or Ackerley a mail.
There is not a lot on the agenda: interaction with guest_memfd and
VM_DONTDUMP. So let's see what else comes up!
To put something to discuss onto the agenda, reply to this mail or add
them to the "Topics/questions for next meeting(s)" section in the
meeting notes as a comment.
[1]
https://docs.google.com/document/d/1M6766BzdY1Lhk7LiR5IqVR8B8mG3cr-cxTxOrAosPOk/edit?usp=sharing
--
Cheers,
David
^ permalink raw reply
* Re: [PATCH v5 0/2] dma-buf: heaps: system: add an option to allocate explicitly shared/decrypted memory
From: Jason Gunthorpe @ 2026-04-02 12:02 UTC (permalink / raw)
To: Brian Starkey
Cc: Jiri Pirko, dri-devel, linaro-mm-sig, iommu, linux-media,
sumit.semwal, benjamin.gaignard, jstultz, tjmercier,
christian.koenig, m.szyprowski, robin.murphy, leon, sean.anderson,
ptesarik, catalin.marinas, aneesh.kumar, suzuki.poulose,
steven.price, thomas.lendacky, john.allen, ashish.kalra,
suravee.suthikulpanit, linux-coco, nd
In-Reply-To: <4s75rtsmdfrze4rscbqzyrjyrko7n6sehe7agrngqag2cna5bz@eflmf4kwjocj>
On Thu, Apr 02, 2026 at 10:52:34AM +0100, Brian Starkey wrote:
> I'm not clear why the heap depends on !CONFIG_HIGHMEM, but I also
> don't know anything about SEV/TDX.
It is because the CC apis, set_memory_decrypted()/etc are slightly
mis-designed. They take in a vaddr to represent the address instead of
a phys_addr_t or a page *
This means the user has to use page_address() and then the whole thing
is incompatible with highmem.
Which is fine, highmem and CC are never turned on together.
Jason
^ permalink raw reply
* Re: [PATCH v2 27/31] coco/tdx-host: Implement SPDM session setup
From: Nikolay Borisov @ 2026-04-02 11:29 UTC (permalink / raw)
To: Xu Yilun, linux-coco, linux-pci, dan.j.williams, x86
Cc: chao.gao, dave.jiang, baolu.lu, yilun.xu, zhenzhong.duan, kvm,
rick.p.edgecombe, dave.hansen, kas, xiaoyao.li, vishal.l.verma,
linux-kernel
In-Reply-To: <20260327160132.2946114-28-yilun.xu@linux.intel.com>
On 27.03.26 г. 18:01 ч., Xu Yilun wrote:
> From: Zhenzhong Duan <zhenzhong.duan@intel.com>
>
> Implementation for a most straightforward SPDM session setup, using all
> default session options. Retrieve device info data from TDX Module which
> contains the SPDM negotiation results.
>
> TDH.SPDM.CONNECT/DISCONNECT are TDX Module Extension introduced
> SEAMCALLs which can run for longer periods and interruptible. But there
> is resource constraints that limit how many SEAMCALLs of this kind can
> run simultaneously. The current situation is One SEAMCALL at a time.
> Otherwise TDX_OPERAND_BUSY is returned. To avoid "broken indefinite"
> retry, a tdx_ext_lock is used to guard these SEAMCALLs.
>
> Co-developed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Signed-off-by: Xu Yilun <yilun.xu@linux.intel.com>
> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com>
> ---
> arch/x86/include/asm/shared/tdx_errno.h | 2 +
> drivers/virt/coco/tdx-host/tdx-host.c | 301 +++++++++++++++++++++++-
> 2 files changed, 299 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/include/asm/shared/tdx_errno.h b/arch/x86/include/asm/shared/tdx_errno.h
> index 8bf6765cf082..7db04fe30378 100644
> --- a/arch/x86/include/asm/shared/tdx_errno.h
> +++ b/arch/x86/include/asm/shared/tdx_errno.h
> @@ -29,6 +29,8 @@
> #define TDX_EPT_WALK_FAILED 0xC0000B0000000000ULL
> #define TDX_EPT_ENTRY_STATE_INCORRECT 0xC0000B0D00000000ULL
> #define TDX_METADATA_FIELD_NOT_READABLE 0xC0000C0200000000ULL
> +#define TDX_SPDM_SESSION_KEY_REQUIRE_REFRESH 0xC0000F4500000000ULL
> +#define TDX_SPDM_REQUEST 0xC0000F5700000000ULL
>
> /*
> * SW-defined error codes.
> diff --git a/drivers/virt/coco/tdx-host/tdx-host.c b/drivers/virt/coco/tdx-host/tdx-host.c
> index 06f3d194e0a8..4d127b7c2591 100644
> --- a/drivers/virt/coco/tdx-host/tdx-host.c
> +++ b/drivers/virt/coco/tdx-host/tdx-host.c
> @@ -14,6 +14,7 @@
> #include <linux/pci-doe.h>
> #include <linux/pci-tsm.h>
> #include <linux/tsm.h>
> +#include <linux/vmalloc.h>
>
> #include <asm/cpu_device_id.h>
> #include <asm/tdx.h>
> @@ -32,8 +33,43 @@ MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids);
> */
> static const struct tdx_sys_info *tdx_sysinfo;
>
> +#define TDISP_FUNC_ID GENMASK(15, 0)
> +#define TDISP_FUNC_ID_SEGMENT GENMASK(23, 16)
> +#define TDISP_FUNC_ID_SEG_VALID BIT(24)
> +
> +static inline u32 tdisp_func_id(struct pci_dev *pdev)
> +{
> + u32 func_id;
> +
> + func_id = FIELD_PREP(TDISP_FUNC_ID_SEGMENT, pci_domain_nr(pdev->bus));
> + if (func_id)
> + func_id |= TDISP_FUNC_ID_SEG_VALID;
This check implies pci_domain_nr returning 0 is considered invalid.
Other callers in the kernel seem to not care, they just use the domain
nr, so is this check spurious or intentional ?
> + func_id |= FIELD_PREP(TDISP_FUNC_ID,
> + PCI_DEVID(pdev->bus->number, pdev->devfn));
> +
> + return func_id;
> +}
> +
> +struct spdm_config_info_t {
> + u32 vmm_spdm_cap;
> +#define SPDM_CAP_HBEAT BIT(13)
> +#define SPDM_CAP_KEY_UPD BIT(14)
nit: move those defines above the struct definition, they just break the
reading flow as it is.
> + u8 spdm_session_policy;
> + u8 certificate_slot_mask;
> + u8 raw_bitstream_requested;
> +} __packed;
> +
> struct tdx_tsm_link {
> struct pci_tsm_pf0 pci;
> + u32 func_id;
> + struct page *in_msg;
> + struct page *out_msg;
> +
> + u64 spdm_id;
> + struct page *spdm_conf;
> + struct tdx_page_array *spdm_mt;
> + unsigned int dev_info_size;
> + void *dev_info_data;
> };
>
> static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm)
<snip>
> +
> +static void *tdx_dup_array_data(struct tdx_page_array *array,
> + unsigned int data_size)
> +{
> + unsigned int npages = (data_size + PAGE_SIZE - 1) / PAGE_SIZE;
nit: There's DIV_ROUND_UP
> + void *data, *dup_data;
> +
> + if (npages > array->nr_pages)
> + return NULL;
> +
> + data = vm_map_ram(array->pages, npages, -1);
> + if (!data)
> + return NULL;
> +
> + dup_data = kmemdup(data, data_size, GFP_KERNEL);
> + vm_unmap_ram(data, npages);
> +
> + return dup_data;
> +}
> +
<snip>
> +
> +DEFINE_FREE(tdx_spdm_session_teardown, struct tdx_tsm_link *,
> + if (!IS_ERR_OR_NULL(_T)) tdx_spdm_session_teardown(_T))
> +
> static int tdx_tsm_link_connect(struct pci_dev *pdev)
> {
> - return -ENXIO;
> + struct tdx_tsm_link *tlink = to_tdx_tsm_link(pdev->tsm);
> +
> + struct tdx_tsm_link *tlink_spdm __free(tdx_spdm_session_teardown) =
> + tdx_spdm_session_setup(tlink);
Is the free() really needed here, either the session is correctly setup
and tlink_spdm is returned. But if session_setup() files then what about
calling spdm_session_disconnect() on an unestablished session?
> + if (IS_ERR(tlink_spdm)) {
> + pci_err(pdev, "fail to setup spdm session\n");
> + return PTR_ERR(tlink_spdm);
> + }
> +
> + retain_and_null_ptr(tlink_spdm);
> +
> + return 0;
> }
>
<snip>
^ permalink raw reply
* Re: [PATCH v5 0/2] dma-buf: heaps: system: add an option to allocate explicitly shared/decrypted memory
From: Brian Starkey @ 2026-04-02 9:52 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, jstultz, tjmercier, christian.koenig,
m.szyprowski, robin.murphy, jgg, leon, sean.anderson, ptesarik,
catalin.marinas, aneesh.kumar, suzuki.poulose, steven.price,
thomas.lendacky, john.allen, ashish.kalra, suravee.suthikulpanit,
linux-coco, nd
In-Reply-To: <20260325192352.437608-1-jiri@resnulli.us>
Hi,
I know I'm late to the party here...
Like John, I'm also not very close to this stuff any more, but I agree
with the other discussions: makes sense for this to be a separate
heap, and cc_shared makes sense too.
I'm not clear why the heap depends on !CONFIG_HIGHMEM, but I also
don't know anything about SEV/TDX.
-Brian
On Wed, Mar 25, 2026 at 08:23:50PM +0000, Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
>
> Confidential computing (CoCo) VMs/guests, such as AMD SEV and Intel TDX,
> run with private/encrypted memory which creates a challenge
> for devices that do not support DMA to it (no TDISP support).
>
> For kernel-only DMA operations, swiotlb bounce buffering provides a
> transparent solution by copying data through shared memory.
> However, the only way to get this memory into userspace is via the DMA
> API's dma_alloc_pages()/dma_mmap_pages() type interfaces which limits
> the use of the memory to a single DMA device, and is incompatible with
> pin_user_pages().
>
> These limitations are particularly problematic for the RDMA subsystem
> which makes heavy use of pin_user_pages() and expects flexible memory
> usage between many different DMA devices.
>
> This patch series enables userspace to explicitly request shared
> (decrypted) memory allocations from new dma-buf system_cc_shared heap.
> Userspace can mmap this memory and pass the dma-buf fd to other
> existing importers such as RDMA or DRM devices to access the
> memory. The DMA API is improved to allow the dma heap exporter to DMA
> map the shared memory to each importing device.
>
> Based on dma-mapping-for-next e7442a68cd1ee797b585f045d348781e9c0dde0d
>
> Jiri Pirko (2):
> dma-mapping: introduce DMA_ATTR_CC_SHARED for shared memory
> dma-buf: heaps: system: add system_cc_shared heap for explicitly
> shared memory
>
> drivers/dma-buf/heaps/system_heap.c | 103 ++++++++++++++++++++++++++--
> include/linux/dma-mapping.h | 10 +++
> include/trace/events/dma.h | 3 +-
> kernel/dma/direct.h | 14 +++-
> kernel/dma/mapping.c | 13 +++-
> 5 files changed, 132 insertions(+), 11 deletions(-)
>
> --
> 2.51.1
>
^ permalink raw reply
* RE: [PATCH 2/2] x86/tdx: Accept hotplugged memory before online
From: Reshetova, Elena @ 2026-04-02 8:18 UTC (permalink / raw)
To: Edgecombe, Rick P, pbonzini@redhat.com, prsampat@amd.com
Cc: dave.hansen@linux.intel.com, marcandre.lureau@redhat.com,
kas@kernel.org, bp@alien8.de, linux-kernel@vger.kernel.org,
mingo@redhat.com, x86@kernel.org, Qiang, Chenyi, tglx@kernel.org,
hpa@zytor.com, kvm@vger.kernel.org, linux-coco@lists.linux.dev
In-Reply-To: <49e8b24d836c1883e83ad72d1ab279f9e3eb7455.camel@intel.com>
> On Mon, 2026-03-30 at 11:10 -0400, Pratik R. Sampat wrote:
> > SNP likely has an analogous issue too.
> > Failing to switch states on remove will cause that RMP entry to
> > remain validated. A malicious hypervisor could then remap this GPA to
> > another HPA which would put this in the Guest-Invalid state. On re-
> > hotplug if we ignore errors suggested by Patch 1 (in our case that'd
> > be PVALIDATE_FAIL_NOUPDATE error likely), we could have two RMP
> > entries for the same GPA and both being validated. This is dangerous
> > because hypervisor could swap these at will.
>
> Oh, I was just wondering if we could just zero the page on accept
> failure for the case of already accepted. Handle the issue internally
> and actually go back to something like patch 1. Will it work for SNP?
I don't know about SNP, but if you are proposing to zero the page on
double acceptance, this is not great from security pov. It creates a
predictable behaviour primitive for the host to zero any data inside
the confidential guest and it can be misused (think of zeroing out a
page containing a cryptographic key).
^ permalink raw reply
* Re: SVSM Development Call April 1st, 2026
From: Jörg Rödel @ 2026-04-02 7:54 UTC (permalink / raw)
To: coconut-svsm, linux-coco
In-Reply-To: <lbjylrvih3qk4irnbboktpadie4ldtqxzwyqarqubyvw6zkouw@j7kjwm2jts33>
Meeting minutes are now ready:
https://github.com/coconut-svsm/governance/pull/102
-Joerg
^ permalink raw reply
* [PATCH v3 5/5] x86/virt/tdx: Remove kexec docs
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Kai Huang
In-Reply-To: <20260402-fuller_tdx_kexec_support-v3-0-34438d7094bf@intel.com>
From: Rick Edgecombe <rick.p.edgecombe@intel.com>
Recent changes have removed the hard limitations for using kexec and
TDX together. So remove the section in the TDX docs.
Users on partial write erratums will need an updated TDX module to
handle the rare edge cases. The docs do not currently provide any
guidance on recommended TDX module versions, so don't keep a whole
section around to document this interaction.
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Kai Huang <kai.huang@intel.com>
---
Documentation/arch/x86/tdx.rst | 7 -------
1 file changed, 7 deletions(-)
diff --git a/Documentation/arch/x86/tdx.rst b/Documentation/arch/x86/tdx.rst
index ff6b110291bc..1a3b5bac1021 100644
--- a/Documentation/arch/x86/tdx.rst
+++ b/Documentation/arch/x86/tdx.rst
@@ -138,13 +138,6 @@ If the platform has such erratum, the kernel prints additional message in
machine check handler to tell user the machine check may be caused by
kernel bug on TDX private memory.
-Kexec
-~~~~~~~
-
-Currently kexec doesn't work on the TDX platforms with the aforementioned
-erratum. It fails when loading the kexec kernel image. Otherwise it
-works normally.
-
Interaction vs S3 and deeper states
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--
2.53.0
^ permalink raw reply related
* [PATCH v3 4/5] x86/tdx: Disable the TDX module during kexec and kdump
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Kai Huang
In-Reply-To: <20260402-fuller_tdx_kexec_support-v3-0-34438d7094bf@intel.com>
Use the TDH.SYS.DISABLE SEAMCALL, which disables the TDX module,
reclaims all memory resources assigned to TDX, and clears any
partial-write induced poison, to allow kexec and kdump on platforms with
the partial write errata.
On TDX-capable platforms with the partial write erratum, kexec has been
disabled because the new kernel could hit a machine check reading a
previously poisoned memory location.
Later TDX modules support TDH.SYS.DISABLE, which disables the module and
reclaims all TDX memory resources, allowing the new kernel to re-initialize
TDX from scratch. This operation also clears the old memory, cleaning up
any poison.
Add tdx_sys_disable() to tdx_shutdown(), which is called in the
syscore_shutdown path for kexec. This is done just before tdx_shutdown()
disables VMX on all CPUs.
For kdump, call tdx_sys_disable() in the crash path before
x86_virt_emergency_disable_virtualization_cpu() does VMXOFF.
Since this clears any poison on TDX-managed memory, remove the
X86_BUG_TDX_PW_MCE check in machine_kexec() that blocked kexec on
partial write errata platforms.
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Kai Huang <kai.huang@intel.com>
---
arch/x86/kernel/crash.c | 2 ++
arch/x86/kernel/machine_kexec_64.c | 16 ----------------
arch/x86/virt/vmx/tdx/tdx.c | 1 +
3 files changed, 3 insertions(+), 16 deletions(-)
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index cd796818d94d..623d4474631a 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -38,6 +38,7 @@
#include <linux/kdebug.h>
#include <asm/cpu.h>
#include <asm/reboot.h>
+#include <asm/tdx.h>
#include <asm/intel_pt.h>
#include <asm/crash.h>
#include <asm/cmdline.h>
@@ -112,6 +113,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
crash_smp_send_stop();
+ tdx_sys_disable();
x86_virt_emergency_disable_virtualization_cpu();
/*
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 0590d399d4f1..c3f4a389992d 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -347,22 +347,6 @@ int machine_kexec_prepare(struct kimage *image)
unsigned long reloc_end = (unsigned long)__relocate_kernel_end;
int result;
- /*
- * Some early TDX-capable platforms have an erratum. A kernel
- * partial write (a write transaction of less than cacheline
- * lands at memory controller) to TDX private memory poisons that
- * memory, and a subsequent read triggers a machine check.
- *
- * On those platforms the old kernel must reset TDX private
- * memory before jumping to the new kernel otherwise the new
- * kernel may see unexpected machine check. For simplicity
- * just fail kexec/kdump on those platforms.
- */
- if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE)) {
- pr_info_once("Not allowed on platform with tdx_pw_mce bug\n");
- return -EOPNOTSUPP;
- }
-
/* Setup the identity mapped 64bit page table */
result = init_pgtable(image, __pa(control_page));
if (result)
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index 1ae558bcca3a..c0c6281b08a5 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -259,6 +259,7 @@ static void tdx_shutdown_cpu(void *ign)
static void tdx_shutdown(void *ign)
{
+ tdx_sys_disable();
on_each_cpu(tdx_shutdown_cpu, NULL, 1);
}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 3/5] x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Chao Gao, Kai Huang
In-Reply-To: <20260402-fuller_tdx_kexec_support-v3-0-34438d7094bf@intel.com>
Some early TDX-capable platforms have an erratum where a partial write
to TDX private memory can cause a machine check on a subsequent read.
On these platforms, kexec and kdump have been disabled in these cases,
because the old kernel cannot safely hand off TDX state to the new
kernel. Later TDX modules support the TDH.SYS.DISABLE SEAMCALL, which
provides a way to cleanly disable TDX and allow kexec to proceed.
The new SEAMCALL has an enumeration bit, but that is ignored. It is
expected that users will be using the latest TDX module, and the failure
mode for running the missing SEAMCALL on an older module is not fatal.
This can be a long running operation, and the time needed largely
depends on the amount of memory that has been allocated to TDs. If all
TDs have been destroyed prior to the sys_disable call, then it is fast,
with only needing to override the TDX module memory.
After the SEAMCALL completes, the TDX module is disabled and all memory
resources allocated to TDX are freed and reset. The next kernel can then
re-initialize the TDX module from scratch via the normal TDX bring-up
sequence.
The SEAMCALL can return two different error codes that expect a retry.
- TDX_INTERRUPTED_RESUMABLE can be returned in the case of a host
interrupt. However, it will not return until it makes some forward
progress, so we can expect to complete even in the case of interrupt
storms.
- TDX_SYS_BUSY will be returned on contention with other TDH.SYS.*
SEAMCALLs, however a side effect of TDH.SYS.DISABLE is that it will
block other SEAMCALLs once it gets going. So this contention will be
short lived.
So loop infinitely on either of these error codes, until success or other
error.
An error is printed if the SEAMCALL fails with anything other than the
error codes that cause retries, or 'synthesized' error codes produced
for #GP or #UD. e.g., an old module that has been properly initialized,
that doesn't implement SYS_DISABLE, returns TDX_OPERAND_INVALID. This
prints:
virt/tdx: TDH.SYS.DISABLE failed: 0xc000010000000000
But a system that doesn't have any TDX support at all doesn't print
anything.
Co-developed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Kai Huang <kai.huang@intel.com>
---
arch/x86/include/asm/shared/tdx_errno.h | 1 +
arch/x86/include/asm/tdx.h | 3 +++
arch/x86/virt/vmx/tdx/tdx.h | 1 +
arch/x86/virt/vmx/tdx/tdx.c | 31 +++++++++++++++++++++++++++++++
4 files changed, 36 insertions(+)
diff --git a/arch/x86/include/asm/shared/tdx_errno.h b/arch/x86/include/asm/shared/tdx_errno.h
index 3c1e8ce716e3..ee411b360e20 100644
--- a/arch/x86/include/asm/shared/tdx_errno.h
+++ b/arch/x86/include/asm/shared/tdx_errno.h
@@ -13,6 +13,7 @@
#define TDX_NON_RECOVERABLE_TD_NON_ACCESSIBLE 0x6000000500000000ULL
#define TDX_NON_RECOVERABLE_TD_WRONG_APIC_MODE 0x6000000700000000ULL
#define TDX_INTERRUPTED_RESUMABLE 0x8000000300000000ULL
+#define TDX_SYS_BUSY 0x8000020200000000ULL
#define TDX_OPERAND_INVALID 0xC000010000000000ULL
#define TDX_OPERAND_BUSY 0x8000020000000000ULL
#define TDX_PREVIOUS_TLB_EPOCH_BUSY 0x8000020100000000ULL
diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
index bf83a974a0d5..15eac89b0afb 100644
--- a/arch/x86/include/asm/tdx.h
+++ b/arch/x86/include/asm/tdx.h
@@ -193,6 +193,8 @@ static inline int pg_level_to_tdx_sept_level(enum pg_level level)
return level - 1;
}
+void tdx_sys_disable(void);
+
u64 tdh_vp_enter(struct tdx_vp *vp, struct tdx_module_args *args);
u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page);
u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2);
@@ -224,6 +226,7 @@ static inline void tdx_init(void) { }
static inline u32 tdx_get_nr_guest_keyids(void) { return 0; }
static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
static inline const struct tdx_sys_info *tdx_get_sysinfo(void) { return NULL; }
+static inline void tdx_sys_disable(void) { }
#endif /* CONFIG_INTEL_TDX_HOST */
#endif /* !__ASSEMBLER__ */
diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h
index dde219c823b4..e2cf2dd48755 100644
--- a/arch/x86/virt/vmx/tdx/tdx.h
+++ b/arch/x86/virt/vmx/tdx/tdx.h
@@ -46,6 +46,7 @@
#define TDH_PHYMEM_PAGE_WBINVD 41
#define TDH_VP_WR 43
#define TDH_SYS_CONFIG 45
+#define TDH_SYS_DISABLE 69
/*
* SEAMCALL leaf:
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index 1b2d854ba664..1ae558bcca3a 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -37,6 +37,7 @@
#include <asm/msr.h>
#include <asm/cpufeature.h>
#include <asm/tdx.h>
+#include <asm/shared/tdx_errno.h>
#include <asm/cpu_device_id.h>
#include <asm/processor.h>
#include <asm/mce.h>
@@ -1947,3 +1948,33 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
}
EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
+
+void tdx_sys_disable(void)
+{
+ struct tdx_module_args args = {};
+ u64 ret;
+
+ /*
+ * Don't loop forever.
+ *
+ * - TDX_INTERRUPTED_RESUMABLE guarantees forward progress between
+ * calls.
+ *
+ * - TDX_SYS_BUSY could be returned due to contention with other
+ * TDH.SYS.* SEAMCALLs, but will lock out *new* TDH.SYS.* SEAMCALLs,
+ * so that SYS.DISABLE can eventually make progress.
+ *
+ * This is a 'destructive' SEAMCALL, in that no other SEAMCALL can be
+ * run after this until a full reinitialization is done.
+ */
+ do {
+ ret = seamcall(TDH_SYS_DISABLE, &args);
+ } while (ret == TDX_INTERRUPTED_RESUMABLE || ret == TDX_SYS_BUSY);
+
+ /*
+ * Print SEAMCALL failures, but not SW-defined error codes
+ * (SEAMCALL faulted with #GP/#UD, TDX not supported).
+ */
+ if (ret && (ret & TDX_SW_ERROR) != TDX_SW_ERROR)
+ pr_err("TDH.SYS.DISABLE failed: 0x%016llx\n", ret);
+}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 2/5] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Chao Gao, Kai Huang
In-Reply-To: <20260402-fuller_tdx_kexec_support-v3-0-34438d7094bf@intel.com>
From: Rick Edgecombe <rick.p.edgecombe@intel.com>
KVM tries to take care of some required cache flushing earlier in the
kexec path in order to be kind to some long standing races that can occur
later in the operation. Until recently, VMXOFF was handled within KVM.
Since VMX being enabled is required to make a SEAMCALL, it had the best
per-cpu scoped operation to plug the flushing into. So it is kicked off
from there.
This early kexec cache flushing in KVM happens via a syscore shutdown
callback. Now that VMX enablement control has moved to arch/x86, which has
grown its own syscore shutdown callback, it no longer make sense for it to
live in KVM. It fits better with the TDX enablement managing code.
In addition, future changes will add a SEAMCALL that happens immediately
before VMXOFF, which means the cache flush in KVM will be too late to
flush the cache before the last SEAMCALL. So move it to the newly added TDX
arch/x86 syscore shutdown handler.
Since tdx_cpu_flush_cache_for_kexec() is no longer needed by KVM, make it
static and remove the export. Since it is also not part of an operation
spread across disparate components, remove the redundant comments and
verbose naming.
In the existing KVM based code, CPU offline also funnels through
tdx_cpu_flush_cache_for_kexec(). Add an explicit WBINVD in
tdx_offline_cpu() as well, even though it may be redundant with WBINVD
done elsewhere during CPU offline (e.g. hlt_play_dead()). This avoids
relying on fragile code ordering for cache coherency safety.
[Vishal: add explicit WBINVD in tdx_offline_cpu()]
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
Acked-by: Kai Huang <kai.huang@intel.com>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Acked-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/include/asm/tdx.h | 6 ------
arch/x86/kvm/vmx/tdx.c | 10 ----------
arch/x86/virt/vmx/tdx/tdx.c | 46 ++++++++++++++++++++++++++-------------------
3 files changed, 27 insertions(+), 35 deletions(-)
diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
index a149740b24e8..bf83a974a0d5 100644
--- a/arch/x86/include/asm/tdx.h
+++ b/arch/x86/include/asm/tdx.h
@@ -226,11 +226,5 @@ static inline const char *tdx_dump_mce_info(struct mce *m) { return NULL; }
static inline const struct tdx_sys_info *tdx_get_sysinfo(void) { return NULL; }
#endif /* CONFIG_INTEL_TDX_HOST */
-#ifdef CONFIG_KEXEC_CORE
-void tdx_cpu_flush_cache_for_kexec(void);
-#else
-static inline void tdx_cpu_flush_cache_for_kexec(void) { }
-#endif
-
#endif /* !__ASSEMBLER__ */
#endif /* _ASM_X86_TDX_H */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b7264b533feb..50a5cfdbd33e 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -440,16 +440,6 @@ void tdx_disable_virtualization_cpu(void)
tdx_flush_vp(&arg);
}
local_irq_restore(flags);
-
- /*
- * Flush cache now if kexec is possible: this is necessary to avoid
- * having dirty private memory cachelines when the new kernel boots,
- * but WBINVD is a relatively expensive operation and doing it during
- * kexec can exacerbate races in native_stop_other_cpus(). Do it
- * now, since this is a safe moment and there is going to be no more
- * TDX activity on this CPU from this point on.
- */
- tdx_cpu_flush_cache_for_kexec();
}
#define TDX_SEAMCALL_RETRIES 10000
diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
index cb9b3210ab71..1b2d854ba664 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
+++ b/arch/x86/virt/vmx/tdx/tdx.c
@@ -184,6 +184,17 @@ static int tdx_online_cpu(unsigned int cpu)
return ret;
}
+static void tdx_cpu_flush_cache(void)
+{
+ lockdep_assert_preemption_disabled();
+
+ if (!this_cpu_read(cache_state_incoherent))
+ return;
+
+ wbinvd();
+ this_cpu_write(cache_state_incoherent, false);
+}
+
static int tdx_offline_cpu(unsigned int cpu)
{
int i;
@@ -220,12 +231,28 @@ static int tdx_offline_cpu(unsigned int cpu)
return -EBUSY;
done:
+ /*
+ * Flush cache on the CPU going offline to ensure no dirty
+ * cachelines of TDX private memory remain. This may be
+ * redundant with WBINVD done elsewhere during CPU offline
+ * (e.g. hlt_play_dead()), but do it explicitly for safety.
+ */
+ tdx_cpu_flush_cache();
x86_virt_put_ref(X86_FEATURE_VMX);
return 0;
}
static void tdx_shutdown_cpu(void *ign)
{
+ /*
+ * Flush cache in preparation for kexec - this is necessary to avoid
+ * having dirty private memory cachelines when the new kernel boots,
+ * but WBINVD is a relatively expensive operation and doing it during
+ * kexec can exacerbate races in native_stop_other_cpus(). Do it
+ * now, since this is a safe moment and there is going to be no more
+ * TDX activity on this CPU from this point on.
+ */
+ tdx_cpu_flush_cache();
x86_virt_put_ref(X86_FEATURE_VMX);
}
@@ -1920,22 +1947,3 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
}
EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
-
-#ifdef CONFIG_KEXEC_CORE
-void tdx_cpu_flush_cache_for_kexec(void)
-{
- lockdep_assert_preemption_disabled();
-
- if (!this_cpu_read(cache_state_incoherent))
- return;
-
- /*
- * Private memory cachelines need to be clean at the time of
- * kexec. Write them back now, as the caller promises that
- * there should be no more SEAMCALLs on this CPU.
- */
- wbinvd();
- this_cpu_write(cache_state_incoherent, false);
-}
-EXPORT_SYMBOL_FOR_KVM(tdx_cpu_flush_cache_for_kexec);
-#endif
--
2.53.0
^ permalink raw reply related
* [PATCH v3 1/5] x86/tdx: Move TDX architectural error codes into <asm/shared/tdx_errno.h>
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Chao Gao,
Kiryl Shutsemau, Kiryl Shutsemau
In-Reply-To: <20260402-fuller_tdx_kexec_support-v3-0-34438d7094bf@intel.com>
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Today there are two separate locations where TDX error codes are defined:
arch/x86/include/asm/tdx.h
arch/x86/kvm/vmx/tdx_errno.h
They have some overlap that is already defined similarly. Reduce the
duplication by unifying the architectural error codes at:
asm/shared/tdx_errno.h
...and update the headers that contained the duplicated definitions to
include the new unified header.
"asm/shared" is used for sharing TDX code between the early compressed
code and the normal kernel code. While the compressed code for the guest
doesn't use these error code header definitions today, it does make the
types of calls that return the values they define. So place the defines in
"shared" location so that it can, but leave such cleanups for future
changes.
[Rick: enhance log]
[Vishal: reduce to a simple move of architectural defines only]
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Reviewed-by: Chao Gao <chao.gao@intel.com>
---
arch/x86/include/asm/shared/tdx.h | 1 +
arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h | 7 +++----
arch/x86/kvm/vmx/tdx.h | 1 -
3 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/arch/x86/include/asm/shared/tdx.h b/arch/x86/include/asm/shared/tdx.h
index 8bc074c8d7c6..6a1646fc2b2f 100644
--- a/arch/x86/include/asm/shared/tdx.h
+++ b/arch/x86/include/asm/shared/tdx.h
@@ -4,6 +4,7 @@
#include <linux/bits.h>
#include <linux/types.h>
+#include <asm/shared/tdx_errno.h>
#define TDX_HYPERCALL_STANDARD 0
diff --git a/arch/x86/kvm/vmx/tdx_errno.h b/arch/x86/include/asm/shared/tdx_errno.h
similarity index 92%
rename from arch/x86/kvm/vmx/tdx_errno.h
rename to arch/x86/include/asm/shared/tdx_errno.h
index 6ff4672c4181..3c1e8ce716e3 100644
--- a/arch/x86/kvm/vmx/tdx_errno.h
+++ b/arch/x86/include/asm/shared/tdx_errno.h
@@ -1,8 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* architectural status code for SEAMCALL */
-
-#ifndef __KVM_X86_TDX_ERRNO_H
-#define __KVM_X86_TDX_ERRNO_H
+#ifndef _ASM_X86_SHARED_TDX_ERRNO_H
+#define _ASM_X86_SHARED_TDX_ERRNO_H
#define TDX_SEAMCALL_STATUS_MASK 0xFFFFFFFF00000000ULL
@@ -37,4 +36,4 @@
#define TDX_OPERAND_ID_SEPT 0x92
#define TDX_OPERAND_ID_TD_EPOCH 0xa9
-#endif /* __KVM_X86_TDX_ERRNO_H */
+#endif /* _ASM_X86_SHARED_TDX_ERRNO_H */
diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h
index b5cd2ffb303e..ac8323a68b16 100644
--- a/arch/x86/kvm/vmx/tdx.h
+++ b/arch/x86/kvm/vmx/tdx.h
@@ -3,7 +3,6 @@
#define __KVM_X86_VMX_TDX_H
#include "tdx_arch.h"
-#include "tdx_errno.h"
#ifdef CONFIG_KVM_INTEL_TDX
#include "common.h"
--
2.53.0
^ permalink raw reply related
* [PATCH v3 0/5] Fuller TDX kexec support
From: Vishal Verma @ 2026-04-02 6:32 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Kiryl Shutsemau, Rick Edgecombe,
Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Vishal Verma, Chao Gao, Kai Huang,
Kiryl Shutsemau
Hi,
This series adds a couple of cool things -
1. Allow kexec and kdump on systems with the partial write errata
2. Allow using TDX in the second (kexec'ed) kernel
It has been waiting for VMXON refactor to land because the implementation
is much cleaner on top of that.
KVM folks, just a few deletions on your side and the long discussed moving
of tdx_errno.h. Tip folks and reviewers, the changes here are pretty small.
Optimistically, I'm hoping we can iterate this quickly and see it off the
list in the next few weeks.
Background
==========
Some early TDX-capable platforms have an erratum where a partial write
to TDX private memory can cause a machine check on a subsequent read.
Currently, kexec and kdump are disabled on these platforms because the
new (or kdump) kernel may trip over these, causing a machine check.
Future TDX modules will support TDH.SYS.DISABLE SEAMCALL, which disables
the TDX module and reclaims all memory resources allocated to TDX, and
cleans up any poison. After this SEAMCALL succeeds, the new kernel
can also re-initialize the TDX module from scratch via the normal bring-up
sequence.
It is probably worth mentioning that this is a different kind of cleanup
than the WBINVD stuff that was the cause of all the fuss in the earlier
kexec enabling. The WBINVD is flushing private keyid cachelines so they
are not later written back over the new kernels memory. It needs to happen
after the last SEAMCALL that might have produced them. So this new
SEAMCALL is for something else, but also needs to be slotted with respect
to WBINVD.
Implementation
==============
The series adds:
1. A pre-requisite patch to move TDX error code definitions to a
shared location so that TDX_INTERRUPTED_RESUMABLE etc. are
accessible from arch/x86/virt/vmx/tdx/. This comes from the Dynamic
PAMT series [0], but is also needed by some other series, and can
benefit them all from an early merge. This has now also been reduced
to a simple header move, since the new additions in the original
patch aren't needed here.
2. A preparatory patch to move some straggling stuff into arch/x86 in the
wake of the VMXON series. As noted in the discussion in v2, this
may need coordination with Kai's patch [1] as they will cause a
conflict depending on which is merged first.
3. A tdx_sys_disable() helper that wraps calls TDH.SYS.DISABLE with a
retry loop to handle TDX_INTERRUPTED_RESUMABLE.
4. Integration into the kexec path: Remove the check for partial write
errata platforms as this is addressed by the SEAMCALL clearing any
poisoned memory locations. Call tdx_sys_disable() in tdx_shutdown
which is called via syscore ops in the kexec path. Call
tdx_sys_disable() in native_machine_crash_shutdown() to cover the
crash (kdump) path.
5. A patch to update (remove) the kexec section in TDX docs.
Testing
=======
The implementation is based on the draft TDX module spec available at
[2], and was smoke tested with an engineering build of the TDX module
that supports the new SEAMCALL. The new kernel was able to initialize
the TDX module successfully:
kvm: exiting hardware virtualization
kexec_core: Starting new kernel
Linux version 7.0.0-rc2-g0077f702b21c...
...
virt/tdx: 1034220 KB allocated for PAMT
virt/tdx: TDX-Module initialized
All the other TDX CI tests pass, and some other scenarios that were
manually tested and also pass or behave as expected:
- Running on a completely non-TDX system
- Running on a TDX capable system with an old module
- Running on a TDX capable system where the module hasn't been
initialized
During development further testing was done by mocking up the new
SEAMCALL to introduce delays and exercise the retry loops, combined with
kexec, kdump, reboot and shutdown flows.
Base
====
This series is based on the vmxon branch Sean pushed to kvm_x86,
kvm-x86-vmxon-2026.03.05.
[0]: https://lore.kernel.org/kvm/20260129011517.3545883-11-seanjc@google.com/
[1]: https://lore.kernel.org/lkml/20260312100009.924136-1-kai.huang@intel.com/
[2]: https://cdrdv2.intel.com/v1/dl/getContent/871617
---
Changes in v3:
- Various: Collect tags (Kai, Chao, Kiryl, Sean)
- Patch 1: Collapse to a simple header move, drop new defines, and
non-architectural software defines. (Sean)
- Patch 2: Add WBINVD in the tdx_offline_cpu() path (Sean, Dave)
- Add a paragraph to the seamcall wrapper commit message talking about
printed errors for seamcall failures (Rick)
- Patch 2: Reword comment in tdx_shutdown_cpu (Chao)
- Patch 3: Reword comment about the TDX_SYS_BUSY error case (Chao)
- Patch 3: Formatting fixes in comments (Kiryl)
- Link to v2: https://patch.msgid.link/20260323-fuller_tdx_kexec_support-v2-0-87a36409e051@intel.com
Changes in v2:
- Use patch 1 from the DPAMT series with other feedback (Kai)
- Fix commit message typo (s/adjust_/adjust /)
- In patch 2, drop "too late to be helpful" in favor of something more
explicit (Kai)
- Fix commit message typo in patch 2 (s/both/bother/)
- In patch 2, add a bit about dropping the TDX specific WBINVD (Kai)
- Reword some commit logs to use the imperative mood (Chao)
- Kai raised offline that TDH.SYS.DISABLE can return TDX_SYS_BUSY too.
In theory this could happen if another SEAMCALL happens concurrently,
however that contention should be short lived. Update the loop to
continue on a TDX_SYS_BUSY error code too. (Kai)
- Patch 3: Add a print for SEAMCALL errors reported by the TDX module
(excluding SW errors like #UD and #GP) (Kiryl)
- Patch 3: Add a sentence to the log about skipping enumeration for the
new SEAMCALL (Kiryl)
- Adjust the patch 4 subject (Chao)
- Add a new patch to update the docs (Chao)
- Smoke test with TDX module engineering build with the new SEAMCALL.
Kiryl Shutsemau (1):
x86/tdx: Move TDX architectural error codes into <asm/shared/tdx_errno.h>
Rick Edgecombe (2):
x86/virt/tdx: Pull kexec cache flush logic into arch/x86
x86/virt/tdx: Remove kexec docs
Vishal Verma (2):
x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
x86/tdx: Disable the TDX module during kexec and kdump
Documentation/arch/x86/tdx.rst | 7 -------
arch/x86/include/asm/shared/tdx.h | 1 +
arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h | 8 +++----
arch/x86/include/asm/tdx.h | 9 +++-----
arch/x86/kvm/vmx/tdx.h | 1 -
arch/x86/virt/vmx/tdx/tdx.h | 1 +
arch/x86/kernel/crash.c | 2 ++
arch/x86/kernel/machine_kexec_64.c | 16 --------------
arch/x86/kvm/vmx/tdx.c | 10 ---------
arch/x86/virt/vmx/tdx/tdx.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-----------
10 files changed, 63 insertions(+), 56 deletions(-)
--
2.53.0
---
Kiryl Shutsemau (1):
x86/tdx: Move TDX architectural error codes into <asm/shared/tdx_errno.h>
Rick Edgecombe (2):
x86/virt/tdx: Pull kexec cache flush logic into arch/x86
x86/virt/tdx: Remove kexec docs
Vishal Verma (2):
x86/virt/tdx: Add SEAMCALL wrapper for TDH.SYS.DISABLE
x86/tdx: Disable the TDX module during kexec and kdump
Documentation/arch/x86/tdx.rst | 7 ---
arch/x86/include/asm/shared/tdx.h | 1 +
.../{kvm/vmx => include/asm/shared}/tdx_errno.h | 8 +--
arch/x86/include/asm/tdx.h | 9 +--
arch/x86/kvm/vmx/tdx.h | 1 -
arch/x86/virt/vmx/tdx/tdx.h | 1 +
arch/x86/kernel/crash.c | 2 +
arch/x86/kernel/machine_kexec_64.c | 16 ------
arch/x86/kvm/vmx/tdx.c | 10 ----
arch/x86/virt/vmx/tdx/tdx.c | 64 ++++++++++++++++++----
10 files changed, 63 insertions(+), 56 deletions(-)
---
base-commit: f630de1f8d70d7e29e12bc25dc63f9c5f771dc59
change-id: 20260317-fuller_tdx_kexec_support-bc79694678be
Best regards,
--
Vishal Verma <vishal.l.verma@intel.com>
^ permalink raw reply
* Re: [PATCH v5 0/2] dma-buf: heaps: system: add an option to allocate explicitly shared/decrypted memory
From: Marek Szyprowski @ 2026-04-02 5:35 UTC (permalink / raw)
To: Sumit Semwal, Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, benjamin.gaignard,
Brian.Starkey, jstultz, tjmercier, christian.koenig, robin.murphy,
jgg, leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <CAO_48GEUXpcFBiyJAMgTcGPSq56-mZ0qnO3FrFRM2LoGd8W6HA@mail.gmail.com>
On 02.04.2026 06:41, Sumit Semwal wrote:
> On Thu, 26 Mar 2026 at 00:53, Jiri Pirko <jiri@resnulli.us> wrote:
>> From: Jiri Pirko <jiri@nvidia.com>
>>
>> Confidential computing (CoCo) VMs/guests, such as AMD SEV and Intel TDX,
>> run with private/encrypted memory which creates a challenge
>> for devices that do not support DMA to it (no TDISP support).
>>
>> For kernel-only DMA operations, swiotlb bounce buffering provides a
>> transparent solution by copying data through shared memory.
>> However, the only way to get this memory into userspace is via the DMA
>> API's dma_alloc_pages()/dma_mmap_pages() type interfaces which limits
>> the use of the memory to a single DMA device, and is incompatible with
>> pin_user_pages().
>>
>> These limitations are particularly problematic for the RDMA subsystem
>> which makes heavy use of pin_user_pages() and expects flexible memory
>> usage between many different DMA devices.
>>
>> This patch series enables userspace to explicitly request shared
>> (decrypted) memory allocations from new dma-buf system_cc_shared heap.
>> Userspace can mmap this memory and pass the dma-buf fd to other
>> existing importers such as RDMA or DRM devices to access the
>> memory. The DMA API is improved to allow the dma heap exporter to DMA
>> map the shared memory to each importing device.
> Thank you for the patch series, it looks good to me.
>
> Marek, if you are ok, please could you take it through your tree, with my
> Acked-by: Sumit Semwal <sumit.semwal@linaro.org>
I've applied both patches to dma-mapping-for-next. Thanks!
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
^ permalink raw reply
* Re: [PATCH v5 0/2] dma-buf: heaps: system: add an option to allocate explicitly shared/decrypted memory
From: Sumit Semwal @ 2026-04-02 4:41 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, benjamin.gaignard,
Brian.Starkey, jstultz, tjmercier, christian.koenig, m.szyprowski,
robin.murphy, jgg, leon, sean.anderson, ptesarik, catalin.marinas,
aneesh.kumar, suzuki.poulose, steven.price, thomas.lendacky,
john.allen, ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260325192352.437608-1-jiri@resnulli.us>
Hello Jiri,
On Thu, 26 Mar 2026 at 00:53, Jiri Pirko <jiri@resnulli.us> wrote:
>
> From: Jiri Pirko <jiri@nvidia.com>
>
> Confidential computing (CoCo) VMs/guests, such as AMD SEV and Intel TDX,
> run with private/encrypted memory which creates a challenge
> for devices that do not support DMA to it (no TDISP support).
>
> For kernel-only DMA operations, swiotlb bounce buffering provides a
> transparent solution by copying data through shared memory.
> However, the only way to get this memory into userspace is via the DMA
> API's dma_alloc_pages()/dma_mmap_pages() type interfaces which limits
> the use of the memory to a single DMA device, and is incompatible with
> pin_user_pages().
>
> These limitations are particularly problematic for the RDMA subsystem
> which makes heavy use of pin_user_pages() and expects flexible memory
> usage between many different DMA devices.
>
> This patch series enables userspace to explicitly request shared
> (decrypted) memory allocations from new dma-buf system_cc_shared heap.
> Userspace can mmap this memory and pass the dma-buf fd to other
> existing importers such as RDMA or DRM devices to access the
> memory. The DMA API is improved to allow the dma heap exporter to DMA
> map the shared memory to each importing device.
Thank you for the patch series, it looks good to me.
Marek, if you are ok, please could you take it through your tree, with my
Acked-by: Sumit Semwal <sumit.semwal@linaro.org>
Best,
Sumit.
>
> Based on dma-mapping-for-next e7442a68cd1ee797b585f045d348781e9c0dde0d
>
> Jiri Pirko (2):
> dma-mapping: introduce DMA_ATTR_CC_SHARED for shared memory
> dma-buf: heaps: system: add system_cc_shared heap for explicitly
> shared memory
>
> drivers/dma-buf/heaps/system_heap.c | 103 ++++++++++++++++++++++++++--
> include/linux/dma-mapping.h | 10 +++
> include/trace/events/dma.h | 3 +-
> kernel/dma/direct.h | 14 +++-
> kernel/dma/mapping.c | 13 +++-
> 5 files changed, 132 insertions(+), 11 deletions(-)
>
> --
> 2.51.1
>
^ permalink raw reply
* Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature
From: Huang, Kai @ 2026-04-02 1:06 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, Hansen, Dave, Edgecombe, Rick P,
yilun.xu@linux.intel.com, x86@kernel.org
Cc: Gao, Chao, Xu, Yilun, dave.hansen@linux.intel.com, kas@kernel.org,
baolu.lu@linux.intel.com, Li, Xiaoyao,
linux-kernel@vger.kernel.org, Verma, Vishal L, Jiang, Dave,
kvm@vger.kernel.org, Duan, Zhenzhong
In-Reply-To: <5183437e-bbe8-4453-9b03-9bef2e025c0a@intel.com>
On Wed, 2026-04-01 at 17:48 -0700, Dave Hansen wrote:
> On 4/1/26 17:40, Huang, Kai wrote:
> ...
> > They might not care about losing ~50M memory, though, but that's a different
> > story, and it could be more in the future.
>
> If anyone _does_ care about 50MB of memory, I expect they'll speak up.
> Until they do, could we please err on the side of least complexity?
Yeah agreed.
>
> I'm not even sure the Kconfig is worth it to be honest.
I am not sure either.
My main comment is to make the code more friendly to opt-in more features,
actually.
^ permalink raw reply
* Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature
From: Dave Hansen @ 2026-04-02 0:48 UTC (permalink / raw)
To: Huang, Kai, Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, Edgecombe, Rick P,
yilun.xu@linux.intel.com, x86@kernel.org
Cc: Gao, Chao, Xu, Yilun, dave.hansen@linux.intel.com,
baolu.lu@linux.intel.com, kas@kernel.org, Li, Xiaoyao,
linux-kernel@vger.kernel.org, Verma, Vishal L, Jiang, Dave,
kvm@vger.kernel.org, Duan, Zhenzhong
In-Reply-To: <847eb26878cb06afb47f4ecdb0bfe8ab7e432e8a.camel@intel.com>
On 4/1/26 17:40, Huang, Kai wrote:
...
> They might not care about losing ~50M memory, though, but that's a different
> story, and it could be more in the future.
If anyone _does_ care about 50MB of memory, I expect they'll speak up.
Until they do, could we please err on the side of least complexity?
I'm not even sure the Kconfig is worth it to be honest.
^ permalink raw reply
* Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature
From: Huang, Kai @ 2026-04-02 0:40 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, Edgecombe, Rick P,
yilun.xu@linux.intel.com, x86@kernel.org
Cc: Gao, Chao, Xu, Yilun, dave.hansen@linux.intel.com,
baolu.lu@linux.intel.com, kas@kernel.org, Li, Xiaoyao,
linux-kernel@vger.kernel.org, Verma, Vishal L, Jiang, Dave,
kvm@vger.kernel.org, Duan, Zhenzhong
In-Reply-To: <8c4dfadaf73ed175a959ab16335a8d90266453b5.camel@intel.com>
On Wed, 2026-04-01 at 23:53 +0000, Edgecombe, Rick P wrote:
> On Wed, 2026-04-01 at 23:42 +0000, Huang, Kai wrote:
> > Maybe a more generic comment:
> >
> > I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside
> > config_tdx_module(), especially currently we just unconditionally opt it in
> > if the module support this feature.
> >
> > Initializing TDX Connect (and other features via TDX Module Extensions)
> > consumes more memory. It would be better if we can choose to opt-in when
> > the kernel has enabled TDX Connect (or any other feature via TDX module
> > Extensions) in the Kconfig.
>
> Better how? TDX uses a lot of memory. There are many possible optimizations to
> reduce this. Why focus on this one? Do we think any TDX users would actually
> reconfigure there kernel for this reason?
>
> I mean, I don't actually know how much memory this is, but to me the reasoning
> doesn't seem in balance with the wider TDX situation.
In another patch, Yilun said:
For now, TDX Module Extensions consume relatively large amount of
memory (~50MB).
Distros tend to enable all I assume. I guess the CSPs tend to enable all as
well, but I am not sure whether CSPs can also choose to only enable basic
TDX w/o other features like runtime update, TDX Connect etc, depending on
how they want to "sell" TDX VMs.
E.g., I assume a TD without GPU passthrough could be cheaper the one which
has, right? Can the CSPs host such TDs on dedicated machine pools? Can
they choose to disable TDX Connect on these machines?
They might not care about losing ~50M memory, though, but that's a different
story, and it could be more in the future.
My thinking is, this series actually introduced a new "TDX_CONNNECT"
Kconfig, so why not only consume the memory when it's on?
At last, just my 2cents, I kinda overall don't agree we always assume
"everything will be on" but neglect avoiding unnecessary code/cost that is
not reachable when some option is off. That would defeat the purpose of
having Kconfig option.
^ permalink raw reply
* Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT
From: Huang, Kai @ 2026-04-02 0:05 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, yilun.xu@linux.intel.com,
x86@kernel.org
Cc: Gao, Chao, Edgecombe, Rick P, Xu, Yilun, Jiang, Dave,
dave.hansen@linux.intel.com, baolu.lu@linux.intel.com,
Duan, Zhenzhong, kas@kernel.org, Verma, Vishal L, Li, Xiaoyao,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <20260327160132.2946114-6-yilun.xu@linux.intel.com>
On Sat, 2026-03-28 at 00:01 +0800, Xu Yilun wrote:
> IOMMU_MT is another TDX Module defined structure similar to HPA_ARRAY_T
> and HPA_LIST_INFO. The difference is it requires multi-order contiguous
> pages for some entries. It adds an additional NUM_PAGES field for every
> multi-order page entry.
>
> Add a dedicated allocation helper for IOMMU_MT. Fortunately put_page()
> works well for both single pages and multi-order folios, simplifying the
> cleanup logic for all allocation methods.
Well I guess you can have a 'free_fn' to free the pages you allocated via
'alloc_fn'? Will this simplify the code and at least keep tdx_page_array
implementation cleaner?
It's strange that you only have a 'alloc_fn' but doesn't have a 'free_fn'
anyway.
^ permalink raw reply
* Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature
From: Edgecombe, Rick P @ 2026-04-01 23:53 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, Huang, Kai, yilun.xu@linux.intel.com,
x86@kernel.org
Cc: Gao, Chao, Xu, Yilun, Jiang, Dave, baolu.lu@linux.intel.com,
kas@kernel.org, dave.hansen@linux.intel.com, Li, Xiaoyao,
Verma, Vishal L, Duan, Zhenzhong, kvm@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <6a28710b65d1a291213d28a8297477bf0287df7d.camel@intel.com>
On Wed, 2026-04-01 at 23:42 +0000, Huang, Kai wrote:
> Maybe a more generic comment:
>
> I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside
> config_tdx_module(), especially currently we just unconditionally opt it in
> if the module support this feature.
>
> Initializing TDX Connect (and other features via TDX Module Extensions)
> consumes more memory. It would be better if we can choose to opt-in when
> the kernel has enabled TDX Connect (or any other feature via TDX module
> Extensions) in the Kconfig.
Better how? TDX uses a lot of memory. There are many possible optimizations to
reduce this. Why focus on this one? Do we think any TDX users would actually
reconfigure there kernel for this reason?
I mean, I don't actually know how much memory this is, but to me the reasoning
doesn't seem in balance with the wider TDX situation.
^ permalink raw reply
* Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature
From: Huang, Kai @ 2026-04-01 23:42 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, yilun.xu@linux.intel.com,
x86@kernel.org
Cc: Gao, Chao, Edgecombe, Rick P, Xu, Yilun, Jiang, Dave,
dave.hansen@linux.intel.com, baolu.lu@linux.intel.com,
Duan, Zhenzhong, kas@kernel.org, Verma, Vishal L, Li, Xiaoyao,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <20260327160132.2946114-9-yilun.xu@linux.intel.com>
> static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
> {
> struct tdx_module_args args = {};
> + u64 seamcall_fn = TDH_SYS_CONFIG_V0;
> u64 *tdmr_pa_array;
> size_t array_sz;
> int i, ret;
> @@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
> args.rcx = __pa(tdmr_pa_array);
> args.rdx = tdmr_list->nr_consumed_tdmrs;
> args.r8 = global_keyid;
> - ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
> +
> + if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
> + args.r9 |= TDX_FEATURES0_TDXCONNECT;
> + args.r11 = ktime_get_real_seconds();
> + /* These parameters requires version >= 1 */
> + seamcall_fn = TDH_SYS_CONFIG;
> + }
> +
> + ret = seamcall_prerr(seamcall_fn, &args);
>
> /* Free the array as it is not required anymore. */
> kfree(tdmr_pa_array);
> @@ -1537,6 +1546,11 @@ static int init_tdx_module(void)
> if (ret)
> goto err_free_pamts;
>
> + /* configuration to tdx module may change tdx_sysinfo, update it */
> + ret = get_tdx_sys_info(&tdx_sysinfo);
> + if (ret)
> + goto err_reset_pamts;
> +
> /* Config the key of global KeyID on all packages */
> ret = config_global_keyid();
> if (ret)
Maybe a more generic comment:
I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside
config_tdx_module(), especially currently we just unconditionally opt it in
if the module support this feature.
Initializing TDX Connect (and other features via TDX Module Extensions)
consumes more memory. It would be better if we can choose to opt-in when
the kernel has enabled TDX Connect (or any other feature via TDX module
Extensions) in the Kconfig.
Unfortunately we need to opt-in all these features together during module
initialization, so we cannot make tdx_enable() to accept the additional
features to enable, and in each in-kernel TDX user, call tdx_enable() with
the new feature that that TDX user concerns.
But I think it makes sense to have a dedicated place to calculate all opt-in
features. E.g., assuming we eventually are going to support TDX Connect and
live migration:
static u64 get_ext_features_tdx_connect(struct tdx_sys_info * sysinfo)
{
if (!IS_ENABLED(TDX_CONNECT))
return 0;
return sysinfo->features.tdx_features0 & TDX_FEATURES0_TDXCONNECT ?
TDX_FEATURES0_TDXCONNECT : 0;
}
static u64 get_ext_features_live_migration(struct tdx_sys_info *sysinfo)
{
u64 mig_features = TDX_FEATURES0_NRX | TDX_FEATURES0_NON_BLOCKING;
if (!IS_ENABLED(TDX_LIVE_MIGRATION))
return 0;
return sysinfo->features.tdx_features0 & mig_features;
}
static u64 calculate_ext_features(struct tdx_sys_info *sysinfo)
{
u64 ext_features = 0;
ext_features |= get_ext_features_tdx_connect(sysinfo);
ext_features |= get_ext_features_live_migration(sysinfo);
return ext_features;
}
int init_tdx_module()
{
u64 ext_features = calculate_ext_features(&tdx_sysinfo);
ret = config_tdx_module(&tdx_tdmr_list, &tdx_global_keyid,
ext_features);
/* do other initializations like TDH.SYS.KEY.CONFIG */
...
/*
* TDX Module Extension features must be initialized
* after TDH.SYS.KEY.CONFIG.
*/
if (ext_features)
ret = init_tdx_ext();
...
}
One nasty thing is per public spec R11 of TDH.SYS.CONFIG needs to be RTC if
TDX_CONNECT is on, so we still need some special handing in
config_tdx_module():
if (ext_features & TDX_RFEAURES0_TDXCONNECT)
args.r11 = ktime_get_real_seconds();
But I think this is acceptable.
^ permalink raw reply
* Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect
From: Huang, Kai @ 2026-04-01 21:36 UTC (permalink / raw)
To: Williams, Dan J, linux-pci@vger.kernel.org,
linux-coco@lists.linux.dev, yilun.xu@linux.intel.com,
x86@kernel.org
Cc: Gao, Chao, Edgecombe, Rick P, Xu, Yilun, Jiang, Dave,
dave.hansen@linux.intel.com, baolu.lu@linux.intel.com,
Duan, Zhenzhong, kas@kernel.org, Verma, Vishal L, Li, Xiaoyao,
kvm@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <20260327160132.2946114-7-yilun.xu@linux.intel.com>
On Sat, 2026-03-28 at 00:01 +0800, Xu Yilun wrote:
> Add reading of the global metadata for TDX Module Extensions & TDX
> Connect. Add them in a batch as TDX Connect is currently the only user
> of TDX Module Extensions and no way to initialize TDX Module Extensions
> without firstly enabling TDX Connect.
>
> TDX Module Extensions & TDX Connect are optional features enumerated by
> TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to
> avoid failing the whole TDX initialization.
Maybe it's better to split this patch into two, one to read generic "TDX
Module Extension" related global metadata, and the other to read TDX Connect
specific ones?
They are logically two separate things anyway. And there are other features
also need to enable TDX Module Extensions (e.g., NRX for migration), and we
can just reuse the generic metadata patch from this series.
^ permalink raw reply
* Re: [PATCH v2 2/5] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Dave Hansen @ 2026-04-01 18:30 UTC (permalink / raw)
To: Sean Christopherson, H. Peter Anvin
Cc: Rick P Edgecombe, Vishal L Verma, Kai Huang, bp@alien8.de,
x86@kernel.org, kas@kernel.org, mingo@redhat.com,
linux-kernel@vger.kernel.org, dave.hansen@linux.intel.com,
tglx@kernel.org, pbonzini@redhat.com, linux-coco@lists.linux.dev,
kvm@vger.kernel.org
In-Reply-To: <ac1f9KntTv0SaktS@google.com>
On 4/1/26 11:12, Sean Christopherson wrote:
> Sorry, "costly" wasn't the right word. I know WBINVD super
> expensive, but unless someone cares deeply about the latency of
> offlining a CPU after its down TDX stuff, the "cost" is effectively
> zero.
I once increased the CPU online/offline latency once and got nastygrams
from folks. IIRC, I added a synchronize_rcu() which incurs way more
latency than WBINVD, but folks _do_ care about CPU online/offline
latency surprisingly.
In this case, though, I'm happy to add the WBINVD for simplicity and
wait for a possible repeat of the torches an pitchforks.
^ permalink raw reply
* Re: [PATCH v2 2/5] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: Sean Christopherson @ 2026-04-01 18:12 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Dave Hansen, Rick P Edgecombe, Vishal L Verma, Kai Huang,
bp@alien8.de, x86@kernel.org, kas@kernel.org, mingo@redhat.com,
linux-kernel@vger.kernel.org, dave.hansen@linux.intel.com,
tglx@kernel.org, pbonzini@redhat.com, linux-coco@lists.linux.dev,
kvm@vger.kernel.org
In-Reply-To: <76F29857-47D8-470B-9F4D-DD98D8755EB0@zytor.com>
On Wed, Apr 01, 2026, H. Peter Anvin wrote:
> On April 1, 2026 8:03:02 AM PDT, Dave Hansen <dave.hansen@intel.com> wrote:
> >On 3/31/26 16:04, Sean Christopherson wrote:
> >> But unless the WBINVD is actually costly, why bother getting fancy?
> >
> >WBINVD might be the most expensive single instruction in the whole ISA.
> >
> >That said, I'd much rather have a potentially unnecessary WBINVD than
> >miss one. The thing I'd be worried about would be something wonky like:
> >
> > 1. CPU offline does WBINVD
> > 2. Some other TDX call gets made, dirties caches again
> > 3. tdx_offline_cpu() skips WBINVD
> >
> >So, let's just do both for now: Do WBINVD in tdx_offline_cpu() and
> >comment that it might be redundant with other things in the CPU offline
> >procedure.
> >
> >This really needs to be solved with infrastructure and keeping data
> >about the reasons for needing WBINVD, not relying on code ordering or
> >fragile semantics.
>
> It is, *by far*, the most expensive *uninterruptible* instruction in the ISA.
> REP string instructions can of course be arbitrarily long, but are
> interruptible and so don't really count.
>
> Some MSRs used during very early (pre-OS) initialization might be even slower
> on some implementations, but that's not visible to Linux and no workload of
> any kind is running.
Sorry, "costly" wasn't the right word. I know WBINVD super expensive, but unless
someone cares deeply about the latency of offlining a CPU after its down TDX stuff,
the "cost" is effectively zero.
^ permalink raw reply
* Re: [PATCH v2 2/5] x86/virt/tdx: Pull kexec cache flush logic into arch/x86
From: H. Peter Anvin @ 2026-04-01 17:42 UTC (permalink / raw)
To: Dave Hansen, Sean Christopherson, Rick P Edgecombe
Cc: Vishal L Verma, Kai Huang, bp@alien8.de, x86@kernel.org,
kas@kernel.org, mingo@redhat.com, linux-kernel@vger.kernel.org,
dave.hansen@linux.intel.com, tglx@kernel.org, pbonzini@redhat.com,
linux-coco@lists.linux.dev, kvm@vger.kernel.org
In-Reply-To: <830f1e46-0fb7-4756-827b-c8f46af24374@intel.com>
On April 1, 2026 8:03:02 AM PDT, Dave Hansen <dave.hansen@intel.com> wrote:
>On 3/31/26 16:04, Sean Christopherson wrote:
>> But unless the WBINVD is actually costly, why bother getting fancy?
>
>WBINVD might be the most expensive single instruction in the whole ISA.
>
>That said, I'd much rather have a potentially unnecessary WBINVD than
>miss one. The thing I'd be worried about would be something wonky like:
>
> 1. CPU offline does WBINVD
> 2. Some other TDX call gets made, dirties caches again
> 3. tdx_offline_cpu() skips WBINVD
>
>So, let's just do both for now: Do WBINVD in tdx_offline_cpu() and
>comment that it might be redundant with other things in the CPU offline
>procedure.
>
>This really needs to be solved with infrastructure and keeping data
>about the reasons for needing WBINVD, not relying on code ordering or
>fragile semantics.
It is, *by far*, the most expensive *uninterruptible* instruction in the ISA. REP string instructions can of course be arbitrarily long, but are interruptible and so don't really count.
Some MSRs used during very early (pre-OS) initialization might be even slower on some implementations, but that's not visible to Linux and no workload of any kind is running.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox