* Re: [PATCH v3 2/2] x86/sev: Add support to unaccept memory after hot-remove
From: Kiryl Shutsemau @ 2026-01-29 10:40 UTC (permalink / raw)
To: Dave Hansen
Cc: Pratik R. Sampat, linux-mm, linux-coco, x86, linux-kernel, tglx,
mingo, bp, dave.hansen, ardb, akpm, david, osalvador,
thomas.lendacky, michael.roth
In-Reply-To: <16aa84a9-83ce-4748-bc73-ccf5cb6ce376@intel.com>
On Wed, Jan 28, 2026 at 01:15:06PM -0800, Dave Hansen wrote:
> On 1/28/26 12:41, Pratik R. Sampat wrote:
> > +static inline void arch_unaccept_memory(phys_addr_t start, phys_addr_t end)
> > +{
> > + if (cc_platform_has(CC_ATTR_GUEST_SEV_SNP)) {
> > + snp_unaccept_memory(start, end);
> > + } else {
> > + panic("Cannot unaccept memory: unknown platform\n");
> > + }
> > +}
>
> This panic() is pretty nasty.
>
> Can't we just disable memory hotplug up front if it's:
>
> !cc_platform_has(CC_ATTR_GUEST_SEV_SNP)
>
> ?
I don't understand SEV-SNP situation, but I don't think we need to do
anything on unplug for TDX. We should expect the unplugged memory to be
removed from SEPT. If VMM doesn't do this, it is effectively DoS and we
don't protect against DoS in CoCo.
Converting the memory to shared will do no good for us.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCH v3 1/2] mm/memory_hotplug: Add support to accept memory during hot-add
From: Kiryl Shutsemau @ 2026-01-29 10:35 UTC (permalink / raw)
To: Pratik R. Sampat
Cc: linux-mm, linux-coco, x86, linux-kernel, tglx, mingo, bp,
dave.hansen, ardb, akpm, david, osalvador, thomas.lendacky,
michael.roth
In-Reply-To: <20260128204105.508855-2-prsampat@amd.com>
On Wed, Jan 28, 2026 at 02:41:04PM -0600, Pratik R. Sampat wrote:
> Confidential computing guests require memory to be accepted before use.
> The unaccepted memory bitmap maintained by firmware does not track
> most hotplugged memory ranges apart from system memory annotated to be
> cold plugged at boot.
>
> Explicitly validate and transition the newly added memory to a private
> state, making it usable by the guest.
>
> Signed-off-by: Pratik R. Sampat <prsampat@amd.com>
> ---
> drivers/firmware/efi/unaccepted_memory.c | 18 ++++++++++++++++++
> include/linux/mm.h | 5 +++++
> mm/memory_hotplug.c | 2 ++
> 3 files changed, 25 insertions(+)
>
> diff --git a/drivers/firmware/efi/unaccepted_memory.c b/drivers/firmware/efi/unaccepted_memory.c
> index c2c067eff634..5a4c8b0f56c8 100644
> --- a/drivers/firmware/efi/unaccepted_memory.c
> +++ b/drivers/firmware/efi/unaccepted_memory.c
> @@ -209,6 +209,24 @@ bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size)
> return ret;
> }
>
> +/*
> + * Unaccepted memory bitmap only covers initial boot memory and not the
> + * hotpluggable range that is part of SRAT parsing. However, some initial memory
> + * with the attribute EFI_MEMORY_HOT_PLUGGABLE can indicate boot time memory
> + * that can be hot-removed. Hence, handle acceptance in accordance with the
> + * unaccepted bitmap. Otherwise, perform the state change for the memory range
> + * up-front.
> + */
> +void accept_hotplug_memory(phys_addr_t start, unsigned long size)
> +{
> + if (range_contains_unaccepted_memory(start, size)) {
> + accept_memory(start, size);
> + return;
> + }
No. This is buggy. The memory has to be accepted regardless of state in
the bitmap. If the memory is ever unplugged the bitmap state is not
relevant.
So, accept it unconditionally and mark the memory accepted in the
bitmap.
> +
> + arch_accept_memory(start, start + size);
> +}
> +
> #ifdef CONFIG_PROC_VMCORE
> static bool unaccepted_memory_vmcore_pfn_is_ram(struct vmcore_cb *cb,
> unsigned long pfn)
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 15076261d0c2..2d3c1ea40606 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -4504,6 +4504,7 @@ int set_anon_vma_name(unsigned long addr, unsigned long size,
>
> bool range_contains_unaccepted_memory(phys_addr_t start, unsigned long size);
> void accept_memory(phys_addr_t start, unsigned long size);
> +void accept_hotplug_memory(phys_addr_t start, unsigned long size);
>
> #else
>
> @@ -4517,6 +4518,10 @@ static inline void accept_memory(phys_addr_t start, unsigned long size)
> {
> }
>
> +static inline void accept_hotplug_memory(phys_addr_t start, unsigned long size)
> +{
> +}
> +
> #endif
>
> static inline bool pfn_is_unaccepted_memory(unsigned long pfn)
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index a63ec679d861..549ccfd190ee 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1567,6 +1567,8 @@ int add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
> if (!strcmp(res->name, "System RAM"))
> firmware_map_add_hotplug(start, start + size, "System RAM");
>
> + accept_hotplug_memory(start, size);
> +
> /* device_online() will take the lock when calling online_pages() */
> mem_hotplug_done();
>
> --
> 2.52.0
>
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* COCONUT-SVSM Development Release v2026.01-devel
From: Jörg Rödel @ 2026-01-29 10:12 UTC (permalink / raw)
To: coconut-svsm, linux-coco
Hi,
The first development release of COCONUT-SVSM in 2026 it tagged and ready for
wider testing and use. This release includes 96 non-merge commits which bring
really nice improvements to the project.
The highlights include:
- Move to the Rust 2024 edition. This includes updates to adapt the
code-base to changes in Rust language semantics and quite a number of
updates for cargo-fmt compliance.
- Symbolized stack traces for easier debugging. When COCONUT-SVSM
prints a kernel stack trace it will now include the symbols in debug
builds.
- Boot flow improvements. The stage2 loader can now allocate directly
from the kernel heap and build the initial page-tables for
COCONUT-SVSM. Once the kernel boots it does not need to re-build its
page-tables.
- A critical bug has been fixed which caused boot failures using
upstream EDK2 firmware.
Besides these highlights there have been numerous other changes to improve the
code-base and fix smaller issues. A big THANKS again to the COCONUT-SVSM
community for all the efforts and contributions. The attached shortlog has all
the details.
Quite a few members of our community will attend FOSDEM in Brussels this
weekend. If you are interested come by and say Hi, we will coordinate us in our
project Matrix chat:
https://matrix.to/#/#coconut-svsm:matrix.org
I will hang out mostly in the Virtualization, Confidential Computing, and
Kernel Devrooms. Hope to see many of you there.
Regards,
Joerg
Shortlog:
Carlos López (52):
xbuild: add shorthand method to get component features
xbuild/features: simply return feature list to caller
xbuild: use features in make-based build
mm/alloc: constify allocate_slab_page()
mm/alloc: check slab size at compile time
mm/alloc: convert slab const generics to usize
address: implement NonNull to VirtAddr conversion
mm/alloc: type slab allocations as NonNull
mm/alloc: type SlabPage::next_page as NonNull
mm/alloc: ensure that SlabPage data pages are freed once
mm/alloc: introduce SlabCommon::allocate_page_slot()
mm/alloc: consolidate SlabPage initialization/destruction
mm/guestmem: implement NonNull to GuestPtr conversion
cpu/percpu: represent CAA as NonNull instead of VirtAddr
Makefile: build test kernels via xbuild
github/workflows: call xbuild on JSON recipes only
repo: move verification crates into verification/
platform: mark all page validation operations as unsafe
debug/stacktrace: cleanup argument usage
debug/stacktrace: do not allocate when printing a stacktrace
sev/secrets_page: remove SecretsPageRef
virtio-drivers: remove unused Cargo.toml file
cpuarch: update to Rust 2024 edition (no changes)
libtcgtpm: update to Rust 2024 edition (no changes)
release: update to Rust 2024 edition (no changes)
stage1: update to Rust 2024 edition (no changes)
test: update to Rust 2024 edition (no changes)
bootlib: update to Rust 2024 edition (format changes)
elf: update to Rust 2024 edition (format changes)
fuzz: update to Rust 2024 edition (format changes)
libaproxy: update to Rust 2024 edition (format changes)
syscall: update to Rust 2024 edition (format changes)
tools/aproxy: update to Rust 2024 edition (format changes)
tools/igvmmeasure: update to Rust 2024 edition (format changes)
verification: update to Rust 2024 edition (format changes)
virtio-drivers: update to Rust 2024 edition (format changes)
xbuild: update to Rust 2024 edition (format changes)
user: update to Rust 2024 edition (misc. changes)
tools/igvmbuilder: update to Rust 2024 edition (several changes)
kernel: update to Rust 2024 edition (several changes)
repo: use single workspace edition
scripts/pre-commit: pick up Rust edition from rustfmt.toml
elf/file: introduce ElfFile::read_verified_phdr()
elf/file: introduce ElfFile::read_verified_shdr()
build: keep ELF symbol information in debug builds
elf/file: parse .symtab section
elf/file: parse .strtab section
bootlib: add kernel symbol definitions
stage2: decouple ELF reading and loading
stage2: parse ELF symbol information
SVSM: introduce symbol resolution infrastructure
debug/stacktrace: resolve symbol addresses
Jon Lange (17):
igvmmeasure: remove SEV features check
sev: require the debug register virtualization feature
vc: remove DR7 ghcb test
stage2: allocate kernel launch info from kernel heap
svsm: eliminate static `LAUNCH_INFO`
kernel: remove valid page bitmap from stage2
stage2: remove heap alignment requirement
kernel: allocate initial kernel stack from kernel heap
kernel: free BSP init stack when it is no longer needed
kernel: alllocate CPUID page from kernel heap
kernel: alllocate secrets page from kernel heap
cpu/idt: don't panic on unhandled user-mode exceptions
cpu/idt: remove unused handlers
vpu/vc: panic on a page not validated #VC
tools/igvmbuilder: fix highest VTL for native builds
tdp: start APs without going through stage2
virtio: suppress virtio detection when no fw_cfg is present
Nicola Ramacciotti (7):
kernel/mm/alloc: Remove unnecessary any()
kernel/mm/alloc: Remove dead code allowance
kernel,stage1: Remove the explicit rust-version number
kernel: Apply clippy suggestion
repo: Update rust version to 1.87.0
kernel/mm/address_space: Fix conditional compilation in test module
kernel/lib: Use compact cfg_attr syntax for test inside svsm
Luigi Leonardi (5):
virtio: put block code behind the new `block` feature
verification: add missing SPDX header
virtio: introduce MMIOSlots structure
block: introduce BLOCK_DEVICE variable
virtio: integrate MmioSlot infrastructure with block device subsystem
Peter Fang (5):
kernel/igvm_params: Allow unaligned guest memory map
igvmbuilder/ovmf: Use (start, size) for regions instead of (start, end)
igvmbuilder/ovmf: Sanity check pre-validated regions
igvmbuilder: Sanity check firmware regions
kernel/config: Remove check_ovmf_regions()
Joerg Roedel (3):
kernel: Fix nightly warnings
kernel/vc: Fix nightly warnings
COCONUT-SVSM Release 2026.01-devel
Stefano Garzarella (2):
kernel: fix unused `slots` variable
virtio/mmio: move fw_cfg check into probe_mmio_slots()
Vaishali Thakkar (2):
sev/utils: Use iterator combinators for pvalidate and RMP revoke paths
sev/utils: Improve error handling for pvalidate callers
Ziqiao Zhou (2):
Upgrade verus lib to 2025-12-07-0054
verify_external: Remove spec for From, Into, and Integer in verify_external
Geoffrey Ndu (1):
kernel/protocol/attest: Add extended attestation support
^ permalink raw reply
* Re: [PATCH v3 06/26] x86/virt/tdx: Prepare to support P-SEAMLDR SEAMCALLs
From: Xu Yilun @ 2026-01-29 9:46 UTC (permalink / raw)
To: Dave Hansen
Cc: Chao Gao, linux-coco, linux-kernel, kvm, x86, reinette.chatre,
ira.weiny, kai.huang, dan.j.williams, sagis, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma, Farrah Chen, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, H. Peter Anvin
In-Reply-To: <e2245231-ee39-40aa-bfdc-e43419fa30f4@intel.com>
> > static __always_inline int sc_retry_prerr(sc_func_t func,
> > sc_err_func_t err_func,
> > u64 fn, struct tdx_module_args *args)
> > @@ -96,4 +119,7 @@ static __always_inline int sc_retry_prerr(sc_func_t func,
> > #define seamcall_prerr_ret(__fn, __args) \
> > sc_retry_prerr(__seamcall_ret, seamcall_err_ret, (__fn), (__args))
> >
> > +#define seamldr_prerr(__fn, __args) \
> > + sc_retry_prerr(__seamcall, seamldr_err, (__fn), (__args))
> > +
> > #endif
>
> So, honestly, for me, it's a NAK for this whole patch.
>
> Go change the P-SEAMLDR to use the same error code as the TDX module,
> and fix the documentation. No kernel changes, please.
I'm thinking of ways to avoid a new pseamldr version.
Could we just ask for a unified error code space for both SEAMCALL &
SEAMLDR CALL, eliminating overlaps. There is no overlap now, so this is
just another documentation fix.
Then with all the doc fixes, we only need minor code change:
@@ -127,7 +127,8 @@ static __always_inline u64 sc_retry(sc_func_t func, u64 fn,
preempt_disable();
ret = __seamcall_dirty_cache(func, fn, args);
preempt_enable();
- } while (ret == TDX_RND_NO_ENTROPY && --retry);
+ } while ((ret == TDX_RND_NO_ENTROPY ||
+ ret == SEAMLDR_RND_NO_ENTROPY) && --retry);
I think this is a balance. The existing error code philosophy for SEAM
is as informative as possible, e.g. all kinds of xxx_INVALID,
SEAMLDR_RND_NO_ENTROPY is not that evil among 200+ other error codes.
^ permalink raw reply
* Re: SVSM Development Call January 28, 2026
From: Jörg Rödel @ 2026-01-29 9:40 UTC (permalink / raw)
To: coconut-svsm, linux-coco
In-Reply-To: <ihxnjxdpnguz4ajm44jgkldpg4oddyd6drrtqloawbuyo5ee65@c5dvv3s2g5fi>
Meeting minutes are in this PR:
https://github.com/coconut-svsm/governance/pull/94
-Joerg
^ permalink raw reply
* Re: [PATCH v3 03/26] x86/virt/tdx: Move low level SEAMCALL helpers out of <asm/tdx.h>
From: Chao Gao @ 2026-01-29 8:04 UTC (permalink / raw)
To: Dave Hansen
Cc: linux-coco, linux-kernel, kvm, x86, reinette.chatre, ira.weiny,
kai.huang, dan.j.williams, yilun.xu, sagis, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, H. Peter Anvin
In-Reply-To: <3e134fb9-95c0-442b-8a25-834ffd8d87f9@intel.com>
On Wed, Jan 28, 2026 at 08:37:35AM -0800, Dave Hansen wrote:
>On 1/23/26 06:55, Chao Gao wrote:
>> +++ b/arch/x86/virt/vmx/tdx/seamcall.h
>
>Moving the code to a local header is a good thing. The more private
>these things are, the better.
>
>I _do_ like when I see these things have a label in the filename like:
>
> internal.h
>
>or even:
>
> seamcall_internal.h
>
>That really catches your eye. It would also be ideal to have a small
>blurb at the top of the file to say what its scope is, just to explain
>what folks should be adding to it or not.
>
>If you get a chance to add those, all the better. But either way:
>
>Acked-by: Dave Hansen <dave.hansen@linux.intel.com>
Thanks.
I will rename it to "seamcall_internal.h" and add the following at the top:
/*
* SEAMCALL utilities for TDX host-side operations.
*
* Provides convenient wrappers around SEAMCALL assembly with retry logic,
* error reporting and cache coherency tracking.
*/
^ permalink raw reply
* Re: [PATCH v3 05/26] coco/tdx-host: Expose TDX Module version
From: Xu Yilun @ 2026-01-29 7:38 UTC (permalink / raw)
To: Chao Gao
Cc: linux-coco, linux-kernel, kvm, x86, reinette.chatre, ira.weiny,
kai.huang, dan.j.williams, sagis, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma
In-Reply-To: <20260123145645.90444-6-chao.gao@intel.com>
On Fri, Jan 23, 2026 at 06:55:13AM -0800, Chao Gao wrote:
> For TDX Module updates, userspace needs to select compatible update
> versions based on the current module version. This design delegates
> module selection complexity to userspace because TDX Module update
> policies are complex and version series are platform-specific.
>
> For example, the 1.5.x series is for certain platform generations, while
> the 2.0.x series is intended for others. And TDX Module 1.5.x may be
> updated to 1.5.y but not to 1.5.y+1.
>
> Expose the TDX Module version to userspace via sysfs to aid module
> selection. Since the TDX faux device will drive module updates, expose
> the version as its attribute.
>
> This approach follows the pattern used by microcode updates and other
> CoCo implementations:
>
> 1. AMD has a PCI device for the PSP for SEV which provides an existing
> place to hang their equivalent metadata.
>
> 2. ARM CCA will likely have a faux device (although it isn't obvious if
> they have a need to export version information there) [1]
>
> 3. Microcode revisions are exposed as CPU device attributes
>
> One bonus of exposing TDX Module version via sysfs is: TDX Module
> version information remains available even after dmesg logs are cleared.
>
> Signed-off-by: Chao Gao <chao.gao@intel.com>
> Link: https://lore.kernel.org/all/2025073035-bulginess-rematch-b92e@gregkh/ # [1]
Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
^ permalink raw reply
* Re: [PATCH v3 04/26] coco/tdx-host: Introduce a "tdx_host" device
From: Xu Yilun @ 2026-01-29 7:26 UTC (permalink / raw)
To: Binbin Wu
Cc: Chao Gao, linux-coco, linux-kernel, kvm, x86, reinette.chatre,
ira.weiny, kai.huang, dan.j.williams, sagis, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma, Jonathan Cameron, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, H. Peter Anvin
In-Reply-To: <2db22e08-88cd-4873-9645-a2e17af29220@linux.intel.com>
> > index cb52021912b3..b323b0ae4f82 100644
> > --- a/drivers/virt/coco/Makefile
> > +++ b/drivers/virt/coco/Makefile
> > @@ -6,6 +6,7 @@ obj-$(CONFIG_EFI_SECRET) += efi_secret/
> > obj-$(CONFIG_ARM_PKVM_GUEST) += pkvm-guest/
> > obj-$(CONFIG_SEV_GUEST) += sev-guest/
> > obj-$(CONFIG_INTEL_TDX_GUEST) += tdx-guest/
> > +obj-$(CONFIG_INTEL_TDX_HOST) += tdx-host/
>
> IIUC, the folder name "tdx-host" here stands for TDX host services?
Yes. But I think it is fine here to express "seach into the folder if
dependency meets".
For this patch,
Reviewed-by: Xu Yilun <yilun.xu@linux.intel.com>
> Should it use CONFIG_TDX_HOST_SERVICES here?
^ permalink raw reply
* Re: [PATCH v3 01/26] x86/virt/tdx: Print SEAMCALL leaf numbers in decimal
From: Chao Gao @ 2026-01-29 5:44 UTC (permalink / raw)
To: Dave Hansen
Cc: linux-coco, linux-kernel, kvm, x86, reinette.chatre, ira.weiny,
kai.huang, dan.j.williams, yilun.xu, sagis, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma, Kirill A. Shutemov, Farrah Chen,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, H. Peter Anvin
In-Reply-To: <fafd9381-b8be-40eb-a68f-da4c81e2653c@intel.com>
On Wed, Jan 28, 2026 at 08:26:43AM -0800, Dave Hansen wrote:
>On 1/23/26 06:55, Chao Gao wrote:
>> Both TDX spec and kernel defines SEAMCALL leaf numbers as decimal. Printing
>> them in hex makes no sense. Correct it.
>
>This patch has zero to do with "Runtime TDX Module update support". Why
>is it in this series?
Will drop it.
It was included because it came up during review of previous versions.
^ permalink raw reply
* 6.18 CVM guest kernel boot issues with non-UEFI bootloader
From: Kevin Hui @ 2026-01-29 3:57 UTC (permalink / raw)
To: linux-coco
Cc: thomas.lendacky, youngjaelee, hvolkmer, chrisboltz, tzn,
andrisaar, grobler
Hello,
We’ve been using a non-UEFI bootloader to launch SEV-SNP CVMs and noticed that
there’s an issue with booting the newest 6.18 guest kernel with
the stage0 bootloader (https://github.com/project-oak/oak/tree/main/stage0_bin).
The guest kernel boots successfully with 6.17 and below, but fails with 6.18. We
were able to reproduce this with host kernels 6.13.2 and 6.18.3 on Milan/Genoa
hosts.
We traced the commit that started causing boot issues to
68a501d7fd82454525797971c6a0005ceeb93153 and noticed that for some reason the
variable snp_vmpl was non-zero, even though our stack doesn’t run SVSM. This
triggers the sev_es_terminate() call and subsequently crashes the CVM. We
noticed that the commit removes a supposedly redundant rmpadjust() check, but
from our observations it seems that the failed rmpadjust() short-circuited the
check and avoided the underlying issue.
I was chatting with Tom about this, and taking a deeper look at the issue, we
suspect that the BSS is cleared after the sev_enable() call in
arch/x86/boot/compressed/head_64.S, and that because of this snp_vmpl contains
random junk and is not zeroed. When coming through UEFI, it seems that the BSS
is cleared via drivers/firmware/efi/libstub/x86-stub.c, but in a non-UEFI
bootloader there is no call to startup_64 and so this path is never invoked,
leaving whatever random data was in bss to remain.
Perhaps the proper fix for this is to put the variables that are set as part of
sev_enable() into .data so that both non-UEFI and UEFI bootloaders will have the
same treatment from the kernel, but I would love to hear everyone’s thoughts on
this.
Thanks,
Kevin
^ permalink raw reply
* Re: [PATCH v3 00/26] Runtime TDX Module update support
From: Chao Gao @ 2026-01-29 1:51 UTC (permalink / raw)
To: Sagi Shahar
Cc: linux-coco, linux-kernel, kvm, x86, reinette.chatre, ira.weiny,
kai.huang, dan.j.williams, yilun.xu, vannapurve, paulmck,
nik.borisov, zhenzhong.duan, seanjc, rick.p.edgecombe, kas,
dave.hansen, vishal.l.verma, Borislav Petkov, H. Peter Anvin,
Ingo Molnar, Paolo Bonzini, Thomas Gleixner
In-Reply-To: <CAAhR5DG7EOpmKYV4WmiyNYr14rKMNuTcqgvoaeZt5-==kSPmuw@mail.gmail.com>
On Wed, Jan 28, 2026 at 11:52:57AM -0600, Sagi Shahar wrote:
>On Fri, Jan 23, 2026 at 9:00 AM Chao Gao <chao.gao@intel.com> wrote:
>>
>> Hi Reviewers,
>>
>> With this posting, I'm hoping to collect more Reviewed-by or Acked-by tags.
>> Dave, since this version is still light on acks, it might not be ready for
>> your review.
>>
>> Changelog:
>> v2->v3:
>> - Make this series self-contained and independently runnable, testable and
>> reviewable by
>>
>> * Including dependent patches such as TDX Module version exposure and TDX
>> faux device creation
>
>I see "x86/virt/tdx: Retrieve TDX module version" and "x86/virt/tdx:
>Print TDX module version during init" in the github link but I don't
>see them as part of this series. Were they posted/accepted as part of
>a different series?
Yes. https://lore.kernel.org/kvm/20260109-tdx_print_module_version-v2-0-e10e4ca5b450@intel.com/
^ permalink raw reply
* Re: [PATCH v4 07/16] x86/virt/tdx: Add tdx_alloc/free_page() helpers
From: Sean Christopherson @ 2026-01-29 1:19 UTC (permalink / raw)
To: Rick P Edgecombe
Cc: binbin.wu@linux.intel.com, kvm@vger.kernel.org,
linux-coco@lists.linux.dev, Kai Huang, Xiaoyao Li, Dave Hansen,
Yan Y Zhao, Binbin Wu, kas@kernel.org, mingo@redhat.com,
linux-kernel@vger.kernel.org, pbonzini@redhat.com, Isaku Yamahata,
tglx@linutronix.de, Vishal Annapurve, Chao Gao, bp@alien8.de,
x86@kernel.org
In-Reply-To: <67d55b24ef1a80af615c3672e8436e0ac32e8efa.camel@intel.com>
Gah, forgot to hit "send" on this.
On Wed, Nov 26, 2025, Rick P Edgecombe wrote:
> On Tue, 2025-11-25 at 16:09 +0800, Binbin Wu wrote:
> > On 11/21/2025 8:51 AM, Rick Edgecombe wrote:
> > > +/*
> > > + * The TDX spec treats the registers like an array, as they are ordered
> > > + * in the struct. The array size is limited by the number or registers,
> > > + * so define the max size it could be for worst case allocations and sanity
> > > + * checking.
> > > + */
> > > +#define MAX_TDX_ARG_SIZE(reg) (sizeof(struct tdx_module_args) - \
> > > + offsetof(struct tdx_module_args, reg))
> >
> > This should be the maximum number of registers could be used to pass the
> > addresses of the pages (or other information), it needs to be divided by sizeof(u64).
>
> Oops, right.
>
> >
> > Also, "SIZE" in the name could be confusing.
>
> Yes LENGTH is probably better.
>
> >
> > > +#define TDX_ARG_INDEX(reg) (offsetof(struct tdx_module_args, reg) / \
> > > + sizeof(u64))
Honestly, the entire scheme is a mess. Four days of staring at this and I finally
undertand what the code is doing. The whole "struct tdx_module_array_args" union
is completely unnecessary, the resulting args.args crud is ugly, having a pile of
duplicate accessors is brittle, the code obfuscates a simple concept, and the end
result doesn't provide any actual protection since the kernel will happily overflow
the buffer after the WARN.
It's also relying on the developer to correctly copy+paste the same register in
multiple locations: ~5 depending on how you want to count.
static u64 *dpamt_args_array_ptr_r12(struct tdx_module_array_args *args)
#1
{
WARN_ON_ONCE(tdx_dpamt_entry_pages() > MAX_TDX_ARGS(r12));
#2
return &args->args_array[TDX_ARG_INDEX(r12)];
#3
u64 guest_memory_pamt_page[MAX_TDX_ARGS(r12)];
#4
u64 *args_array = dpamt_args_array_ptr_r12(&args);
#5
After all of that boilerplate, the caller _still_ has to do the actual memcpy(),
and for me at least, all of the above makes it _harder_ to understand what the code
is doing.
Drop the struct+union overlay and just provide a helper with wrappers to copy
to/from a tdx_module_args structure. It's far from bulletproof, but it at least
avoids an immediate buffer overflow, and defers to the kernel owner with respect
to handling uninitialized stack data.
/*
* For SEAMCALLs that pass a bundle of pages, the TDX spec treats the registers
* like an array, as they are ordered in the struct. The effective array size
* is (obviously) limited by the number or registers, relative to the starting
* register. Fill the register array at a given starting register, with sanity
* checks to avoid overflowing the args structure.
*/
static void dpamt_copy_regs_array(struct tdx_module_args *args, void *reg,
u64 *pamt_pa_array, bool copy_to_regs)
{
int size = tdx_dpamt_entry_pages() * sizeof(*pamt_pa_array);
if (WARN_ON_ONCE(reg + size > (void *)args) + sizeof(*args))
return;
/* Copy PAMT page PA's to/from the struct per the TDX ABI. */
if (copy_to_regs)
memcpy(reg, pamt_pa_array, size);
else
memcpy(pamt_pa_array, reg, size);
}
#define dpamt_copy_from_regs(dst, args, reg) \
dpamt_copy_regs_array(args, &(args)->reg, dst, false)
#define dpamt_copy_to_regs(args, reg, src) \
dpamt_copy_regs_array(args, &(args)->reg, src, true)
As far as the on-stack allocations go, why bother being precise? Except for
paranoid setups which explicitly initialize the stack, "allocating" ~48 unused
bytes is literally free. Not to mention the cost relative to the latency of a
SEAMCALL is in the noise.
/*
* When declaring PAMT arrays on the stack, use the maximum theoretical number
* of entries that can be squeezed into a SEAMCALL, as stack allocations are
* practically free, i.e. any wasted space is a non-issue.
*/
#define MAX_NR_DPAMT_ARGS (sizeof(struct tdx_module_args) / sizeof(u64))
With that, callers don't have to regurgitate the same register multiple times,
and we don't need a new wrapper for every variation of SEAMCALL. E.g.
u64 pamt_pa_array[MAX_NR_DPAMT_ARGS];
...
bool dpamt = tdx_supports_dynamic_pamt(&tdx_sysinfo) && level == PG_LEVEL_2M;
u64 pamt_pa_array[MAX_NR_DPAMT_ARGS];
struct tdx_module_args args = {
.rcx = gpa | pg_level_to_tdx_sept_level(level),
.rdx = tdx_tdr_pa(td),
.r8 = page_to_phys(new_sp),
};
u64 ret;
if (!tdx_supports_demote_nointerrupt(&tdx_sysinfo))
return TDX_SW_ERROR;
if (dpamt) {
if (alloc_pamt_array(pamt_pa_array, pamt_cache))
return TDX_SW_ERROR;
dpamt_copy_to_regs(&args, r12, pamt_pa_array);
}
Which to me is easier to read and much more intuitive than:
u64 guest_memory_pamt_page[MAX_TDX_ARGS(r12)];
struct tdx_module_array_args args = {
.args.rcx = gpa | pg_level_to_tdx_sept_level(level),
.args.rdx = tdx_tdr_pa(td),
.args.r8 = PFN_PHYS(page_to_pfn(new_sp)),
};
struct tdx_module_array_args retry_args;
int i = 0;
u64 ret;
if (dpamt) {
u64 *args_array = dpamt_args_array_ptr_r12(&args);
if (alloc_pamt_array(guest_memory_pamt_page, pamt_cache))
return TDX_SW_ERROR;
/*
* Copy PAMT page PAs of the guest memory into the struct per the
* TDX ABI
*/
memcpy(args_array, guest_memory_pamt_page,
tdx_dpamt_entry_pages() * sizeof(*args_array));
}
^ permalink raw reply
* [RFC PATCH v5 45/45] KVM: TDX: Turn on PG_LEVEL_2M
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
From: Yan Zhao <yan.y.zhao@intel.com>
Turn on PG_LEVEL_2M in tdx_gmem_private_max_mapping_level() when TDX huge
page is enabled and TD is RUNNABLE.
Introduce a module parameter named "tdx_huge_page" for kvm-intel.ko to
enable/disable TDX huge page. Turn TDX huge page off if the TDX module does
not support TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY.
Force page size to 4KB during TD build time to simplify code design, since
- tdh_mem_page_add() only adds private pages at 4KB.
- The amount of initial memory pages is usually limited (e.g. ~4MB in a
typical linux TD).
Update the warnings and KVM_BUG_ON() info to match the conditions when 2MB
mappings are permitted.
Signed-off-by: Xiaoyao Li <xiaoyao.li@intel.com>
Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/vmx/tdx.c | 37 ++++++++++++++++++++++++++++++-------
1 file changed, 30 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 774d395e5c73..8f9b4ad9871f 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -55,6 +55,8 @@
bool enable_tdx __ro_after_init;
module_param_named(tdx, enable_tdx, bool, 0444);
+static bool __read_mostly enable_tdx_huge_page = true;
+module_param_named(tdx_huge_page, enable_tdx_huge_page, bool, 0444);
#define TDX_SHARED_BIT_PWL_5 gpa_to_gfn(BIT_ULL(51))
#define TDX_SHARED_BIT_PWL_4 gpa_to_gfn(BIT_ULL(47))
@@ -1703,8 +1705,9 @@ static int tdx_sept_map_leaf_spte(struct kvm *kvm, gfn_t gfn, u64 new_spte,
kvm_pfn_t pfn = spte_to_pfn(new_spte);
int ret;
- /* TODO: handle large pages. */
- if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
+ /* TODO: Support hugepages when building the initial TD image. */
+ if (KVM_BUG_ON(level != PG_LEVEL_4K &&
+ to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE, kvm))
return -EIO;
if (KVM_BUG_ON(!vcpu, kvm))
@@ -1885,10 +1888,6 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
return -EIO;
- /* TODO: handle large pages. */
- if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
- return -EIO;
-
err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
level, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
@@ -3474,12 +3473,34 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
return ret;
}
+/*
+ * For private pages:
+ *
+ * Force KVM to map at 4KB level when !enable_tdx_huge_page (e.g., due to
+ * incompatible TDX module) or before TD state is RUNNABLE.
+ *
+ * Always allow KVM to map at 2MB level in other cases, though KVM may still map
+ * the page at 4KB (i.e., passing in PG_LEVEL_4K to AUG) due to
+ * (1) the backend folio is 4KB,
+ * (2) disallow_lpage restrictions:
+ * - mixed private/shared pages in the 2MB range
+ * - level misalignment due to slot base_gfn, slot size, and ugfn
+ * - guest_inhibit bit set due to guest's 4KB accept level
+ * (3) page merging is disallowed (e.g., when part of a 2MB range has been
+ * mapped at 4KB level during TD build time).
+ */
int tdx_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private)
{
if (!is_private)
return 0;
- return PG_LEVEL_4K;
+ if (!enable_tdx_huge_page)
+ return PG_LEVEL_4K;
+
+ if (unlikely(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE))
+ return PG_LEVEL_4K;
+
+ return PG_LEVEL_2M;
}
static int tdx_online_cpu(unsigned int cpu)
@@ -3665,6 +3686,8 @@ static int __init __tdx_bringup(void)
if (misc_cg_set_capacity(MISC_CG_RES_TDX, tdx_get_nr_guest_keyids()))
goto get_sysinfo_err;
+ if (enable_tdx_huge_page && !tdx_supports_demote_nointerrupt(tdx_sysinfo))
+ enable_tdx_huge_page = false;
/*
* Leave hardware virtualization enabled after TDX is enabled
* successfully. TDX CPU hotplug depends on this.
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 44/45] KVM: x86/mmu: Add support for splitting S-EPT hugepages on conversion
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Add support for splitting S-EPT hugepages in preparation for converting a
subset of a hugepage to be shared, as KVM must precisely zap/remove S-EPT
entries to avoid clobbering guest memory (the lifetime of guest private
memory is tied to the S-EPT). I.e. KVM needs to first split a hugepage so
that only the to-be-converted small pages can be zapped.
To avoid unnecessary work, e.g. if only the tail/end page of massive region
isn't aligned to the conversion, explicitly detect unaligned head and tail
pages relative to the max page size support by KVM, i.e. head/tail pages
that will undergo partial conversion.
To support splitting an S-EPT hugepage without a vCPU, add a per-VM PAMT
cache, along with a mutex to guard the cache. Using a mutex, e.g. versus
a spinlock, is important at it allows KVM to allocate memory *without*
dropping the lock, i.e. so that the PAMT cache can be topped-up as needed
without needed to juggle arch.tdp_mmu_external_cache_lock.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/include/asm/kvm_host.h | 8 +++-
arch/x86/kvm/mmu/mmu.c | 2 +-
arch/x86/kvm/mmu/tdp_mmu.c | 72 +++++++++++++++++++++++++++++++--
arch/x86/kvm/vmx/tdx.c | 34 +++++++++++++---
arch/x86/kvm/vmx/tdx.h | 2 +
5 files changed, 107 insertions(+), 11 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 385f1cf32d70..54dea90a53dc 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1563,6 +1563,12 @@ struct kvm_arch {
* the code to do so.
*/
spinlock_t tdp_mmu_pages_lock;
+
+ /*
+ * Protect the per-VM cache of pre-allocate pages used to populate the
+ * Dynamic PAMT when splitting S-EPT huge pages without a vCPU.
+ */
+ struct mutex tdp_mmu_external_cache_lock;
#endif /* CONFIG_X86_64 */
/*
@@ -1861,7 +1867,7 @@ struct kvm_x86_ops {
u64 new_spte, enum pg_level level);
void (*reclaim_external_sp)(struct kvm *kvm, gfn_t gfn,
struct kvm_mmu_page *sp);
- int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min);
+ int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu, int min);
bool (*has_wbinvd_exit)(void);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c2765bfc8492..62bf6bec2df2 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -606,7 +606,7 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
if (r)
return r;
- r = kvm_x86_call(topup_external_cache)(vcpu, PT64_ROOT_MAX_LEVEL);
+ r = kvm_x86_call(topup_external_cache)(vcpu->kvm, vcpu, PT64_ROOT_MAX_LEVEL);
if (r)
return r;
}
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index c46ebdacdb50..3181406c5e0b 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1447,7 +1447,8 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm,
return spte_set;
}
-static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct tdp_iter *iter)
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,
+ struct tdp_iter *iter)
{
struct kvm_mmu_page *sp;
@@ -1464,7 +1465,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct tdp_iter *iter)
if (!sp->external_spt)
goto err_external_spt;
- if (kvm_x86_call(topup_external_cache)(kvm_get_running_vcpu(), 1))
+ if (kvm_x86_call(topup_external_cache)(kvm, kvm_get_running_vcpu(), 1))
goto err_external_split;
}
@@ -1556,7 +1557,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
else
write_unlock(&kvm->mmu_lock);
- sp = tdp_mmu_alloc_sp_for_split(&iter);
+ sp = tdp_mmu_alloc_sp_for_split(kvm, &iter);
if (shared)
read_lock(&kvm->mmu_lock);
@@ -1631,9 +1632,74 @@ int kvm_tdp_mmu_split_huge_pages(struct kvm_vcpu *vcpu, gfn_t start, gfn_t end,
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_split_huge_pages);
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static int __tdp_mmu_split_mirror_huge_pages(struct kvm *kvm,
+ struct kvm_mmu_page *root,
+ gfn_t gfn, int target_level)
+{
+ gfn_t end = gfn + KVM_PAGES_PER_HPAGE(target_level + 1);
+
+ return tdp_mmu_split_huge_pages_root(kvm, root, gfn, end, target_level, false);
+}
+
+static int tdp_mmu_split_mirror_huge_pages(struct kvm *kvm,
+ struct kvm_mmu_page *root,
+ gfn_t start, gfn_t end, int level)
+{
+
+ gfn_t head = gfn_round_for_level(start, level + 1);
+ gfn_t tail = gfn_round_for_level(end, level + 1);
+ int r;
+
+ if (head != start) {
+ r = __tdp_mmu_split_mirror_huge_pages(kvm, root, head, level);
+ if (r)
+ return r;
+ }
+
+ if (tail != end && (head != tail || head == start)) {
+ r = __tdp_mmu_split_mirror_huge_pages(kvm, root, tail, level);
+ if (r)
+ return r;
+ }
+
+ return 0;
+}
+
int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t start, gfn_t end,
bool to_private)
{
+ struct kvm_mmu_page *root;
+ int r;
+
+ /*
+ * When converting from private=>shared, KVM must first split potential
+ * hugepages, as KVM mustn't overzap private mappings for TDX guests,
+ * i.e. must zap _exactly_ [start, end). Split potential hugepages at
+ * the head and tail of the to-be-converted (and thus zapped) range so
+ * that KVM doesn't overzap due to dropping a hugepage that doesn't
+ * fall wholly inside the range.
+ */
+ if (to_private || !kvm_has_mirrored_tdp(kvm))
+ return 0;
+
+ /*
+ * Acquire the external cache lock, a.k.a. the Dynamic PAMT lock, to
+ * protect the per-VM cache of pre-allocate pages used to populate the
+ * Dynamic PAMT when splitting S-EPT huge pages.
+ */
+ guard(mutex)(&kvm->arch.tdp_mmu_external_cache_lock);
+
+ guard(write_lock)(&kvm->mmu_lock);
+
+ /*
+ * TODO: Also split from PG_LEVEL_1G => PG_LEVEL_2M when KVM supports
+ * 1GiB S-EPT pages.
+ */
+ __for_each_tdp_mmu_root_yield_safe(kvm, root, 0, KVM_MIRROR_ROOTS) {
+ r = tdp_mmu_split_mirror_huge_pages(kvm, root, start, end, PG_LEVEL_4K);
+ if (r)
+ return r;
+ }
return 0;
}
#endif /* CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 098954f5e07c..774d395e5c73 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -607,6 +607,8 @@ void tdx_vm_destroy(struct kvm *kvm)
{
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ tdx_free_pamt_cache(&kvm_tdx->pamt_cache);
+
tdx_reclaim_td_control_pages(kvm);
kvm_tdx->state = TD_STATE_UNINITIALIZED;
@@ -629,6 +631,8 @@ int tdx_vm_init(struct kvm *kvm)
{
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ tdx_init_pamt_cache(&kvm_tdx->pamt_cache);
+
kvm->arch.has_protected_state = true;
/*
* TDX Module doesn't allow the hypervisor to modify the EOI-bitmap,
@@ -1621,15 +1625,32 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)
td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa);
}
-static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min)
+static struct tdx_pamt_cache *tdx_get_pamt_cache(struct kvm *kvm,
+ struct kvm_vcpu *vcpu)
{
+ if (KVM_BUG_ON(vcpu && vcpu->kvm != kvm, kvm))
+ return NULL;
+
+ if (vcpu)
+ return &to_tdx(vcpu)->pamt_cache;
+
+ lockdep_assert_held(&kvm->arch.tdp_mmu_external_cache_lock);
+ return &to_kvm_tdx(kvm)->pamt_cache;
+}
+
+static int tdx_topup_external_pamt_cache(struct kvm *kvm,
+ struct kvm_vcpu *vcpu, int min)
+{
+ struct tdx_pamt_cache *pamt_cache;
+
if (!tdx_supports_dynamic_pamt(tdx_sysinfo))
return 0;
- if (WARN_ON_ONCE(!vcpu))
+ pamt_cache = tdx_get_pamt_cache(kvm, vcpu);
+ if (!pamt_cache)
return -EIO;
- return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min);
+ return tdx_topup_pamt_cache(pamt_cache, min);
}
static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level level,
@@ -1792,8 +1813,8 @@ static struct page *tdx_spte_to_external_spt(struct kvm *kvm, gfn_t gfn,
static int tdx_sept_split_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level)
{
- struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ struct tdx_pamt_cache *pamt_cache;
gpa_t gpa = gfn_to_gpa(gfn);
u64 err, entry, level_state;
struct page *external_spt;
@@ -1804,7 +1825,8 @@ static int tdx_sept_split_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
if (!external_spt)
return -EIO;
- if (KVM_BUG_ON(!vcpu || vcpu->kvm != kvm, kvm))
+ pamt_cache = tdx_get_pamt_cache(kvm, kvm_get_running_vcpu());
+ if (!pamt_cache)
return -EIO;
err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
@@ -1816,7 +1838,7 @@ static int tdx_sept_split_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
err = tdh_do_no_vcpus(tdh_mem_page_demote, kvm, &kvm_tdx->td, gpa,
level, spte_to_pfn(old_spte), external_spt,
- &to_tdx(vcpu)->pamt_cache, &entry, &level_state);
+ pamt_cache, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_DEMOTE, entry, level_state, kvm))
return -EIO;
diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h
index f444fc84d93b..57d7e70ffe7d 100644
--- a/arch/x86/kvm/vmx/tdx.h
+++ b/arch/x86/kvm/vmx/tdx.h
@@ -48,6 +48,8 @@ struct kvm_tdx {
* Set/unset is protected with kvm->mmu_lock.
*/
bool wait_for_sept_zap;
+
+ struct tdx_pamt_cache pamt_cache;
};
/* TDX module vCPU states */
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 43/45] *** DO NOT MERGE *** KVM: guest_memfd: Add pre-zap arch hook for shared<=>private conversion
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Add a gmem "pre-zap" hook to allow arch code to take action before a
shared<=>private conversion, and just as importantly, to let arch code
reject/fail a conversion, e.g. if the conversion requires new page tables
and KVM hits in OOM situation.
The new hook will be used by TDX to split hugepages as necessary to avoid
overzapping PTEs, which for all intents and purposes corrupts guest data
for TDX VMs (memory is wiped when private PTEs are removed).
TODO: Wire this up the convert path, not the PUNCH_HOLE path, once in-place
conversion is supported.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/Kconfig | 1 +
arch/x86/kvm/mmu/tdp_mmu.c | 8 ++++++
include/linux/kvm_host.h | 5 ++++
virt/kvm/Kconfig | 4 +++
virt/kvm/guest_memfd.c | 50 ++++++++++++++++++++++++++++++++++++--
5 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index d916bd766c94..5f8d8daf4289 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -138,6 +138,7 @@ config KVM_INTEL_TDX
depends on INTEL_TDX_HOST
select KVM_GENERIC_MEMORY_ATTRIBUTES
select HAVE_KVM_ARCH_GMEM_POPULATE
+ select HAVE_KVM_ARCH_GMEM_CONVERT
help
Provides support for launching Intel Trust Domain Extensions (TDX)
confidential VMs on Intel processors.
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 0cdc6782e508..c46ebdacdb50 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1630,6 +1630,14 @@ int kvm_tdp_mmu_split_huge_pages(struct kvm_vcpu *vcpu, gfn_t start, gfn_t end,
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_split_huge_pages);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t start, gfn_t end,
+ bool to_private)
+{
+ return 0;
+}
+#endif /* CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT */
+
static bool tdp_mmu_need_write_protect(struct kvm *kvm, struct kvm_mmu_page *sp)
{
/*
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 782f4d670793..c0bafff274b6 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2588,6 +2588,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t gfn, void __user *src, long npages
kvm_gmem_populate_cb post_populate, void *opaque);
#endif
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t start, gfn_t end,
+ bool to_private);
+#endif
+
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
#endif
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 267c7369c765..05d69eaa50ae 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -125,3 +125,7 @@ config HAVE_KVM_ARCH_GMEM_INVALIDATE
config HAVE_KVM_ARCH_GMEM_POPULATE
bool
depends on KVM_GUEST_MEMFD
+
+config HAVE_KVM_ARCH_GMEM_CONVERT
+ bool
+ depends on KVM_GUEST_MEMFD
\ No newline at end of file
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 51dbb309188f..b01f333a5e95 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -164,6 +164,46 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
return folio;
}
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+static int __kvm_gmem_convert(struct gmem_file *f, pgoff_t start, pgoff_t end,
+ bool to_private)
+{
+ struct kvm_memory_slot *slot;
+ unsigned long index;
+ int r;
+
+ xa_for_each_range(&f->bindings, index, slot, start, end - 1) {
+ r = kvm_arch_gmem_convert(f->kvm,
+ kvm_gmem_get_start_gfn(slot, start),
+ kvm_gmem_get_end_gfn(slot, end),
+ to_private);
+ if (r)
+ return r;
+ }
+ return 0;
+}
+
+static int kvm_gmem_convert(struct inode *inode, pgoff_t start, pgoff_t end,
+ bool to_private)
+{
+ struct gmem_file *f;
+ int r;
+
+ kvm_gmem_for_each_file(f, inode->i_mapping) {
+ r = __kvm_gmem_convert(f, start, end, to_private);
+ if (r)
+ return r;
+ }
+ return 0;
+}
+#else
+static int kvm_gmem_convert(struct inode *inode, pgoff_t start, pgoff_t end,
+ bool to_private)
+{
+ return 0;
+}
+#endif
+
static enum kvm_gfn_range_filter kvm_gmem_get_invalidate_filter(struct inode *inode)
{
if (GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)
@@ -244,6 +284,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
{
pgoff_t start = offset >> PAGE_SHIFT;
pgoff_t end = (offset + len) >> PAGE_SHIFT;
+ int r;
/*
* Bindings must be stable across invalidation to ensure the start+end
@@ -253,13 +294,18 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
kvm_gmem_invalidate_begin(inode, start, end);
- truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+ /*
+ * For demonstration purposes, pretend this is a private=>shared conversion.
+ */
+ r = kvm_gmem_convert(inode, start, end, false);
+ if (!r)
+ truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
kvm_gmem_invalidate_end(inode, start, end);
filemap_invalidate_unlock(inode->i_mapping);
- return 0;
+ return r;
}
static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 42/45] KVM: guest_memfd: Add helpers to get start/end gfns give gmem+slot+pgoff
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Add helpers for getting a gfn given a gmem slot+pgoff, and for getting a
gfn given a starting or ending pgoff, i.e. an offset that may be beyond
the range of the memslot binding. Providing helpers will avoid duplicate
boilerplate code "if" future code also needs to iterate over gfn ranges.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
virt/kvm/guest_memfd.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 923c51a3a525..51dbb309188f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -59,6 +59,21 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
return gfn - slot->base_gfn + slot->gmem.pgoff;
}
+static gfn_t kvm_gmem_get_gfn(struct kvm_memory_slot *slot, pgoff_t pgoff)
+{
+ return slot->base_gfn + pgoff - slot->gmem.pgoff;
+}
+
+static gfn_t kvm_gmem_get_start_gfn(struct kvm_memory_slot *slot, pgoff_t start)
+{
+ return kvm_gmem_get_gfn(slot, max(slot->gmem.pgoff, start));
+}
+
+static gfn_t kvm_gmem_get_end_gfn(struct kvm_memory_slot *slot, pgoff_t end)
+{
+ return kvm_gmem_get_gfn(slot, min(slot->gmem.pgoff + slot->npages, end));
+}
+
static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
pgoff_t index, struct folio *folio)
{
@@ -167,11 +182,9 @@ static void __kvm_gmem_invalidate_begin(struct gmem_file *f, pgoff_t start,
unsigned long index;
xa_for_each_range(&f->bindings, index, slot, start, end - 1) {
- pgoff_t pgoff = slot->gmem.pgoff;
-
struct kvm_gfn_range gfn_range = {
- .start = slot->base_gfn + max(pgoff, start) - pgoff,
- .end = slot->base_gfn + min(pgoff + slot->npages, end) - pgoff,
+ .start = kvm_gmem_get_start_gfn(slot, start),
+ .end = kvm_gmem_get_end_gfn(slot, end),
.slot = slot,
.may_block = true,
.attr_filter = attr_filter,
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 41/45] KVM: TDX: Honor the guest's accept level contained in an EPT violation
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
From: Yan Zhao <yan.y.zhao@intel.com>
TDX requires guests to accept S-EPT mappings created by the host KVM. Due
to the current implementation of the TDX module, if a guest accepts a GFN
at a lower level after KVM maps it at a higher level, the TDX module will
synthesize an EPT Violation VM-Exit to KVM instead of returning a size
mismatch error to the guest. If KVM fails to perform page splitting in the
EPT Violation handler, the guest's ACCEPT operation will be triggered
again upon re-entering the guest, causing a repeated EPT Violation VM-Exit.
To ensure forward progress, honor the guest's accept level if an EPT
Violation VMExit contains guest accept level (the TDX-Module provides the
level when synthesizing a VM-Exit in response to a failed guest ACCEPT).
(1) Set the guest inhibit bit in the lpage info to prevent KVM's MMU
from mapping at a higher level than the guest's accept level.
(2) Split any existing mapping higher than the guest's accept level.
For now, take mmu_lock for write across the entire operation to keep things
simple. This can/will be revisited when the TDX-Module adds support for
NON-BLOCKING-RESIZE, at which point KVM can split the hugepage without
needing to handle UNBLOCK failure if the DEMOTE fails.
To avoid unnecessarily contending mmu_lock, check if the inhibit flag is
already set before acquiring mmu_lock, e.g. so that a vCPUs doing ACCEPT
on a region of memory aren't completely serialized. Note, this relies on
(a) setting the inhibit after performing the split, and (b) never clearing
the flag, e.g. to avoid false positives and potentially triggering the
zero-step mitigation.
Note: EPT Violation VM-Exits without the guest's accept level are *never*
caused by the guest's ACCEPT operation, but are instead occur if the guest
accesses of memory before said memory is accepted. Since KVM can't obtain
the guest accept level info from such EPT Violations (the ACCEPT operation
hasn't occurred yet), KVM may still map at a higher level than the later
guest's ACCEPT level.
So, the typical guest/KVM interaction flow is:
- If guest accesses private memory without first accepting it,
(like non-Linux guests):
1. Guest accesses a private memory.
2. KVM finds it can map the GFN at 2MB. So, AUG at 2MB.
3. Guest accepts the GFN at 4KB.
4. KVM receives an EPT violation with eeq_type of ACCEPT + 4KB level.
5. KVM splits the 2MB mapping.
6. Guest accepts successfully and accesses the page.
- If guest first accepts private memory before accessing it,
(like Linux guests):
1. Guest accepts a private memory at 4KB.
2. KVM receives an EPT violation with eeq_type of ACCEPT + 4KB level.
3. KVM AUG at 4KB.
4. Guest accepts successfully and accesses the page.
Link: https://lore.kernel.org/all/a6ffe23fb97e64109f512fa43e9f6405236ed40a.camel@intel.com
Suggested-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu/tdp_mmu.c | 11 ++++++
arch/x86/kvm/mmu/tdp_mmu.h | 2 +
arch/x86/kvm/vmx/tdx.c | 76 +++++++++++++++++++++++++++++++++++++
arch/x86/kvm/vmx/tdx_arch.h | 3 ++
4 files changed, 92 insertions(+)
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index e32034bfca5a..0cdc6782e508 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1619,6 +1619,17 @@ void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm,
}
}
+/* Split huge pages for the current root. */
+int kvm_tdp_mmu_split_huge_pages(struct kvm_vcpu *vcpu, gfn_t start, gfn_t end,
+ int target_level)
+{
+ struct kvm_mmu_page *root = root_to_sp(vcpu->arch.mmu->root.hpa);
+
+ return tdp_mmu_split_huge_pages_root(vcpu->kvm, root, start, end,
+ target_level, false);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_split_huge_pages);
+
static bool tdp_mmu_need_write_protect(struct kvm *kvm, struct kvm_mmu_page *sp)
{
/*
diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h
index bd62977c9199..cdb0b4ecaa37 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.h
+++ b/arch/x86/kvm/mmu/tdp_mmu.h
@@ -97,6 +97,8 @@ void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm,
const struct kvm_memory_slot *slot,
gfn_t start, gfn_t end,
int target_level, bool shared);
+int kvm_tdp_mmu_split_huge_pages(struct kvm_vcpu *vcpu, gfn_t start, gfn_t end,
+ int target_level);
static inline void kvm_tdp_mmu_walk_lockless_begin(void)
{
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index af63364c8713..098954f5e07c 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -13,6 +13,7 @@
#include "tdx.h"
#include "vmx.h"
#include "mmu/spte.h"
+#include "mmu/tdp_mmu.h"
#include "common.h"
#include "posted_intr.h"
#include "irq.h"
@@ -1958,6 +1959,77 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
}
+static bool tdx_is_mismatched_accepted(struct kvm_vcpu *vcpu)
+{
+ return (to_tdx(vcpu)->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK) ==
+ TDX_EXT_EXIT_QUAL_TYPE_ACCEPT;
+}
+
+static int tdx_get_ept_violation_level(struct kvm_vcpu *vcpu)
+{
+ u64 ext_exit_qual = to_tdx(vcpu)->ext_exit_qualification;
+
+ return (((ext_exit_qual & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+ TDX_EXT_EXIT_QUAL_INFO_SHIFT) & GENMASK(2, 0)) + 1;
+}
+
+/*
+ * An EPT violation can be either due to the guest's ACCEPT operation or
+ * due to the guest's access of memory before the guest accepts the
+ * memory.
+ *
+ * Type TDX_EXT_EXIT_QUAL_TYPE_ACCEPT in the extended exit qualification
+ * identifies the former case, which must also contain a valid guest
+ * accept level.
+ *
+ * For the former case, honor guest's accept level by setting guest inhibit bit
+ * on levels above the guest accept level and split the existing mapping for the
+ * faulting GFN if it's with a higher level than the guest accept level.
+ *
+ * Do nothing if the EPT violation is due to the latter case. KVM will map the
+ * GFN without considering the guest's accept level (unless the guest inhibit
+ * bit is already set).
+ */
+static int tdx_handle_mismatched_accept(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+ struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+ struct kvm *kvm = vcpu->kvm;
+ gfn_t start, end;
+ int level, r;
+
+ if (!slot || !tdx_is_mismatched_accepted(vcpu))
+ return 0;
+
+ if (WARN_ON_ONCE(!VALID_PAGE(vcpu->arch.mmu->root.hpa)))
+ return 0;
+
+ level = tdx_get_ept_violation_level(vcpu);
+ if (level > PG_LEVEL_2M)
+ return 0;
+
+ if (hugepage_test_guest_inhibit(slot, gfn, level + 1))
+ return 0;
+
+ guard(write_lock)(&kvm->mmu_lock);
+
+ start = gfn_round_for_level(gfn, level);
+ end = start + KVM_PAGES_PER_HPAGE(level);
+
+ r = kvm_tdp_mmu_split_huge_pages(vcpu, start, end, level);
+ if (r)
+ return r;
+
+ /*
+ * No TLB flush is required, as the "BLOCK + TRACK + kick off vCPUs"
+ * sequence required by the TDX-Module includes a TLB flush.
+ */
+ hugepage_set_guest_inhibit(slot, gfn, level + 1);
+ if (level == PG_LEVEL_4K)
+ hugepage_set_guest_inhibit(slot, gfn, level + 2);
+
+ return 0;
+}
+
static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
{
unsigned long exit_qual;
@@ -1983,6 +2055,10 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
*/
exit_qual = EPT_VIOLATION_ACC_WRITE;
+ ret = tdx_handle_mismatched_accept(vcpu, gpa_to_gfn(gpa));
+ if (ret)
+ return ret;
+
/* Only private GPA triggers zero-step mitigation */
local_retry = true;
} else {
diff --git a/arch/x86/kvm/vmx/tdx_arch.h b/arch/x86/kvm/vmx/tdx_arch.h
index a30e880849e3..af006a73ee05 100644
--- a/arch/x86/kvm/vmx/tdx_arch.h
+++ b/arch/x86/kvm/vmx/tdx_arch.h
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
#define TDX_TD_ATTR_PERFMON BIT_ULL(63)
#define TDX_EXT_EXIT_QUAL_TYPE_MASK GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT 1
#define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION 6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT 32
/*
* TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
*/
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 40/45] KVM: x86: Introduce hugepage_set_guest_inhibit()
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
From: Yan Zhao <yan.y.zhao@intel.com>
TDX requires guests to accept S-EPT mappings created by the host KVM. Due
to the current implementation of the TDX module, if a guest accepts a GFN
at a lower level after KVM maps it at a higher level, the TDX module will
emulate an EPT violation VMExit to KVM instead of returning a size mismatch
error to the guest. If KVM fails to perform page splitting in the VMExit
handler, the guest's accept operation will be triggered again upon
re-entering the guest, causing a repeated EPT violation VMExit.
To facilitate passing the guest's accept level information to the KVM MMU
core and to prevent the repeated mapping of a GFN at different levels due
to different accept levels specified by different vCPUs, introduce the
interface hugepage_set_guest_inhibit(). This interface specifies across
vCPUs that mapping at a certain level is inhibited from the guest.
Intentionally don't provide an API to clear KVM_LPAGE_GUEST_INHIBIT_FLAG
for the time being, as detecting that it's ok to (re)install a hugepage is
tricky (and costly if KVM wants to be 100% accurate), and KVM doesn't
currently support hugepage promotion (only direct installation of
hugepages) for S-EPT.
As a result, the only scenario where clearing the flag would likely allow
KVM to install a hugepage is when an entire 2MiB / 1GiB range is converted
to shared or private. But if the guest is accepting at 4KiB granulairty,
odds are good the guest is using the memory for something "special" and
will never convert the entire range to shared (and/or back to private).
Punt that optimization to the future, if it's ever needed.
Link: https://lore.kernel.org/all/a6ffe23fb97e64109f512fa43e9f6405236ed40a.camel@intel.com [1]
Suggested-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
[sean: explain *why* the flag is never cleared]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu.h | 4 ++++
arch/x86/kvm/mmu/mmu.c | 21 ++++++++++++++++++---
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 830f46145692..fa6a8daf4b05 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -322,4 +322,8 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
{
return gfn & kvm_gfn_direct_bits(kvm);
}
+
+void hugepage_set_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level);
+bool hugepage_test_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level);
+
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 45650f70eeab..c2765bfc8492 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -718,12 +718,14 @@ static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
}
/*
- * The most significant bit in disallow_lpage tracks whether or not memory
- * attributes are mixed, i.e. not identical for all gfns at the current level.
+ * The most 2 significant bits in disallow_lpage tracks whether or not memory
+ * attributes are mixed, i.e. not identical for all gfns at the current level,
+ * or whether or not guest inhibits the current level of hugepage at the gfn.
* The lower order bits are used to refcount other cases where a hugepage is
* disallowed, e.g. if KVM has shadow a page table at the gfn.
*/
#define KVM_LPAGE_MIXED_FLAG BIT(31)
+#define KVM_LPAGE_GUEST_INHIBIT_FLAG BIT(30)
static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
gfn_t gfn, int count)
@@ -736,7 +738,8 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
old = linfo->disallow_lpage;
linfo->disallow_lpage += count;
- WARN_ON_ONCE((old ^ linfo->disallow_lpage) & KVM_LPAGE_MIXED_FLAG);
+ WARN_ON_ONCE((old ^ linfo->disallow_lpage) &
+ (KVM_LPAGE_MIXED_FLAG | KVM_LPAGE_GUEST_INHIBIT_FLAG));
}
}
@@ -1648,6 +1651,18 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
start, end - 1, can_yield, true, flush);
}
+bool hugepage_test_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level)
+{
+ return lpage_info_slot(gfn, slot, level)->disallow_lpage & KVM_LPAGE_GUEST_INHIBIT_FLAG;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(hugepage_test_guest_inhibit);
+
+void hugepage_set_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level)
+{
+ lpage_info_slot(gfn, slot, level)->disallow_lpage |= KVM_LPAGE_GUEST_INHIBIT_FLAG;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(hugepage_set_guest_inhibit);
+
bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
{
bool flush = false;
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 39/45] KVM: TDX: Add core support for splitting/demoting 2MiB S-EPT to 4KiB
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
From: Yan Zhao <yan.y.zhao@intel.com>
Add support for splitting, a.k.a. demoting, a 2MiB S-EPT hugepage to its
512 constituent 4KiB pages. As per the TDX-Module rules, first invoke
MEM.RANGE.BLOCK to put the huge S-EPTE entry into a splittable state, then
do MEM.TRACK and kick all vCPUs outside of guest mode to flush TLBs, and
finally do MEM.PAGE.DEMOTE to demote/split the huge S-EPT entry.
Assert the mmu_lock is held for write, as the BLOCK => TRACK => DEMOTE
sequence needs to be "atomic" to guarantee success (and because mmu_lock
must be held for write to use tdh_do_no_vcpus()).
Note, even with kvm->mmu_lock held for write, tdh_mem_page_demote() may
contend with tdh_vp_enter() and potentially with the guest's S-EPT entry
operations. Therefore, wrap the call with tdh_do_no_vcpus() to kick other
vCPUs out of the guest and prevent tdh_vp_enter() to ensure success.
Signed-off-by: Xiaoyao Li <xiaoyao.li@intel.com>
Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
[sean: wire up via tdx_sept_link_private_spt(), massage changelog]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/vmx/tdx.c | 51 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 48 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index e90610540a0b..af63364c8713 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1776,6 +1776,52 @@ static struct page *tdx_spte_to_external_spt(struct kvm *kvm, gfn_t gfn,
return virt_to_page(sp->external_spt);
}
+/*
+ * Split a huge mapping into the target level. Currently only supports 2MiB
+ * mappings (KVM doesn't yet support 1GiB mappings for TDX guests).
+ *
+ * Invoke "BLOCK + TRACK + kick off vCPUs (inside tdx_track())" since DEMOTE
+ * now does not support yet the NON-BLOCKING-RESIZE feature. No UNBLOCK is
+ * needed after a successful DEMOTE.
+ *
+ * Under write mmu_lock, kick off all vCPUs (inside tdh_do_no_vcpus()) to ensure
+ * DEMOTE will succeed on the second invocation if the first invocation returns
+ * BUSY.
+ */
+static int tdx_sept_split_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
+ u64 new_spte, enum pg_level level)
+{
+ struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ gpa_t gpa = gfn_to_gpa(gfn);
+ u64 err, entry, level_state;
+ struct page *external_spt;
+
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ external_spt = tdx_spte_to_external_spt(kvm, gfn, new_spte, level);
+ if (!external_spt)
+ return -EIO;
+
+ if (KVM_BUG_ON(!vcpu || vcpu->kvm != kvm, kvm))
+ return -EIO;
+
+ err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
+ level, &entry, &level_state);
+ if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
+ return -EIO;
+
+ tdx_track(kvm);
+
+ err = tdh_do_no_vcpus(tdh_mem_page_demote, kvm, &kvm_tdx->td, gpa,
+ level, spte_to_pfn(old_spte), external_spt,
+ &to_tdx(vcpu)->pamt_cache, &entry, &level_state);
+ if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_DEMOTE, entry, level_state, kvm))
+ return -EIO;
+
+ return 0;
+}
+
static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, u64 new_spte,
enum pg_level level)
{
@@ -1853,9 +1899,8 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level)
{
- /* TODO: Support replacing huge SPTE with non-leaf SPTE. (a.k.a. demotion). */
- if (KVM_BUG_ON(is_shadow_present_pte(old_spte) && is_shadow_present_pte(new_spte), kvm))
- return -EIO;
+ if (is_shadow_present_pte(old_spte) && is_shadow_present_pte(new_spte))
+ return tdx_sept_split_private_spte(kvm, gfn, old_spte, new_spte, level);
else if (is_shadow_present_pte(old_spte))
return tdx_sept_remove_private_spte(kvm, gfn, old_spte, level);
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 38/45] KVM: x86/mmu: Add Dynamic PAMT support in TDP MMU for vCPU-induced page split
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Extend the TDP MMU to support vCPU-induced hugepage splits in mirror roots
when Dynamic PAMT is enabled. I.e. top-up the PAMT cache when allocating
a new child page table, so that if the split is successful, there will be
a PAMT paging waiting to associated with the new less/non-huge mapping.
Note, the allocation is for the guest memory, not the S-EPT page, as PAMT
pages are accounted up front by .alloc_external_sp().
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu/tdp_mmu.c | 25 ++++++++++++++++---------
arch/x86/kvm/vmx/tdx.c | 3 +++
2 files changed, 19 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 4f5b80f0ca03..e32034bfca5a 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1456,21 +1456,28 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct tdp_iter *iter)
return NULL;
sp->spt = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
- if (!sp->spt) {
- kmem_cache_free(mmu_page_header_cache, sp);
- return NULL;
- }
+ if (!sp->spt)
+ goto err_spt;
if (is_mirror_sptep(iter->sptep)) {
sp->external_spt = (void *)kvm_x86_call(alloc_external_sp)(GFP_KERNEL_ACCOUNT);
- if (!sp->external_spt) {
- free_page((unsigned long)sp->spt);
- kmem_cache_free(mmu_page_header_cache, sp);
- return NULL;
- }
+ if (!sp->external_spt)
+ goto err_external_spt;
+
+ if (kvm_x86_call(topup_external_cache)(kvm_get_running_vcpu(), 1))
+ goto err_external_split;
}
return sp;
+
+err_external_split:
+ kvm_x86_call(free_external_sp)((unsigned long)sp->external_spt);
+err_external_spt:
+ free_page((unsigned long)sp->spt);
+err_spt:
+ kmem_cache_free(mmu_page_header_cache, sp);
+ return NULL;
+
}
/* Note, the caller is responsible for initializing @sp. */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 59b7ba36d3d9..e90610540a0b 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1625,6 +1625,9 @@ static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min)
if (!tdx_supports_dynamic_pamt(tdx_sysinfo))
return 0;
+ if (WARN_ON_ONCE(!vcpu))
+ return -EIO;
+
return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min);
}
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 37/45] KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table splitting
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
From: Isaku Yamahata <isaku.yamahata@intel.com>
Enhance tdp_mmu_alloc_sp_for_split() to allocate a page table page for the
external page table for splitting the mirror page table.
Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
Co-developed-by: Yan Zhao <yan.y.zhao@intel.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
[sean: use kvm_x86_ops.alloc_external_sp()]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu/tdp_mmu.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 3b0da898824a..4f5b80f0ca03 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1447,7 +1447,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm,
return spte_set;
}
-static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct tdp_iter *iter)
{
struct kvm_mmu_page *sp;
@@ -1461,6 +1461,15 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
return NULL;
}
+ if (is_mirror_sptep(iter->sptep)) {
+ sp->external_spt = (void *)kvm_x86_call(alloc_external_sp)(GFP_KERNEL_ACCOUNT);
+ if (!sp->external_spt) {
+ free_page((unsigned long)sp->spt);
+ kmem_cache_free(mmu_page_header_cache, sp);
+ return NULL;
+ }
+ }
+
return sp;
}
@@ -1540,7 +1549,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
else
write_unlock(&kvm->mmu_lock);
- sp = tdp_mmu_alloc_sp_for_split();
+ sp = tdp_mmu_alloc_sp_for_split(&iter);
if (shared)
read_lock(&kvm->mmu_lock);
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 36/45] KVM: TDX: Move S-EPT page demotion TODO to tdx_sept_set_private_spte()
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Now that handle_changed_spte() can handles all mirror SPTE updates, move
the TDP MMU's assertion that it doesn't replace a shadow-present mirror
SPTE with another shadow-present SPTE into TDX, in the form of a TODO
that calls out that KVM needs to add support for splitting/demoting
hugepage.
Drop the "!is_leaf" condition so that an unexpected/unsupported update to
a shadow-present S-EPT triggers a KVM_BUG_ON(), versus being silently
ignored (well, silent until it causes explosions in the future).
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu/tdp_mmu.c | 9 +--------
arch/x86/kvm/vmx/tdx.c | 5 ++++-
2 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index d49aecba18d8..3b0da898824a 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -572,7 +572,7 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
if (was_present && !was_leaf &&
(is_leaf || !is_present || WARN_ON_ONCE(pfn_changed)))
handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared);
- else if (was_leaf && is_mirror_sptep(sptep) && !is_leaf)
+ else if (was_leaf && is_mirror_sptep(sptep))
KVM_BUG_ON(kvm_x86_call(set_external_spte)(kvm, gfn, old_spte,
new_spte, level), kvm);
}
@@ -704,13 +704,6 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
handle_changed_spte(kvm, as_id, sptep, gfn, old_spte, new_spte, level, false);
- /*
- * Users that do non-atomic setting of PTEs don't operate on mirror
- * roots. Bug the VM as this path doesn't propagate such writes to the
- * external page tables.
- */
- KVM_BUG_ON(is_mirror_sptep(sptep) && is_shadow_present_pte(new_spte), kvm);
-
return old_spte;
}
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index e6ac4aca8114..59b7ba36d3d9 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1850,7 +1850,10 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level)
{
- if (is_shadow_present_pte(old_spte))
+ /* TODO: Support replacing huge SPTE with non-leaf SPTE. (a.k.a. demotion). */
+ if (KVM_BUG_ON(is_shadow_present_pte(old_spte) && is_shadow_present_pte(new_spte), kvm))
+ return -EIO;
+ else if (is_shadow_present_pte(old_spte))
return tdx_sept_remove_private_spte(kvm, gfn, old_spte, level);
if (KVM_BUG_ON(!is_shadow_present_pte(new_spte), kvm))
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 35/45] KVM: TDX: Add helper to handle mapping leaf SPTE into S-EPT
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Add a helper, tdx_sept_map_leaf_spte(), to wrap and isolate PAGE.ADD and
PAGE.AUG operations, and thus complete tdx_sept_set_private_spte()'s
transition into a "dispatch" routine for setting/writing S-EPT entries.
Opportunistically tweak the prototypes for tdx_sept_remove_private_spte()
and tdx_sept_link_private_spt() to align with tdx_sept_set_private_spte()
and tdx_sept_map_leaf_spte().
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/vmx/tdx.c | 97 ++++++++++++++++++++++--------------------
1 file changed, 51 insertions(+), 46 deletions(-)
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 9f7789c5f0a7..e6ac4aca8114 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1670,6 +1670,50 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
return 0;
}
+static int tdx_sept_map_leaf_spte(struct kvm *kvm, gfn_t gfn, u64 new_spte,
+ enum pg_level level)
+{
+ struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ kvm_pfn_t pfn = spte_to_pfn(new_spte);
+ int ret;
+
+ /* TODO: handle large pages. */
+ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
+ return -EIO;
+
+ if (KVM_BUG_ON(!vcpu, kvm))
+ return -EINVAL;
+
+ WARN_ON_ONCE((new_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK);
+
+ ret = tdx_pamt_get(pfn, level, &to_tdx(vcpu)->pamt_cache);
+ if (ret)
+ return ret;
+
+ /*
+ * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory()
+ * before kvm_tdx->state. Userspace must not be allowed to pre-fault
+ * arbitrary memory until the initial memory image is finalized. Pairs
+ * with the smp_wmb() in tdx_td_finalize().
+ */
+ smp_rmb();
+
+ /*
+ * If the TD isn't finalized/runnable, then userspace is initializing
+ * the VM image via KVM_TDX_INIT_MEM_REGION; ADD the page to the TD.
+ */
+ if (likely(kvm_tdx->state == TD_STATE_RUNNABLE))
+ ret = tdx_mem_page_aug(kvm, gfn, level, pfn);
+ else
+ ret = tdx_mem_page_add(kvm, gfn, level, pfn);
+
+ if (ret)
+ tdx_pamt_put(pfn, level);
+
+ return ret;
+}
+
/*
* Ensure shared and private EPTs to be flushed on all vCPUs.
* tdh_mem_track() is the only caller that increases TD epoch. An increase in
@@ -1729,14 +1773,14 @@ static struct page *tdx_spte_to_external_spt(struct kvm *kvm, gfn_t gfn,
return virt_to_page(sp->external_spt);
}
-static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
- enum pg_level level, u64 mirror_spte)
+static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, u64 new_spte,
+ enum pg_level level)
{
gpa_t gpa = gfn_to_gpa(gfn);
u64 err, entry, level_state;
struct page *external_spt;
- external_spt = tdx_spte_to_external_spt(kvm, gfn, mirror_spte, level);
+ external_spt = tdx_spte_to_external_spt(kvm, gfn, new_spte, level);
if (!external_spt)
return -EIO;
@@ -1752,7 +1796,7 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
}
static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
- enum pg_level level, u64 old_spte)
+ u64 old_spte, enum pg_level level)
{
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
kvm_pfn_t pfn = spte_to_pfn(old_spte);
@@ -1806,55 +1850,16 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level)
{
- struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
- struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
- kvm_pfn_t pfn = spte_to_pfn(new_spte);
- struct vcpu_tdx *tdx = to_tdx(vcpu);
- int ret;
-
if (is_shadow_present_pte(old_spte))
- return tdx_sept_remove_private_spte(kvm, gfn, level, old_spte);
-
- if (KVM_BUG_ON(!vcpu, kvm))
- return -EINVAL;
+ return tdx_sept_remove_private_spte(kvm, gfn, old_spte, level);
if (KVM_BUG_ON(!is_shadow_present_pte(new_spte), kvm))
return -EIO;
if (!is_last_spte(new_spte, level))
- return tdx_sept_link_private_spt(kvm, gfn, level, new_spte);
+ return tdx_sept_link_private_spt(kvm, gfn, new_spte, level);
- /* TODO: handle large pages. */
- if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
- return -EIO;
-
- WARN_ON_ONCE((new_spte & VMX_EPT_RWX_MASK) != VMX_EPT_RWX_MASK);
-
- ret = tdx_pamt_get(pfn, level, &tdx->pamt_cache);
- if (ret)
- return ret;
-
- /*
- * Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory()
- * before kvm_tdx->state. Userspace must not be allowed to pre-fault
- * arbitrary memory until the initial memory image is finalized. Pairs
- * with the smp_wmb() in tdx_td_finalize().
- */
- smp_rmb();
-
- /*
- * If the TD isn't finalized/runnable, then userspace is initializing
- * the VM image via KVM_TDX_INIT_MEM_REGION; ADD the page to the TD.
- */
- if (likely(kvm_tdx->state == TD_STATE_RUNNABLE))
- ret = tdx_mem_page_aug(kvm, gfn, level, pfn);
- else
- ret = tdx_mem_page_add(kvm, gfn, level, pfn);
-
- if (ret)
- tdx_pamt_put(pfn, level);
-
- return ret;
+ return tdx_sept_map_leaf_spte(kvm, gfn, new_spte, level);
}
static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 34/45] KVM: TDX: Handle removal of leaf SPTEs in .set_private_spte()
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Drop kvm_x86_ops.remove_external_spte(), and instead handling the removal
of leaf SPTEs in the S-EPT (a.k.a. external root) in .set_private_spte().
This will allow extending tdx_sept_set_private_spte() to support splitting
a huge S-EPT entry without needing yet another kvm_x86_ops hook.
Bug the VM if the callback fails, as redundant KVM_BUG_ON() calls are
benign (the WARN will fire if and only if the VM isn't already bugged) and
handle_changed_spte() is most definitely not prepared to handle failure.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/include/asm/kvm-x86-ops.h | 1 -
arch/x86/include/asm/kvm_host.h | 2 --
arch/x86/kvm/mmu/tdp_mmu.c | 20 +++++++++++---------
arch/x86/kvm/vmx/tdx.c | 21 ++++++++++++---------
4 files changed, 23 insertions(+), 21 deletions(-)
diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 394dc29483a7..3ca56fe6b951 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -97,7 +97,6 @@ KVM_X86_OP(load_mmu_pgd)
KVM_X86_OP_OPTIONAL(alloc_external_sp)
KVM_X86_OP_OPTIONAL(free_external_sp)
KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
-KVM_X86_OP_OPTIONAL(remove_external_spte)
KVM_X86_OP_OPTIONAL(reclaim_external_sp)
KVM_X86_OP_OPTIONAL_RET0(topup_external_cache)
KVM_X86_OP(has_wbinvd_exit)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 67deec8e205e..385f1cf32d70 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1861,8 +1861,6 @@ struct kvm_x86_ops {
u64 new_spte, enum pg_level level);
void (*reclaim_external_sp)(struct kvm *kvm, gfn_t gfn,
struct kvm_mmu_page *sp);
- void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
- u64 mirror_spte);
int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min);
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 271dd6f875a6..d49aecba18d8 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -559,20 +559,22 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
* SPTE being converted to a hugepage (leaf) or being zapped. Shadow
* pages are kernel allocations and should never be migrated.
*
- * When removing leaf entries from a mirror, immediately propagate the
- * changes to the external page tables. Note, non-leaf mirror entries
- * are handled by handle_removed_pt(), as TDX requires that all leaf
- * entries are removed before the owning page table. Note #2, writes
- * to make mirror PTEs shadow-present are propagated to external page
- * tables by __tdp_mmu_set_spte_atomic(), as KVM needs to ensure the
- * external page table was successfully updated before marking the
- * mirror SPTE present.
+ * When modifying leaf entries in mirrored page tables, propagate the
+ * changes to the external SPTE. Bug the VM on failure, as callers
+ * aren't prepared to handle errors, e.g. due to lock contention in the
+ * TDX-Module. Note, changes to non-leaf mirror SPTEs are handled by
+ * handle_removed_pt() (the TDX-Module requires that child entries are
+ * removed before the parent SPTE), and changes to non-present mirror
+ * SPTEs are handled by __tdp_mmu_set_spte_atomic() (KVM needs to set
+ * the external SPTE while the mirror SPTE is frozen so that installing
+ * a new SPTE is effectively an atomic operation).
*/
if (was_present && !was_leaf &&
(is_leaf || !is_present || WARN_ON_ONCE(pfn_changed)))
handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared);
else if (was_leaf && is_mirror_sptep(sptep) && !is_leaf)
- kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_spte);
+ KVM_BUG_ON(kvm_x86_call(set_external_spte)(kvm, gfn, old_spte,
+ new_spte, level), kvm);
}
static inline int __must_check __tdp_mmu_set_spte_atomic(struct kvm *kvm,
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 0f3d27699a3d..9f7789c5f0a7 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1751,11 +1751,11 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
return 0;
}
-static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
- enum pg_level level, u64 mirror_spte)
+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
+ enum pg_level level, u64 old_spte)
{
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
- kvm_pfn_t pfn = spte_to_pfn(mirror_spte);
+ kvm_pfn_t pfn = spte_to_pfn(old_spte);
gpa_t gpa = gfn_to_gpa(gfn);
u64 err, entry, level_state;
@@ -1767,16 +1767,16 @@ static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
* there can't be anything populated in the private EPT.
*/
if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
- return;
+ return -EIO;
/* TODO: handle large pages. */
if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
- return;
+ return -EIO;
err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
level, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
- return;
+ return -EIO;
/*
* TDX requires TLB tracking before dropping private page. Do
@@ -1792,14 +1792,15 @@ static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa,
level, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
- return;
+ return -EIO;
err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, pfn, level);
if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
- return;
+ return -EIO;
__tdx_quirk_reset_page(pfn, level);
tdx_pamt_put(pfn, level);
+ return 0;
}
static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
@@ -1811,6 +1812,9 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
struct vcpu_tdx *tdx = to_tdx(vcpu);
int ret;
+ if (is_shadow_present_pte(old_spte))
+ return tdx_sept_remove_private_spte(kvm, gfn, level, old_spte);
+
if (KVM_BUG_ON(!vcpu, kvm))
return -EINVAL;
@@ -3639,7 +3643,6 @@ void __init tdx_hardware_setup(void)
vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;
vt_x86_ops.reclaim_external_sp = tdx_sept_reclaim_private_sp;
- vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte;
/*
* FIXME: Wire up the PAMT hook iff DPAMT is supported, once VMXON is
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
* [RFC PATCH v5 33/45] KVM: TDX: Hoist tdx_sept_remove_private_spte() above set_private_spte()
From: Sean Christopherson @ 2026-01-29 1:15 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Sean Christopherson, Paolo Bonzini
Cc: linux-kernel, linux-coco, kvm, Kai Huang, Rick Edgecombe,
Yan Zhao, Vishal Annapurve, Ackerley Tng, Sagi Shahar, Binbin Wu,
Xiaoyao Li, Isaku Yamahata
In-Reply-To: <20260129011517.3545883-1-seanjc@google.com>
Move tdx_sept_remove_private_spte() (and its tdx_track() helper) above
tdx_sept_set_private_spte() in anticipation of routing all non-atomic
S-EPT writes (with the exception of reclaiming non-leaf pages) through
the "set" API.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/vmx/tdx.c | 194 ++++++++++++++++++++---------------------
1 file changed, 97 insertions(+), 97 deletions(-)
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index e451acdb0978..0f3d27699a3d 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1670,6 +1670,52 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
return 0;
}
+/*
+ * Ensure shared and private EPTs to be flushed on all vCPUs.
+ * tdh_mem_track() is the only caller that increases TD epoch. An increase in
+ * the TD epoch (e.g., to value "N + 1") is successful only if no vCPUs are
+ * running in guest mode with the value "N - 1".
+ *
+ * A successful execution of tdh_mem_track() ensures that vCPUs can only run in
+ * guest mode with TD epoch value "N" if no TD exit occurs after the TD epoch
+ * being increased to "N + 1".
+ *
+ * Kicking off all vCPUs after that further results in no vCPUs can run in guest
+ * mode with TD epoch value "N", which unblocks the next tdh_mem_track() (e.g.
+ * to increase TD epoch to "N + 2").
+ *
+ * TDX module will flush EPT on the next TD enter and make vCPUs to run in
+ * guest mode with TD epoch value "N + 1".
+ *
+ * kvm_make_all_cpus_request() guarantees all vCPUs are out of guest mode by
+ * waiting empty IPI handler ack_kick().
+ *
+ * No action is required to the vCPUs being kicked off since the kicking off
+ * occurs certainly after TD epoch increment and before the next
+ * tdh_mem_track().
+ */
+static void tdx_track(struct kvm *kvm)
+{
+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ u64 err;
+
+ /* If TD isn't finalized, it's before any vcpu running. */
+ if (unlikely(kvm_tdx->state != TD_STATE_RUNNABLE))
+ return;
+
+ /*
+ * The full sequence of TDH.MEM.TRACK and forcing vCPUs out of guest
+ * mode must be serialized, as TDH.MEM.TRACK will fail if the previous
+ * tracking epoch hasn't completed.
+ */
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ err = tdh_do_no_vcpus(tdh_mem_track, kvm, &kvm_tdx->td);
+ TDX_BUG_ON(err, TDH_MEM_TRACK, kvm);
+
+ kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);
+}
+
static struct page *tdx_spte_to_external_spt(struct kvm *kvm, gfn_t gfn,
u64 new_spte, enum pg_level level)
{
@@ -1705,6 +1751,57 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
return 0;
}
+static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
+ enum pg_level level, u64 mirror_spte)
+{
+ struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+ kvm_pfn_t pfn = spte_to_pfn(mirror_spte);
+ gpa_t gpa = gfn_to_gpa(gfn);
+ u64 err, entry, level_state;
+
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ /*
+ * HKID is released after all private pages have been removed, and set
+ * before any might be populated. Warn if zapping is attempted when
+ * there can't be anything populated in the private EPT.
+ */
+ if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
+ return;
+
+ /* TODO: handle large pages. */
+ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
+ return;
+
+ err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
+ level, &entry, &level_state);
+ if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
+ return;
+
+ /*
+ * TDX requires TLB tracking before dropping private page. Do
+ * it here, although it is also done later.
+ */
+ tdx_track(kvm);
+
+ /*
+ * When zapping private page, write lock is held. So no race condition
+ * with other vcpu sept operation.
+ * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs.
+ */
+ err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa,
+ level, &entry, &level_state);
+ if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
+ return;
+
+ err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, pfn, level);
+ if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
+ return;
+
+ __tdx_quirk_reset_page(pfn, level);
+ tdx_pamt_put(pfn, level);
+}
+
static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level)
{
@@ -1756,52 +1853,6 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
return ret;
}
-/*
- * Ensure shared and private EPTs to be flushed on all vCPUs.
- * tdh_mem_track() is the only caller that increases TD epoch. An increase in
- * the TD epoch (e.g., to value "N + 1") is successful only if no vCPUs are
- * running in guest mode with the value "N - 1".
- *
- * A successful execution of tdh_mem_track() ensures that vCPUs can only run in
- * guest mode with TD epoch value "N" if no TD exit occurs after the TD epoch
- * being increased to "N + 1".
- *
- * Kicking off all vCPUs after that further results in no vCPUs can run in guest
- * mode with TD epoch value "N", which unblocks the next tdh_mem_track() (e.g.
- * to increase TD epoch to "N + 2").
- *
- * TDX module will flush EPT on the next TD enter and make vCPUs to run in
- * guest mode with TD epoch value "N + 1".
- *
- * kvm_make_all_cpus_request() guarantees all vCPUs are out of guest mode by
- * waiting empty IPI handler ack_kick().
- *
- * No action is required to the vCPUs being kicked off since the kicking off
- * occurs certainly after TD epoch increment and before the next
- * tdh_mem_track().
- */
-static void tdx_track(struct kvm *kvm)
-{
- struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
- u64 err;
-
- /* If TD isn't finalized, it's before any vcpu running. */
- if (unlikely(kvm_tdx->state != TD_STATE_RUNNABLE))
- return;
-
- /*
- * The full sequence of TDH.MEM.TRACK and forcing vCPUs out of guest
- * mode must be serialized, as TDH.MEM.TRACK will fail if the previous
- * tracking epoch hasn't completed.
- */
- lockdep_assert_held_write(&kvm->mmu_lock);
-
- err = tdh_do_no_vcpus(tdh_mem_track, kvm, &kvm_tdx->td);
- TDX_BUG_ON(err, TDH_MEM_TRACK, kvm);
-
- kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);
-}
-
static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
struct kvm_mmu_page *sp)
{
@@ -1824,57 +1875,6 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
sp->external_spt = NULL;
}
-static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
- enum pg_level level, u64 mirror_spte)
-{
- struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
- kvm_pfn_t pfn = spte_to_pfn(mirror_spte);
- gpa_t gpa = gfn_to_gpa(gfn);
- u64 err, entry, level_state;
-
- lockdep_assert_held_write(&kvm->mmu_lock);
-
- /*
- * HKID is released after all private pages have been removed, and set
- * before any might be populated. Warn if zapping is attempted when
- * there can't be anything populated in the private EPT.
- */
- if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
- return;
-
- /* TODO: handle large pages. */
- if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
- return;
-
- err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
- level, &entry, &level_state);
- if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
- return;
-
- /*
- * TDX requires TLB tracking before dropping private page. Do
- * it here, although it is also done later.
- */
- tdx_track(kvm);
-
- /*
- * When zapping private page, write lock is held. So no race condition
- * with other vcpu sept operation.
- * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs.
- */
- err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa,
- level, &entry, &level_state);
- if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
- return;
-
- err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, pfn, level);
- if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
- return;
-
- __tdx_quirk_reset_page(pfn, level);
- tdx_pamt_put(pfn, level);
-}
-
void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
int trig_mode, int vector)
{
--
2.53.0.rc1.217.geba53bf80e-goog
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox