* Re: [RFC PATCH v5 20/45] KVM: x86/mmu: Allocate/free S-EPT pages using tdx_{alloc,free}_control_page()
From: Sean Christopherson @ 2026-02-09 23:20 UTC (permalink / raw)
To: Yan Zhao
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Paolo Bonzini, linux-kernel, linux-coco, kvm,
Kai Huang, Rick Edgecombe, Vishal Annapurve, Ackerley Tng,
Sagi Shahar, Binbin Wu, Xiaoyao Li, Isaku Yamahata
In-Reply-To: <aYmoIaFwgR6+hnGp@yzhao56-desk.sh.intel.com>
On Mon, Feb 09, 2026, Yan Zhao wrote:
> On Fri, Feb 06, 2026 at 07:01:14AM -0800, Sean Christopherson wrote:
> > @@ -2348,7 +2348,7 @@ void __tdx_pamt_put(u64 pfn)
> > if (!atomic_dec_and_test(pamt_refcount))
> > return;
> >
> > - scoped_guard(spinlock, &pamt_lock) {
> > + scoped_guard(raw_spinlock_irqsave, &pamt_lock) {
> > /* Lost race with tdx_pamt_get(). */
> > if (atomic_read(pamt_refcount))
> > return;
>
> This option can get rid of the warning.
>
> However, given the pamt_lock is a global lock, which may be acquired even in the
> softirq context, not sure if this irq disabled version is good.
FWIW, the SEAMCALL itself disables IRQs (and everything else), so it's not _that_
big of a change. But yeah, waiting on the spinlock with IRQs disabled isn't
exactly idea.
> For your reference, I measured some test data by concurrently launching and
> destroying 4 TDs for 3 rounds:
>
> t0 ---------------------
> scoped_guard(spinlock, &pamt_lock) { |->T1=t1-t0 |
> t1 ---------- |
> ... |
> t2 ---------- |->T3=t4-t0
> tdh_phymem_pamt_add/remove() |->T2=t3-t2 |
> t3 ---------- |
> ... |
> t4 ---------------------
> }
>
> (1) for __tdx_pamt_get()
>
> avg us min us max us
> ------|---------------------------
> T1 | 4 0 69
> T2 | 4 2 18
> T3 | 10 3 83
>
>
> (2) for__tdx_pamt_put()
>
> avg us min us max us
> ------|---------------------------
> T1 | 0 0 5
> T2 | 2 1 11
> T3 | 3 2 15
>
>
> > Option #2 would be to immediately free the page in tdx_sept_reclaim_private_sp(),
> > so that pages that freed via handle_removed_pt() don't defer freeing the S-EPT
> > page table (which, IIUC, is safe since the TDX-Module forces TLB flushes and exits).
> >
> > I really, really don't like this option (if it even works).
> I don't like its asymmetry with tdx_sept_link_private_spt().
>
> However, do you think it would be good to have the PAMT pages of the sept pages
> allocated from (*topup_private_mapping_cache) [1]?
Hrm, dunno about "good", but it's definitely not terrible. To get the cache
management right, it means adding yet another use of kvm_get_running_vcpu(), which
I really dislike.
On the other hand, if we combine that with TDX freeing in-use S-EPT page tables,
unless I'm overly simplifying things, it would avoid having to extend
kvm_mmu_memory_cache with the page_{get,free}() hook, and would then eliminate
two kvm_x86_ops hooks, because the alloc/free of _unused_ S-EPT page tables is
no different than regular page tables.
As a bonus, we could keep the topup_external_cache() name and just clarify that
the parameter specifies the number of page table pages, i.e. account for the +1
for the mapping page in TDX code.
All in all, I'm kinda leaning in this direction, because as much as I dislike
kvm_get_running_vcpu(), it does minimize the number of kvm_x86_ops hooks.
Something like this? Also pushed to
https://github.com/sean-jc/linux.git x86/tdx_huge_sept_alt
if it doesn't apply.
---
arch/x86/include/asm/kvm-x86-ops.h | 6 +--
arch/x86/include/asm/kvm_host.h | 15 ++------
arch/x86/kvm/mmu/mmu.c | 3 --
arch/x86/kvm/mmu/tdp_mmu.c | 23 +++++++-----
arch/x86/kvm/vmx/tdx.c | 60 ++++++++++++++++++++----------
5 files changed, 61 insertions(+), 46 deletions(-)
diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 6083fb07cd3b..4b865617a421 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -94,11 +94,9 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr)
KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr)
KVM_X86_OP_OPTIONAL_RET0(get_mt_mask)
KVM_X86_OP(load_mmu_pgd)
-KVM_X86_OP_OPTIONAL(alloc_external_sp)
-KVM_X86_OP_OPTIONAL(free_external_sp)
-KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
-KVM_X86_OP_OPTIONAL(reclaim_external_sp)
+KVM_X86_OP_OPTIONAL(reclaim_external_spt)
KVM_X86_OP_OPTIONAL_RET0(topup_external_cache)
+KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
KVM_X86_OP(has_wbinvd_exit)
KVM_X86_OP(get_l2_tsc_offset)
KVM_X86_OP(get_l2_tsc_multiplier)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index cd3e7dc6ab9b..d3c31eaf18b1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1850,19 +1850,12 @@ struct kvm_x86_ops {
void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
int root_level);
- /*
- * Callbacks to allocate and free external page tables, a.k.a. S-EPT,
- * and to propagate changes in mirror page tables to the external page
- * tables.
- */
- unsigned long (*alloc_external_sp)(gfp_t gfp);
- void (*free_external_sp)(unsigned long addr);
+ void (*reclaim_external_spt)(struct kvm *kvm, gfn_t gfn,
+ struct kvm_mmu_page *sp);
+ int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu,
+ int min_nr_spts);
int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level);
- void (*reclaim_external_sp)(struct kvm *kvm, gfn_t gfn,
- struct kvm_mmu_page *sp);
- int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu, int min);
-
bool (*has_wbinvd_exit)(void);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 62bf6bec2df2..f7cf456d9404 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6714,9 +6714,6 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
if (!vcpu->arch.mmu_shadow_page_cache.init_value)
vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
- vcpu->arch.mmu_external_spt_cache.page_get = kvm_x86_ops.alloc_external_sp;
- vcpu->arch.mmu_external_spt_cache.page_free = kvm_x86_ops.free_external_sp;
-
vcpu->arch.mmu = &vcpu->arch.root_mmu;
vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index fef856323821..732548a678d8 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -53,14 +53,18 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
rcu_barrier();
}
-static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
+static void __tdp_mmu_free_sp(struct kvm_mmu_page *sp)
{
- if (sp->external_spt)
- kvm_x86_call(free_external_sp)((unsigned long)sp->external_spt);
free_page((unsigned long)sp->spt);
kmem_cache_free(mmu_page_header_cache, sp);
}
+static void tdp_mmu_free_unused_sp(struct kvm_mmu_page *sp)
+{
+ free_page((unsigned long)sp->external_spt);
+ __tdp_mmu_free_sp(sp);
+}
+
/*
* This is called through call_rcu in order to free TDP page table memory
* safely with respect to other kernel threads that may be operating on
@@ -74,7 +78,8 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
rcu_head);
- tdp_mmu_free_sp(sp);
+ WARN_ON_ONCE(sp->external_spt);
+ __tdp_mmu_free_sp(sp);
}
void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
@@ -458,7 +463,7 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared)
}
if (is_mirror_sp(sp))
- kvm_x86_call(reclaim_external_sp)(kvm, base_gfn, sp);
+ kvm_x86_call(reclaim_external_spt)(kvm, base_gfn, sp);
call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback);
}
@@ -1266,7 +1271,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
* failed, e.g. because a different task modified the SPTE.
*/
if (r) {
- tdp_mmu_free_sp(sp);
+ tdp_mmu_free_unused_sp(sp);
goto retry;
}
@@ -1461,7 +1466,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,
goto err_spt;
if (is_mirror_sp) {
- sp->external_spt = (void *)kvm_x86_call(alloc_external_sp)(GFP_KERNEL_ACCOUNT);
+ sp->external_spt = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
if (!sp->external_spt)
goto err_external_spt;
@@ -1472,7 +1477,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,
return sp;
err_external_split:
- kvm_x86_call(free_external_sp)((unsigned long)sp->external_spt);
+ free_page((unsigned long)sp->external_spt);
err_external_spt:
free_page((unsigned long)sp->spt);
err_spt:
@@ -1594,7 +1599,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
* installs its own sp in place of the last sp we tried to split.
*/
if (sp)
- tdp_mmu_free_sp(sp);
+ tdp_mmu_free_unused_sp(sp);
return 0;
}
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index ae7b9beb3249..b0fc17baa1fc 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1701,7 +1701,7 @@ static struct tdx_pamt_cache *tdx_get_pamt_cache(struct kvm *kvm,
}
static int tdx_topup_external_pamt_cache(struct kvm *kvm,
- struct kvm_vcpu *vcpu, int min)
+ struct kvm_vcpu *vcpu, int min_nr_spts)
{
struct tdx_pamt_cache *pamt_cache;
@@ -1712,7 +1712,11 @@ static int tdx_topup_external_pamt_cache(struct kvm *kvm,
if (!pamt_cache)
return -EIO;
- return tdx_topup_pamt_cache(pamt_cache, min);
+ /*
+ * Each S-EPT page tables requires a DPAMT pair, plus one more for the
+ * memory being mapped into the guest.
+ */
+ return tdx_topup_pamt_cache(pamt_cache, min_nr_spts + 1);
}
static int tdx_mem_page_add(struct kvm *kvm, gfn_t gfn, enum pg_level level,
@@ -1911,23 +1915,41 @@ static int tdx_sept_split_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, u64 new_spte,
enum pg_level level)
{
+ struct tdx_pamt_cache *pamt_cache;
gpa_t gpa = gfn_to_gpa(gfn);
u64 err, entry, level_state;
struct page *external_spt;
+ int r;
external_spt = tdx_spte_to_external_spt(kvm, gfn, new_spte, level);
if (!external_spt)
return -EIO;
+ pamt_cache = tdx_get_pamt_cache(kvm, kvm_get_running_vcpu());
+ if (!pamt_cache)
+ return -EIO;
+
+ r = tdx_pamt_get(page_to_pfn(external_spt), PG_LEVEL_4K, pamt_cache);
+ if (r)
+ return r;
+
err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, external_spt,
&entry, &level_state);
- if (unlikely(IS_TDX_OPERAND_BUSY(err)))
- return -EBUSY;
+ if (unlikely(IS_TDX_OPERAND_BUSY(err))) {
+ r = -EBUSY;
+ goto err;
+ }
- if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm))
- return -EIO;
+ if (TDX_BUG_ON_2(err, TDH_MEM_SEPT_ADD, entry, level_state, kvm)) {
+ r = -EIO;
+ goto err;
+ }
return 0;
+
+err:
+ tdx_pamt_put(page_to_pfn(external_spt), PG_LEVEL_4K);
+ return r;
}
static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
@@ -1995,8 +2017,8 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
return tdx_sept_map_leaf_spte(kvm, gfn, new_spte, level);
}
-static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
- struct kvm_mmu_page *sp)
+static void tdx_sept_reclaim_private_spt(struct kvm *kvm, gfn_t gfn,
+ struct kvm_mmu_page *sp)
{
/*
* KVM doesn't (yet) zap page table pages in mirror page table while
@@ -2014,7 +2036,16 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
*/
if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) ||
tdx_reclaim_page(virt_to_page(sp->external_spt)))
- sp->external_spt = NULL;
+ goto out;
+
+ /*
+ * Immediately free the S-EPT page as the TDX subsystem doesn't support
+ * freeing pages from RCU callbacks, and more importantly because
+ * TDH.PHYMEM.PAGE.RECLAIM ensures there are no outstanding readers.
+ */
+ tdx_free_control_page((unsigned long)sp->external_spt);
+out:
+ sp->external_spt = NULL;
}
void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
@@ -3869,17 +3900,8 @@ void __init tdx_hardware_setup(void)
*/
vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, sizeof(struct kvm_tdx));
- /*
- * TDX uses the external_spt cache to allocate S-EPT page table pages,
- * which (a) don't need to be initialized by KVM as the TDX-Module will
- * initialize the page (using the guest's encryption key), and (b) need
- * to use a custom allocator to be compatible with Dynamic PAMT.
- */
- vt_x86_ops.alloc_external_sp = tdx_alloc_control_page;
- vt_x86_ops.free_external_sp = tdx_free_control_page;
-
+ vt_x86_ops.reclaim_external_spt = tdx_sept_reclaim_private_spt;
vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;
- vt_x86_ops.reclaim_external_sp = tdx_sept_reclaim_private_sp;
vt_x86_ops.gmem_convert = tdx_gmem_convert;
base-commit: 7adb9e428488cf7873a122043385a50dc1eebc8f
--
^ permalink raw reply related
* Re: [RFC PATCH v5 20/45] KVM: x86/mmu: Allocate/free S-EPT pages using tdx_{alloc,free}_control_page()
From: Sean Christopherson @ 2026-02-09 22:44 UTC (permalink / raw)
To: Kai Huang
Cc: Yan Y Zhao, kvm@vger.kernel.org, linux-coco@lists.linux.dev,
Xiaoyao Li, dave.hansen@linux.intel.com, kas@kernel.org,
binbin.wu@linux.intel.com, mingo@redhat.com, pbonzini@redhat.com,
ackerleytng@google.com, linux-kernel@vger.kernel.org,
Isaku Yamahata, sagis@google.com, tglx@kernel.org,
Rick P Edgecombe, bp@alien8.de, Vishal Annapurve, x86@kernel.org
In-Reply-To: <c753636171f82c3b6d64e7734be3a70c60775546.camel@intel.com>
On Mon, Feb 09, 2026, Kai Huang wrote:
> > Option #2 would be to immediately free the page in tdx_sept_reclaim_private_sp(),
> > so that pages that freed via handle_removed_pt() don't defer freeing the S-EPT
> > page table (which, IIUC, is safe since the TDX-Module forces TLB flushes and exits).
> >
> > I really, really don't like this option (if it even works).
> >
> > diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
> > index ae7b9beb3249..4726011ad624 100644
> > --- a/arch/x86/kvm/vmx/tdx.c
> > +++ b/arch/x86/kvm/vmx/tdx.c
> > @@ -2014,7 +2014,15 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
> > */
> > if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) ||
> > tdx_reclaim_page(virt_to_page(sp->external_spt)))
> > - sp->external_spt = NULL;
> > + goto out;
> > +
> > + /*
> > + * Immediately free the control page, as the TDX subsystem doesn't
> > + * support freeing pages from RCU callbacks.
> > + */
> > + tdx_free_control_page((unsigned long)sp->external_spt);
> > +out:
> > + sp->external_spt = NULL;
> > }
> >
> > void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
>
> I don't think this is so bad, given we already have a bunch of
>
> is_mirror_sp(sp)
> kvm_x86_call(xx_external_spt)(..);
>
> in TDP MMU code?
>
> I suppose this won't make a lot of difference, but does below make you
> slightly happier?
Heh, slightly, but I still don't love it :-D
> diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
> index 3181406c5e0b..3588265098a8 100644
> --- a/arch/x86/kvm/mmu/tdp_mmu.c
> +++ b/arch/x86/kvm/mmu/tdp_mmu.c
> @@ -55,8 +55,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
>
> static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
> {
> - if (sp->external_spt)
> - kvm_x86_call(free_external_sp)((unsigned long)sp-
> >external_spt);
> + WARN_ON_ONCE(sp->external_spt);
Doesn't work, because sp->external_spt will be non-NULL when KVM is freeing
unused pages in tdp_mmu_split_huge_pages_root() and kvm_tdp_mmu_map(). That's
solvable, but it's part of the asymmetry I don't love. AFAICT, unless we do
something truly awful, there's no way to avoid having common KVM free unused
S-EPT pages.
That said, while I don't love the asymmetry, it's not a deal breaker, especially
if we make the asymmetry super obvious and cleanly delineated. Specifically, if
we differentiate between freeing unused page tables and freeing used (linked at
any point) page tables.
This would also allow us to address the naming than Yan doesn't like around
reclaim_external_sp(), because we could have both free_external_sp() and
free_unused_external_spt(), where the lack of "unused" gives the reader a hint
that there's interesting work to be done for in-use external page tables.
This won't apply cleanly due to other fixups. It's also at:
https://github.com/sean-jc/linux.git x86/tdx_huge_sept
---
arch/x86/include/asm/kvm-x86-ops.h | 8 ++++----
arch/x86/include/asm/kvm_host.h | 10 +++++-----
arch/x86/kvm/mmu/mmu.c | 4 ++--
arch/x86/kvm/mmu/tdp_mmu.c | 27 +++++++++++++++++----------
arch/x86/kvm/vmx/tdx.c | 22 +++++++++++++++-------
5 files changed, 43 insertions(+), 28 deletions(-)
diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 87826176fa8d..9593d8d97f6b 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -94,11 +94,11 @@ KVM_X86_OP_OPTIONAL_RET0(set_tss_addr)
KVM_X86_OP_OPTIONAL_RET0(set_identity_map_addr)
KVM_X86_OP_OPTIONAL_RET0(get_mt_mask)
KVM_X86_OP(load_mmu_pgd)
-KVM_X86_OP_OPTIONAL(alloc_external_sp)
-KVM_X86_OP_OPTIONAL(free_external_sp)
-KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
-KVM_X86_OP_OPTIONAL(reclaim_external_sp)
+KVM_X86_OP_OPTIONAL_RET0(alloc_external_spt)
+KVM_X86_OP_OPTIONAL(free_external_spt)
+KVM_X86_OP_OPTIONAL(free_unused_external_spt)
KVM_X86_OP_OPTIONAL_RET0(topup_private_mapping_cache)
+KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
KVM_X86_OP(has_wbinvd_exit)
KVM_X86_OP(get_l2_tsc_offset)
KVM_X86_OP(get_l2_tsc_multiplier)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 7ff72c04d575..5fc25508548b 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1855,13 +1855,13 @@ struct kvm_x86_ops {
* and to propagate changes in mirror page tables to the external page
* tables.
*/
- unsigned long (*alloc_external_sp)(gfp_t gfp);
- void (*free_external_sp)(unsigned long addr);
+ unsigned long (*alloc_external_spt)(gfp_t gfp);
+ void (*free_external_spt)(struct kvm *kvm, gfn_t gfn,
+ struct kvm_mmu_page *sp);
+ void (*free_unused_external_spt)(unsigned long external_spt);
+ int (*topup_private_mapping_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu);
int (*set_external_spte)(struct kvm *kvm, gfn_t gfn, u64 old_spte,
u64 new_spte, enum pg_level level);
- void (*reclaim_external_sp)(struct kvm *kvm, gfn_t gfn,
- struct kvm_mmu_page *sp);
- int (*topup_private_mapping_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu);
bool (*has_wbinvd_exit)(void);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 6a911aec075b..2ad417ac6e1f 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6714,8 +6714,8 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu)
if (!vcpu->arch.mmu_shadow_page_cache.init_value)
vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO;
- vcpu->arch.mmu_external_spt_cache.page_get = kvm_x86_ops.alloc_external_sp;
- vcpu->arch.mmu_external_spt_cache.page_free = kvm_x86_ops.free_external_sp;
+ vcpu->arch.mmu_external_spt_cache.page_get = kvm_x86_ops.alloc_external_spt;
+ vcpu->arch.mmu_external_spt_cache.page_free = kvm_x86_ops.free_unused_external_spt;
vcpu->arch.mmu = &vcpu->arch.root_mmu;
vcpu->arch.walk_mmu = &vcpu->arch.root_mmu;
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index b7c13316181b..d43db86b12a7 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -53,12 +53,18 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
rcu_barrier();
}
-static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
+static void __tdp_mmu_free_sp(struct kvm_mmu_page *sp)
+{
+ free_page((unsigned long)sp->spt);
+ kmem_cache_free(mmu_page_header_cache, sp);
+}
+
+static void tdp_mmu_free_unused_sp(struct kvm_mmu_page *sp)
{
if (sp->external_spt)
- kvm_x86_call(free_external_sp)((unsigned long)sp->external_spt);
- free_page((unsigned long)sp->spt);
- kmem_cache_free(mmu_page_header_cache, sp);
+ kvm_x86_call(free_unused_external_spt)((unsigned long)sp->external_spt);
+
+ __tdp_mmu_free_sp(sp);
}
/*
@@ -74,7 +80,8 @@ static void tdp_mmu_free_sp_rcu_callback(struct rcu_head *head)
struct kvm_mmu_page *sp = container_of(head, struct kvm_mmu_page,
rcu_head);
- tdp_mmu_free_sp(sp);
+ WARN_ON_ONCE(sp->external_spt);
+ __tdp_mmu_free_sp(sp);
}
void kvm_tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root)
@@ -458,7 +465,7 @@ static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared)
}
if (is_mirror_sp(sp))
- kvm_x86_call(reclaim_external_sp)(kvm, base_gfn, sp);
+ kvm_x86_call(free_external_spt)(kvm, base_gfn, sp);
call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback);
}
@@ -1266,7 +1273,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
* failed, e.g. because a different task modified the SPTE.
*/
if (r) {
- tdp_mmu_free_sp(sp);
+ tdp_mmu_free_unused_sp(sp);
goto retry;
}
@@ -1461,7 +1468,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,
goto err_spt;
if (is_mirror_sp) {
- sp->external_spt = (void *)kvm_x86_call(alloc_external_sp)(GFP_KERNEL_ACCOUNT);
+ sp->external_spt = (void *)kvm_x86_call(alloc_external_spt)(GFP_KERNEL_ACCOUNT);
if (!sp->external_spt)
goto err_external_spt;
@@ -1472,7 +1479,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm,
return sp;
err_external_split:
- kvm_x86_call(free_external_sp)((unsigned long)sp->external_spt);
+ kvm_x86_call(free_unused_external_spt)((unsigned long)sp->external_spt);
err_external_spt:
free_page((unsigned long)sp->spt);
err_spt:
@@ -1594,7 +1601,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
* installs its own sp in place of the last sp we tried to split.
*/
if (sp)
- tdp_mmu_free_sp(sp);
+ tdp_mmu_free_unused_sp(sp);
return 0;
}
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 957fa59e9a65..aae7af26fe02 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1994,8 +1994,8 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
return tdx_sept_map_leaf_spte(kvm, gfn, new_spte, level);
}
-static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
- struct kvm_mmu_page *sp)
+static void tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,
+ struct kvm_mmu_page *sp)
{
/*
* KVM doesn't (yet) zap page table pages in mirror page table while
@@ -2013,7 +2013,16 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
*/
if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) ||
tdx_reclaim_page(virt_to_page(sp->external_spt)))
- sp->external_spt = NULL;
+ goto out;
+
+ /*
+ * Immediately free the S-EPT page as the TDX subsystem doesn't support
+ * freeing pages from RCU callbacks, and more importantly because
+ * TDH.PHYMEM.PAGE.RECLAIM ensures there are no outstanding readers.
+ */
+ tdx_free_control_page((unsigned long)sp->external_spt);
+out:
+ sp->external_spt = NULL;
}
void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
@@ -3874,11 +3883,10 @@ void __init tdx_hardware_setup(void)
* initialize the page (using the guest's encryption key), and (b) need
* to use a custom allocator to be compatible with Dynamic PAMT.
*/
- vt_x86_ops.alloc_external_sp = tdx_alloc_control_page;
- vt_x86_ops.free_external_sp = tdx_free_control_page;
-
+ vt_x86_ops.alloc_external_spt = tdx_alloc_control_page;
+ vt_x86_ops.free_unused_external_spt = tdx_free_control_page;
+ vt_x86_ops.free_external_spt = tdx_sept_free_private_spt;
vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;
- vt_x86_ops.reclaim_external_sp = tdx_sept_reclaim_private_sp;
vt_x86_ops.gmem_convert = tdx_gmem_convert;
base-commit: 0191132f233a66b5cb1ae9a09c18c6d6669c284a
--
^ permalink raw reply related
* Re: [PATCH] crypto: ccp - allow callers to use HV-Fixed page API when SEV is disabled
From: Tom Lendacky @ 2026-02-09 22:42 UTC (permalink / raw)
To: Ashish Kalra, john.allen, herbert, davem, bp
Cc: linux-crypto, linux-kernel, linux-coco
In-Reply-To: <20260206212645.125485-1-Ashish.Kalra@amd.com>
On 2/6/26 15:26, Ashish Kalra wrote:
> From: Ashish Kalra <ashish.kalra@amd.com>
>
> When SEV is disabled, the HV-Fixed page allocation call fails, which in
> turn causes SFS initialization to fail.
>
> Fix the HV-Fixed API so callers (for example, SFS) can use it even when
> SEV is disabled by performing normal page allocation and freeing.
>
> Fixes: e09701dcdd9c ("crypto: ccp - Add new HV-Fixed page allocation/free API")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
> drivers/crypto/ccp/sev-dev.c | 10 ++++------
> 1 file changed, 4 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
> index 1cdadddb744e..0d90b5f6a454 100644
> --- a/drivers/crypto/ccp/sev-dev.c
> +++ b/drivers/crypto/ccp/sev-dev.c
> @@ -1105,15 +1105,12 @@ struct page *snp_alloc_hv_fixed_pages(unsigned int num_2mb_pages)
> {
> struct psp_device *psp_master = psp_get_master_device();
> struct snp_hv_fixed_pages_entry *entry;
> - struct sev_device *sev;
> unsigned int order;
> struct page *page;
>
> - if (!psp_master || !psp_master->sev_data)
> + if (!psp_master)
> return NULL;
>
> - sev = psp_master->sev_data;
> -
> order = get_order(PMD_SIZE * num_2mb_pages);
>
> /*
> @@ -1126,7 +1123,8 @@ struct page *snp_alloc_hv_fixed_pages(unsigned int num_2mb_pages)
> * This API uses SNP_INIT_EX to transition allocated pages to HV_Fixed
> * page state, fail if SNP is already initialized.
> */
> - if (sev->snp_initialized)
> + if (psp_master->sev_data &&
> + ((struct sev_device *)psp_master->sev_data)->snp_initialized)
> return NULL;
>
> /* Re-use freed pages that match the request */
> @@ -1162,7 +1160,7 @@ void snp_free_hv_fixed_pages(struct page *page)
> struct psp_device *psp_master = psp_get_master_device();
> struct snp_hv_fixed_pages_entry *entry, *nentry;
>
> - if (!psp_master || !psp_master->sev_data)
> + if (!psp_master)
> return;
>
> /*
^ permalink raw reply
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Huang, Kai @ 2026-02-09 21:05 UTC (permalink / raw)
To: seanjc@google.com, Edgecombe, Rick P
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Li, Xiaoyao,
Zhao, Yan Y, dave.hansen@linux.intel.com,
linux-kernel@vger.kernel.org, kas@kernel.org,
binbin.wu@linux.intel.com, pbonzini@redhat.com, mingo@redhat.com,
Yamahata, Isaku, ackerleytng@google.com, tglx@kernel.org,
Annapurve, Vishal, sagis@google.com, bp@alien8.de, x86@kernel.org
In-Reply-To: <ddf6c17664044bf66e7a9a0a58f2b6c1104dfbc4.camel@intel.com>
On Mon, 2026-02-09 at 17:08 +0000, Edgecombe, Rick P wrote:
> On Mon, 2026-02-09 at 10:33 +0000, Huang, Kai wrote:
> > In the fault path, we already know the PFN after
> > kvm_mmu_faultin_pfn(), which is outside of MMU lock.
> >
> > What we still don't know is the actual mapping level, which is
> > currently done in kvm_tdp_mmu_map() via kvm_mmu_hugepage_adjust().
> >
> > However I don't see why we cannot move kvm_mmu_hugepage_adjust() out
> > of it to, e.g., right after kvm_mmu_faultin_pfn()?
> >
> > If we can do this, then AFAICT we can just do:
> >
> > r = kvm_x86_call(prepare_pfn)(vcpu, fault, pfn);
>
> What about the adjustments in disallowed_hugepage_adjust()?
AFAICT that's for preventing replacing existing small leafs with a huge
mapping, which is not supported by TDX in this series (PROMOTE isn't
supported).
Even we want to support PROMOTE in the future, it doesn't impact the logic
that we don't need to call tdx_pamt_get(pfn) if the fault->goal_level is 2M.
When KVM tries to replace the existing small leafs with huge SPTE for TDX,
the PROMOTE returns the now-unneeded PAMT pages and KVM can just free that.
There's no impact to non-TDX case too, I believe, because moving
kvm_mmu_hugepage_adjust() out of kvm_tdp_mmu_map() only moves it out, but
doesn't change the whole logic.
^ permalink raw reply
* Re: [PATCH 4/5] dma-buf: heaps: allow heap to specify valid heap flags
From: John Stultz @ 2026-02-09 20:08 UTC (permalink / raw)
To: Jiri Pirko
Cc: dri-devel, linaro-mm-sig, iommu, linux-media, sumit.semwal,
benjamin.gaignard, Brian.Starkey, tjmercier, christian.koenig,
m.szyprowski, robin.murphy, jgg, leon, sean.anderson, ptesarik,
catalin.marinas, aneesh.kumar, suzuki.poulose, steven.price,
thomas.lendacky, john.allen, ashish.kalra, suravee.suthikulpanit,
linux-coco
In-Reply-To: <20260209153809.250835-5-jiri@resnulli.us>
On Mon, Feb 9, 2026 at 7:38 AM Jiri Pirko <jiri@resnulli.us> wrote:
>
> From: Jiri Pirko <jiri@nvidia.com>
>
> Currently the flags, which are unused, are validated for all heaps.
> Since the follow-up patch introduces a flag valid for only one of the
> heaps, allow to specify the valid flags per-heap.
I'm not really in this space anymore, so take my feedback with a grain of salt.
While the heap allocate flags argument is unused, it was intended to
be used for generic allocation flags that would apply to all or at
least a wide majority of heaps.
It was definitely not added to allow for per-heap or heap specific
flags (as this patch tries to utilize it). That was the mess we had
with ION driver that we were trying to avoid.
The intent of dma-buf heaps is to try to abstract all the different
device memory constraints so there only needs to be a [usage] ->
[heap] mapping, and otherwise userland can be generalized so that it
doesn't need to be re-written to work with different devices/memory
types. Adding heap-specific allocation flags prevents that
generalization.
So instead of adding heap specific flags, the general advice has been
to add a separate heap name for the flag property.
Now, there has been many discussions around "protected buffers" (which
doesn't seem to map exactly to this confidental computing primitive,
but sounds like it might be related) , which have bounced between
being a allocation flag or a device specific heap without much
resolution. I appreciate in this patch seires you've pushed your
concept down into a DMA_ATTR_, as I do feel the kernel should have a
deeper sense of protected buffers (or any general propery like this)
as a concept if it is going to be a generic allocation flag, instead
of it being a somewhat thin creation of the outer heap-driver layer.
But, it seems like the use case here is still far too narrow for a top
level allocation flag.
So I'd advocate against introducing heap-specific flags like this.
thanks
-john
^ permalink raw reply
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Edgecombe, Rick P @ 2026-02-09 17:08 UTC (permalink / raw)
To: seanjc@google.com, Huang, Kai
Cc: kvm@vger.kernel.org, Li, Xiaoyao, linux-coco@lists.linux.dev,
Zhao, Yan Y, dave.hansen@linux.intel.com, kas@kernel.org,
mingo@redhat.com, binbin.wu@linux.intel.com, pbonzini@redhat.com,
Yamahata, Isaku, ackerleytng@google.com,
linux-kernel@vger.kernel.org, tglx@kernel.org, sagis@google.com,
bp@alien8.de, Annapurve, Vishal, x86@kernel.org
In-Reply-To: <94f041b3aa32169fa2e1125edab7bd8fed3a6e59.camel@intel.com>
On Mon, 2026-02-09 at 10:33 +0000, Huang, Kai wrote:
> In the fault path, we already know the PFN after
> kvm_mmu_faultin_pfn(), which is outside of MMU lock.
>
> What we still don't know is the actual mapping level, which is
> currently done in kvm_tdp_mmu_map() via kvm_mmu_hugepage_adjust().
>
> However I don't see why we cannot move kvm_mmu_hugepage_adjust() out
> of it to, e.g., right after kvm_mmu_faultin_pfn()?
>
> If we can do this, then AFAICT we can just do:
>
> r = kvm_x86_call(prepare_pfn)(vcpu, fault, pfn);
What about the adjustments in disallowed_hugepage_adjust()?
^ permalink raw reply
* [PATCH 5/5] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260209153809.250835-1-jiri@resnulli.us>
From: Jiri Pirko <jiri@nvidia.com>
Add a new DMA_HEAP_FLAG_DECRYPTED heap flag to allow userspace to
allocate decrypted (shared) memory from the dma-buf system heap for
confidential computing (CoCo) VMs.
On CoCo VMs, guest memory is encrypted by default. The hardware uses an
encryption bit in page table entries (C-bit on AMD SEV, "shared" bit on
Intel TDX) to control whether a given memory access is encrypted or
decrypted. The kernel's direct map is set up with encryption enabled,
so pages returned by alloc_pages() are encrypted in the direct map
by default. To make this memory usable for devices that do not support
DMA to encrypted memory (no TDISP support), it has to be explicitly
decrypted. A couple of things are needed to properly handle
decrypted memory for the dma-buf use case:
- set_memory_decrypted() on the direct map after allocation:
Besides clearing the encryption bit in the direct map PTEs, this
also notifies the hypervisor about the page state change. On free,
the inverse set_memory_encrypted() must be called before returning
pages to the allocator. If re-encryption fails, pages
are intentionally leaked to prevent decrypted memory from being
reused as private.
- pgprot_decrypted() for userspace and kernel virtual mappings:
Any new mapping of the decrypted pages, be it to userspace via
mmap or to kernel vmalloc space via vmap, creates PTEs independent
of the direct map. These must also have the encryption bit cleared,
otherwise accesses through them would see encrypted (garbage) data.
- DMA_ATTR_CC_DECRYPTED for DMA mapping:
Since the pages are already decrypted, the DMA API needs to be
informed via DMA_ATTR_CC_DECRYPTED so it can map them correctly
as unencrypted for device access.
On non-CoCo VMs the flag is rejected with -EOPNOTSUPP to prevent
misuse by userspace that does not understand the security implications
of explicitly decrypted memory.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
drivers/dma-buf/heaps/system_heap.c | 87 +++++++++++++++++++++++++++--
include/linux/dma-heap.h | 1 +
include/uapi/linux/dma-heap.h | 12 +++-
3 files changed, 94 insertions(+), 6 deletions(-)
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index 124dca56e4d8..0f80ecb660ec 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -10,6 +10,7 @@
* Andrew F. Davis <afd@ti.com>
*/
+#include <linux/cc_platform.h>
#include <linux/dma-buf.h>
#include <linux/dma-mapping.h>
#include <linux/dma-heap.h>
@@ -17,7 +18,9 @@
#include <linux/highmem.h>
#include <linux/mm.h>
#include <linux/module.h>
+#include <linux/pgtable.h>
#include <linux/scatterlist.h>
+#include <linux/set_memory.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -29,6 +32,7 @@ struct system_heap_buffer {
struct sg_table sg_table;
int vmap_cnt;
void *vaddr;
+ bool decrypted;
};
struct dma_heap_attachment {
@@ -36,6 +40,7 @@ struct dma_heap_attachment {
struct sg_table table;
struct list_head list;
bool mapped;
+ bool decrypted;
};
#define LOW_ORDER_GFP (GFP_HIGHUSER | __GFP_ZERO)
@@ -52,6 +57,34 @@ static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
static const unsigned int orders[] = {8, 4, 0};
#define NUM_ORDERS ARRAY_SIZE(orders)
+static int system_heap_set_page_decrypted(struct page *page)
+{
+ unsigned long addr = (unsigned long)page_address(page);
+ unsigned int nr_pages = 1 << compound_order(page);
+ int ret;
+
+ ret = set_memory_decrypted(addr, nr_pages);
+ if (ret)
+ pr_warn_ratelimited("dma-buf system heap: failed to decrypt page at %p\n",
+ page_address(page));
+
+ return ret;
+}
+
+static int system_heap_set_page_encrypted(struct page *page)
+{
+ unsigned long addr = (unsigned long)page_address(page);
+ unsigned int nr_pages = 1 << compound_order(page);
+ int ret;
+
+ ret = set_memory_encrypted(addr, nr_pages);
+ if (ret)
+ pr_warn_ratelimited("dma-buf system heap: failed to re-encrypt page at %p, leaking memory\n",
+ page_address(page));
+
+ return ret;
+}
+
static int dup_sg_table(struct sg_table *from, struct sg_table *to)
{
struct scatterlist *sg, *new_sg;
@@ -90,6 +123,7 @@ static int system_heap_attach(struct dma_buf *dmabuf,
a->dev = attachment->dev;
INIT_LIST_HEAD(&a->list);
a->mapped = false;
+ a->decrypted = buffer->decrypted;
attachment->priv = a;
@@ -119,9 +153,11 @@ static struct sg_table *system_heap_map_dma_buf(struct dma_buf_attachment *attac
{
struct dma_heap_attachment *a = attachment->priv;
struct sg_table *table = &a->table;
+ unsigned long attrs;
int ret;
- ret = dma_map_sgtable(attachment->dev, table, direction, 0);
+ attrs = a->decrypted ? DMA_ATTR_CC_DECRYPTED : 0;
+ ret = dma_map_sgtable(attachment->dev, table, direction, attrs);
if (ret)
return ERR_PTR(ret);
@@ -188,8 +224,13 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
unsigned long addr = vma->vm_start;
unsigned long pgoff = vma->vm_pgoff;
struct scatterlist *sg;
+ pgprot_t prot;
int i, ret;
+ prot = vma->vm_page_prot;
+ if (buffer->decrypted)
+ prot = pgprot_decrypted(prot);
+
for_each_sgtable_sg(table, sg, i) {
unsigned long n = sg->length >> PAGE_SHIFT;
@@ -206,8 +247,7 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
if (addr + size > vma->vm_end)
size = vma->vm_end - addr;
- ret = remap_pfn_range(vma, addr, page_to_pfn(page),
- size, vma->vm_page_prot);
+ ret = remap_pfn_range(vma, addr, page_to_pfn(page), size, prot);
if (ret)
return ret;
@@ -225,6 +265,7 @@ static void *system_heap_do_vmap(struct system_heap_buffer *buffer)
struct page **pages = vmalloc(sizeof(struct page *) * npages);
struct page **tmp = pages;
struct sg_page_iter piter;
+ pgprot_t prot;
void *vaddr;
if (!pages)
@@ -235,7 +276,10 @@ static void *system_heap_do_vmap(struct system_heap_buffer *buffer)
*tmp++ = sg_page_iter_page(&piter);
}
- vaddr = vmap(pages, npages, VM_MAP, PAGE_KERNEL);
+ prot = PAGE_KERNEL;
+ if (buffer->decrypted)
+ prot = pgprot_decrypted(prot);
+ vaddr = vmap(pages, npages, VM_MAP, prot);
vfree(pages);
if (!vaddr)
@@ -296,6 +340,14 @@ static void system_heap_dma_buf_release(struct dma_buf *dmabuf)
for_each_sgtable_sg(table, sg, i) {
struct page *page = sg_page(sg);
+ /*
+ * Intentionally leak pages that cannot be re-encrypted
+ * to prevent decrypted memory from being reused.
+ */
+ if (buffer->decrypted &&
+ system_heap_set_page_encrypted(page))
+ continue;
+
__free_pages(page, compound_order(page));
}
sg_free_table(table);
@@ -344,6 +396,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
unsigned long size_remaining = len;
unsigned int max_order = orders[0];
+ bool decrypted = heap_flags & DMA_HEAP_FLAG_DECRYPTED;
struct dma_buf *dmabuf;
struct sg_table *table;
struct scatterlist *sg;
@@ -351,6 +404,15 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
struct page *page, *tmp_page;
int i, ret = -ENOMEM;
+ if (decrypted) {
+ if (!cc_platform_has(CC_ATTR_MEM_ENCRYPT))
+ return ERR_PTR(-EOPNOTSUPP);
+#ifdef CONFIG_HIGHMEM
+ /* Sanity check, should not happen. */
+ return ERR_PTR(-EINVAL);
+#endif
+ }
+
buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
if (!buffer)
return ERR_PTR(-ENOMEM);
@@ -359,6 +421,7 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
mutex_init(&buffer->lock);
buffer->heap = heap;
buffer->len = len;
+ buffer->decrypted = decrypted;
INIT_LIST_HEAD(&pages);
i = 0;
@@ -393,6 +456,14 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
list_del(&page->lru);
}
+ if (decrypted) {
+ for_each_sgtable_sg(table, sg, i) {
+ ret = system_heap_set_page_decrypted(sg_page(sg));
+ if (ret)
+ goto free_pages;
+ }
+ }
+
/* create the dmabuf */
exp_info.exp_name = dma_heap_get_name(heap);
exp_info.ops = &system_heap_buf_ops;
@@ -410,6 +481,13 @@ static struct dma_buf *system_heap_allocate(struct dma_heap *heap,
for_each_sgtable_sg(table, sg, i) {
struct page *p = sg_page(sg);
+ /*
+ * Intentionally leak pages that cannot be re-encrypted
+ * to prevent decrypted memory from being reused.
+ */
+ if (buffer->decrypted &&
+ system_heap_set_page_encrypted(p))
+ continue;
__free_pages(p, compound_order(p));
}
sg_free_table(table);
@@ -430,6 +508,7 @@ static int __init system_heap_create(void)
struct dma_heap_export_info exp_info = {
.name = "system",
.ops = &system_heap_ops,
+ .valid_heap_flags = DMA_HEAP_FLAG_DECRYPTED,
};
struct dma_heap *sys_heap;
diff --git a/include/linux/dma-heap.h b/include/linux/dma-heap.h
index 7cfb531a9281..295a7eaa19ca 100644
--- a/include/linux/dma-heap.h
+++ b/include/linux/dma-heap.h
@@ -10,6 +10,7 @@
#define _DMA_HEAPS_H
#include <linux/types.h>
+#include <uapi/linux/dma-heap.h>
struct dma_heap;
diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
index a4cf716a49fa..6552c88e52f6 100644
--- a/include/uapi/linux/dma-heap.h
+++ b/include/uapi/linux/dma-heap.h
@@ -18,8 +18,16 @@
/* Valid FD_FLAGS are O_CLOEXEC, O_RDONLY, O_WRONLY, O_RDWR */
#define DMA_HEAP_VALID_FD_FLAGS (O_CLOEXEC | O_ACCMODE)
-/* Currently no heap flags */
-#define DMA_HEAP_VALID_HEAP_FLAGS (0ULL)
+/**
+ * DMA_HEAP_FLAG_DECRYPTED - Allocate decrypted (shared) memory
+ *
+ * For confidential computing guests (AMD SEV, Intel TDX), this flag
+ * requests that the allocated memory be marked as decrypted (shared
+ * with the host).
+ */
+#define DMA_HEAP_FLAG_DECRYPTED (1ULL << 0)
+
+#define DMA_HEAP_VALID_HEAP_FLAGS (DMA_HEAP_FLAG_DECRYPTED)
/**
* struct dma_heap_allocation_data - metadata passed from userspace for
--
2.51.1
^ permalink raw reply related
* [PATCH 4/5] dma-buf: heaps: allow heap to specify valid heap flags
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260209153809.250835-1-jiri@resnulli.us>
From: Jiri Pirko <jiri@nvidia.com>
Currently the flags, which are unused, are validated for all heaps.
Since the follow-up patch introduces a flag valid for only one of the
heaps, allow to specify the valid flags per-heap.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
drivers/dma-buf/dma-heap.c | 5 ++++-
include/linux/dma-heap.h | 2 ++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index 8ab49924f8b7..4751bcef4b19 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -28,6 +28,7 @@
* @name: used for debugging/device-node name
* @ops: ops struct for this heap
* @priv: private data for this heap
+ * @valid_heap_flags: valid heap flags for this heap
* @heap_devt: heap device node
* @list: list head connecting to list of heaps
* @heap_cdev: heap char device
@@ -38,6 +39,7 @@ struct dma_heap {
const char *name;
const struct dma_heap_ops *ops;
void *priv;
+ u64 valid_heap_flags;
dev_t heap_devt;
struct list_head list;
struct cdev heap_cdev;
@@ -105,7 +107,7 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
if (heap_allocation->fd_flags & ~DMA_HEAP_VALID_FD_FLAGS)
return -EINVAL;
- if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
+ if (heap_allocation->heap_flags & ~heap->valid_heap_flags)
return -EINVAL;
fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
@@ -246,6 +248,7 @@ struct dma_heap *dma_heap_add(const struct dma_heap_export_info *exp_info)
heap->name = exp_info->name;
heap->ops = exp_info->ops;
heap->priv = exp_info->priv;
+ heap->valid_heap_flags = exp_info->valid_heap_flags;
/* Find unused minor number */
ret = xa_alloc(&dma_heap_minors, &minor, heap,
diff --git a/include/linux/dma-heap.h b/include/linux/dma-heap.h
index 27d15f60950a..7cfb531a9281 100644
--- a/include/linux/dma-heap.h
+++ b/include/linux/dma-heap.h
@@ -31,6 +31,7 @@ struct dma_heap_ops {
* @name: used for debugging/device-node name
* @ops: ops struct for this heap
* @priv: heap exporter private data
+ * @valid_heap_flags: valid heap flags for this heap
*
* Information needed to export a new dmabuf heap.
*/
@@ -38,6 +39,7 @@ struct dma_heap_export_info {
const char *name;
const struct dma_heap_ops *ops;
void *priv;
+ u64 valid_heap_flags;
};
void *dma_heap_get_drvdata(struct dma_heap *heap);
--
2.51.1
^ permalink raw reply related
* [PATCH 3/5] dma-buf: heaps: use designated initializer for exp_info
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260209153809.250835-1-jiri@resnulli.us>
From: Jiri Pirko <jiri@nvidia.com>
Use designated initializer for dma_heap_export_info instead of
separate field assignments and avoid the need to explicitly
zero fields in preparation to follow-up patch.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
drivers/dma-buf/heaps/cma_heap.c | 7 ++++---
drivers/dma-buf/heaps/system_heap.c | 9 ++++-----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c
index 42f88193eab9..d12c98be7fa9 100644
--- a/drivers/dma-buf/heaps/cma_heap.c
+++ b/drivers/dma-buf/heaps/cma_heap.c
@@ -388,7 +388,10 @@ static const struct dma_heap_ops cma_heap_ops = {
static int __init __add_cma_heap(struct cma *cma, const char *name)
{
- struct dma_heap_export_info exp_info;
+ struct dma_heap_export_info exp_info = {
+ .name = name,
+ .ops = &cma_heap_ops,
+ };
struct cma_heap *cma_heap;
cma_heap = kzalloc(sizeof(*cma_heap), GFP_KERNEL);
@@ -396,8 +399,6 @@ static int __init __add_cma_heap(struct cma *cma, const char *name)
return -ENOMEM;
cma_heap->cma = cma;
- exp_info.name = name;
- exp_info.ops = &cma_heap_ops;
exp_info.priv = cma_heap;
cma_heap->heap = dma_heap_add(&exp_info);
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index 4c782fe33fd4..124dca56e4d8 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -427,13 +427,12 @@ static const struct dma_heap_ops system_heap_ops = {
static int __init system_heap_create(void)
{
- struct dma_heap_export_info exp_info;
+ struct dma_heap_export_info exp_info = {
+ .name = "system",
+ .ops = &system_heap_ops,
+ };
struct dma_heap *sys_heap;
- exp_info.name = "system";
- exp_info.ops = &system_heap_ops;
- exp_info.priv = NULL;
-
sys_heap = dma_heap_add(&exp_info);
if (IS_ERR(sys_heap))
return PTR_ERR(sys_heap);
--
2.51.1
^ permalink raw reply related
* [PATCH 2/5] dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260209153809.250835-1-jiri@resnulli.us>
From: Jiri Pirko <jiri@nvidia.com>
This is only relevant inside confidential computing (CoCo) virtual
machines, not on the hypervisor side.
Current CoCo designs don't place a vIOMMU in front of untrusted devices.
Instead, the DMA API forces all untrusted device DMA through swiotlb
bounce buffers (is_swiotlb_force_bounce()) which copies data into
decrypted memory on behalf of the device.
When a caller has already arranged for the memory to be decrypted
via set_memory_decrypted(), the DMA API needs to know so it can map
directly using the unencrypted physical address rather than bounce
buffering. Following the pattern of DMA_ATTR_MMIO, add
DMA_ATTR_CC_DECRYPTED for this purpose. Like the MMIO case, only the
caller knows what kind of memory it has and must inform the DMA API
for it to work correctly.
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
include/linux/dma-mapping.h | 7 +++++++
include/trace/events/dma.h | 3 ++-
kernel/dma/direct.h | 14 +++++++++++---
3 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index aa36a0d1d9df..052235feb853 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -78,6 +78,13 @@
*/
#define DMA_ATTR_MMIO (1UL << 10)
+/*
+ * DMA_ATTR_CC_DECRYPTED: Indicates memory that has been explicitly decrypted
+ * (shared) for confidential computing guests. The caller must have
+ * called set_memory_decrypted(). A struct page is required.
+ */
+#define DMA_ATTR_CC_DECRYPTED (1UL << 11)
+
/*
* A dma_addr_t can hold any valid DMA or bus address for the platform. It can
* be given to a device to use as a DMA source or target. It is specific to a
diff --git a/include/trace/events/dma.h b/include/trace/events/dma.h
index b3fef140ae15..b3c2cee8841a 100644
--- a/include/trace/events/dma.h
+++ b/include/trace/events/dma.h
@@ -32,7 +32,8 @@ TRACE_DEFINE_ENUM(DMA_NONE);
{ DMA_ATTR_ALLOC_SINGLE_PAGES, "ALLOC_SINGLE_PAGES" }, \
{ DMA_ATTR_NO_WARN, "NO_WARN" }, \
{ DMA_ATTR_PRIVILEGED, "PRIVILEGED" }, \
- { DMA_ATTR_MMIO, "MMIO" })
+ { DMA_ATTR_MMIO, "MMIO" }, \
+ { DMA_ATTR_CC_DECRYPTED, "CC_DECRYPTED" })
DECLARE_EVENT_CLASS(dma_map,
TP_PROTO(struct device *dev, phys_addr_t phys_addr, dma_addr_t dma_addr,
diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h
index 62f0d9d0ba02..ae5bc1919e1c 100644
--- a/kernel/dma/direct.h
+++ b/kernel/dma/direct.h
@@ -87,16 +87,24 @@ static inline dma_addr_t dma_direct_map_phys(struct device *dev,
dma_addr_t dma_addr;
if (is_swiotlb_force_bounce(dev)) {
- if (attrs & DMA_ATTR_MMIO)
- return DMA_MAPPING_ERROR;
+ if (!(attrs & DMA_ATTR_CC_DECRYPTED)) {
+ if (attrs & DMA_ATTR_MMIO)
+ return DMA_MAPPING_ERROR;
- return swiotlb_map(dev, phys, size, dir, attrs);
+ return swiotlb_map(dev, phys, size, dir, attrs);
+ }
+ } else if (attrs & DMA_ATTR_CC_DECRYPTED) {
+ return DMA_MAPPING_ERROR;
}
if (attrs & DMA_ATTR_MMIO) {
dma_addr = phys;
if (unlikely(!dma_capable(dev, dma_addr, size, false)))
goto err_overflow;
+ } else if (attrs & DMA_ATTR_CC_DECRYPTED) {
+ dma_addr = phys_to_dma_unencrypted(dev, phys);
+ if (unlikely(!dma_capable(dev, dma_addr, size, false)))
+ goto err_overflow;
} else {
dma_addr = phys_to_dma(dev, phys);
if (unlikely(!dma_capable(dev, dma_addr, size, true)) ||
--
2.51.1
^ permalink raw reply related
* [PATCH 1/5] dma-mapping: avoid random addr value print out on error path
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
In-Reply-To: <20260209153809.250835-1-jiri@resnulli.us>
From: Jiri Pirko <jiri@nvidia.com>
dma_addr is unitialized in dma_direct_map_phys() when swiotlb is forced
and DMA_ATTR_MMIO is set which leads to random value print out in
warning. Fix that by just returning DMA_MAPPING_ERROR.
Fixes: e53d29f957b3 ("dma-mapping: convert dma_direct_*map_page to be phys_addr_t based")
Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
kernel/dma/direct.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h
index da2fadf45bcd..62f0d9d0ba02 100644
--- a/kernel/dma/direct.h
+++ b/kernel/dma/direct.h
@@ -88,7 +88,7 @@ static inline dma_addr_t dma_direct_map_phys(struct device *dev,
if (is_swiotlb_force_bounce(dev)) {
if (attrs & DMA_ATTR_MMIO)
- goto err_overflow;
+ return DMA_MAPPING_ERROR;
return swiotlb_map(dev, phys, size, dir, attrs);
}
--
2.51.1
^ permalink raw reply related
* [PATCH 0/5] dma-buf: heaps: system: add an option to allocate explicitly decrypted memory
From: Jiri Pirko @ 2026-02-09 15:38 UTC (permalink / raw)
To: dri-devel, linaro-mm-sig, iommu, linux-media
Cc: sumit.semwal, benjamin.gaignard, Brian.Starkey, jstultz,
tjmercier, christian.koenig, m.szyprowski, robin.murphy, jgg,
leon, sean.anderson, ptesarik, catalin.marinas, aneesh.kumar,
suzuki.poulose, steven.price, thomas.lendacky, john.allen,
ashish.kalra, suravee.suthikulpanit, linux-coco
From: Jiri Pirko <jiri@nvidia.com>
Confidential computing (CoCo) VMs/guests, such as AMD SEV and Intel TDX,
run with encrypted/protected memory which creates a challenge
for devices that do not support DMA to it (no TDISP support).
For kernel-only DMA operations, swiotlb bounce buffering provides a
transparent solution by copying data through decrypted memory.
However, the only way to get this memory into userspace is via the DMA
API's dma_alloc_pages()/dma_mmap_pages() type interfaces which limits
the use of the memory to a single DMA device, and is incompatible with
pin_user_pages().
These limitations are particularly problematic for the RDMA subsystem
which makes heavy use of pin_user_pages() and expects flexible memory
usage between many different DMA devices.
This patch series enables userspace to explicitly request decrypted
(shared) memory allocations from the dma-buf system heap.
Userspace can mmap this memory and pass the dma-buf fd to other
existing importers such as RDMA or DRM devices to access the
memory. The DMA API is improved to allow the dma heap exporter to DMA
map the shared memory to each importing device.
Jiri Pirko (5):
dma-mapping: avoid random addr value print out on error path
dma-mapping: introduce DMA_ATTR_CC_DECRYPTED for pre-decrypted memory
dma-buf: heaps: use designated initializer for exp_info
dma-buf: heaps: allow heap to specify valid heap flags
dma-buf: heaps: system: add an option to allocate explicitly decrypted
memory
drivers/dma-buf/dma-heap.c | 5 +-
drivers/dma-buf/heaps/cma_heap.c | 7 ++-
drivers/dma-buf/heaps/system_heap.c | 96 ++++++++++++++++++++++++++---
include/linux/dma-heap.h | 3 +
include/linux/dma-mapping.h | 7 +++
include/trace/events/dma.h | 3 +-
include/uapi/linux/dma-heap.h | 12 +++-
kernel/dma/direct.h | 14 ++++-
8 files changed, 128 insertions(+), 19 deletions(-)
--
2.51.1
^ permalink raw reply
* [PATCH v2] KVM: x86: synthesize CPUID bits only if CPU capability is set
From: Carlos López @ 2026-02-09 15:31 UTC (permalink / raw)
To: seanjc, bp, kvm
Cc: linux-coco, jmattson, binbin.wu, Carlos López, Paolo Bonzini,
Thomas Gleixner, Ingo Molnar, Dave Hansen,
maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), H. Peter Anvin,
open list:X86 ARCHITECTURE (32-BIT AND 64-BIT)
KVM incorrectly synthesizes CPUID bits for KVM-only leaves, as the
following branch in kvm_cpu_cap_init() is never taken:
if (leaf < NCAPINTS)
kvm_cpu_caps[leaf] &= kernel_cpu_caps[leaf];
This means that bits set via SYNTHESIZED_F() for KVM-only leaves are
unconditionally set. This for example can cause issues for SEV-SNP
guests running on Family 19h CPUs, as TSA_SQ_NO and TSA_L1_NO are
always enabled by KVM in 80000021[ECX]. When userspace issues a
SNP_LAUNCH_UPDATE command to update the CPUID page for the guest, SNP
firmware will explicitly reject the command if the page sets sets these
bits on vulnerable CPUs.
To fix this, check in SYNTHESIZED_F() that the corresponding X86
capability is set before adding it to to kvm_cpu_cap_features.
Fixes: 31272abd5974 ("KVM: SVM: Advertise TSA CPUID bits to guests")
Link: https://lore.kernel.org/all/20260208164233.30405-1-clopez@suse.de/
Signed-off-by: Carlos López <clopez@suse.de>
---
v2: fix SYNTHESIZED_F() instead of using SCATTERED_F() for TSA bits
arch/x86/kvm/cpuid.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index 88a5426674a1..5f41924987c7 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -770,7 +770,10 @@ do { \
#define SYNTHESIZED_F(name) \
({ \
kvm_cpu_cap_synthesized |= feature_bit(name); \
- F(name); \
+ \
+ BUILD_BUG_ON(X86_FEATURE_##name >= MAX_CPU_FEATURES); \
+ if (boot_cpu_has(X86_FEATURE_##name)) \
+ F(name); \
})
/*
--
2.51.0
^ permalink raw reply related
* Re: [PATCH v2 0/2] x86/virt/tdx: Print TDX module version to dmesg
From: Kiryl Shutsemau @ 2026-02-09 10:46 UTC (permalink / raw)
To: Dave Hansen
Cc: Verma, Vishal L, dave.hansen, kvm@vger.kernel.org,
linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org,
Gao, Chao, Edgecombe, Rick P, Huang, Kai, bp@alien8.de,
mingo@redhat.com, Williams, Dan J, tglx@linutronix.de,
hpa@zytor.com, x86@kernel.org
In-Reply-To: <2235a1cf-7ccf-4134-80b5-8056537c6d33@intel.com>
On Fri, Feb 06, 2026 at 07:10:58AM -0800, Dave Hansen wrote:
> On 2/6/26 03:39, Kiryl Shutsemau wrote:
> >> Hi Kiryl, just wanted to check on the plan for this, I didn't see it
> >> merged in tip.git x86/tdx (or any other tip branch). Were you planning
> >> to take it through x86/tdx? Can I help with anything to move it along?
> > I guess it has to wait after the merge window at this point.
> >
> > Dave, could you queue this after -rc1 is tagged?
>
> Sure.
>
> Is there any other TDX stuff that needs to get picked up at the same
> time that's been languishing?
Nothing on my side.
--
Kiryl Shutsemau / Kirill A. Shutemov
^ permalink raw reply
* Re: [RFC PATCH v5 20/45] KVM: x86/mmu: Allocate/free S-EPT pages using tdx_{alloc,free}_control_page()
From: Huang, Kai @ 2026-02-09 10:41 UTC (permalink / raw)
To: seanjc@google.com, Zhao, Yan Y
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Li, Xiaoyao,
dave.hansen@linux.intel.com, kas@kernel.org,
binbin.wu@linux.intel.com, mingo@redhat.com, pbonzini@redhat.com,
ackerleytng@google.com, linux-kernel@vger.kernel.org,
Yamahata, Isaku, sagis@google.com, tglx@kernel.org,
Edgecombe, Rick P, bp@alien8.de, Annapurve, Vishal,
x86@kernel.org
In-Reply-To: <aYYCOiMvWfSJR1AL@google.com>
>
> Option #2 would be to immediately free the page in tdx_sept_reclaim_private_sp(),
> so that pages that freed via handle_removed_pt() don't defer freeing the S-EPT
> page table (which, IIUC, is safe since the TDX-Module forces TLB flushes and exits).
>
> I really, really don't like this option (if it even works).
>
> diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
> index ae7b9beb3249..4726011ad624 100644
> --- a/arch/x86/kvm/vmx/tdx.c
> +++ b/arch/x86/kvm/vmx/tdx.c
> @@ -2014,7 +2014,15 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
> */
> if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) ||
> tdx_reclaim_page(virt_to_page(sp->external_spt)))
> - sp->external_spt = NULL;
> + goto out;
> +
> + /*
> + * Immediately free the control page, as the TDX subsystem doesn't
> + * support freeing pages from RCU callbacks.
> + */
> + tdx_free_control_page((unsigned long)sp->external_spt);
> +out:
> + sp->external_spt = NULL;
> }
>
> void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
I don't think this is so bad, given we already have a bunch of
is_mirror_sp(sp)
kvm_x86_call(xx_external_spt)(..);
in TDP MMU code?
I suppose this won't make a lot of difference, but does below make you
slightly happier?
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 3181406c5e0b..3588265098a8 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -55,8 +55,7 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm)
static void tdp_mmu_free_sp(struct kvm_mmu_page *sp)
{
- if (sp->external_spt)
- kvm_x86_call(free_external_sp)((unsigned long)sp-
>external_spt);
+ WARN_ON_ONCE(sp->external_spt);
free_page((unsigned long)sp->spt);
kmem_cache_free(mmu_page_header_cache, sp);
}
@@ -457,8 +456,17 @@ static void handle_removed_pt(struct kvm *kvm,
tdp_ptep_t pt, bool shared)
old_spte, FROZEN_SPTE, level, shared);
}
- if (is_mirror_sp(sp))
+ if (is_mirror_sp(sp)) {
kvm_x86_call(reclaim_external_sp)(kvm, base_gfn, sp);
+ /*
+ * Immediately free the control page, as the TDX subsystem
doesn't
+ * support freeing pages from RCU callbacks.
+ */
+ if (sp->external_spt) {
+ kvm_x86_call(free_external_sp)((unsigned long)sp-
>external_spt);
+ sp->external_spt = NULL;
+ }
+ }
call_rcu(&sp->rcu_head, tdp_mmu_free_sp_rcu_callback);
}
^ permalink raw reply related
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Huang, Kai @ 2026-02-09 10:33 UTC (permalink / raw)
To: seanjc@google.com, Edgecombe, Rick P
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Li, Xiaoyao,
Zhao, Yan Y, dave.hansen@linux.intel.com,
linux-kernel@vger.kernel.org, kas@kernel.org,
binbin.wu@linux.intel.com, pbonzini@redhat.com, mingo@redhat.com,
Yamahata, Isaku, ackerleytng@google.com, tglx@kernel.org,
Annapurve, Vishal, sagis@google.com, bp@alien8.de, x86@kernel.org
In-Reply-To: <aYZ2qft-akOYwkOk@google.com>
On Fri, 2026-02-06 at 15:18 -0800, Sean Christopherson wrote:
> On Fri, Feb 06, 2026, Rick P Edgecombe wrote:
> > On Fri, 2026-02-06 at 08:03 -0800, Sean Christopherson wrote:
> > > > If this external cache is for PAMT pages allocation for guest pages only,
> > > > here
> > > > the min count should be 1 instead of PT64_ROOT_MAX_LEVEL?
> > >
> > > Oh! Right. Hmm, with that in mind, it seems like topup_external_cache()
> > > isn't
> > > quite the right interace. It's not at all clear that, unlike the other
> > > caches,
> > > the DPAMT cache isn't tied to the page tables, it's tied to the physical
> > > memory
> > > being mapped into the guest.
> > >
> > > At the very least, it seems like we should drop the @min parameter?
> > >
> > > int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu);
> > >
> > > Though if someone has a name that better captures what the cache is used for,
> > > without bleeding too many details into common x86...
> >
> > From the TDX perspective we have 4 types of pages that are needed to service
> > faults:
> > 1. "Control pages" (i.e. external page tables themselves)
> > 2. Private guest memory pages
> > 3. DPAMT backing pages for control pages
> > 4. DPAMT backing pages for private pages
> >
> > (3) is totally hidden now, but we need a hook to allocate (4). But from core
> > MMU's perspective we hide the existence of DPAMT backing pages. So we don't want
> > to leak that concept.
>
> Heh, there is no way around that. Common KVM needs to know that the cache is
> tied to mapping a page into the guest, otherwise the parameters don't make any
> sense whatsoever. All we can do is minimize the bleeding.
Actually, maybe we can even get rid of the DPAMT cache for the actual
private pages w/o introducing new field to 'kvm_mmu_page':
The point is:
Once we know the PFN and the actual mapping level, we can know whether we
need DPAMT pages for that PFN. If we can know outside of MMU lock, then
we can call tdx_pamt_get(PFN) directly w/o needing the "cache".
In the fault path, we already know the PFN after kvm_mmu_faultin_pfn(),
which is outside of MMU lock.
What we still don't know is the actual mapping level, which is currently
done in kvm_tdp_mmu_map() via kvm_mmu_hugepage_adjust().
However I don't see why we cannot move kvm_mmu_hugepage_adjust() out of it
to, e.g., right after kvm_mmu_faultin_pfn()?
If we can do this, then AFAICT we can just do:
r = kvm_x86_call(prepare_pfn)(vcpu, fault, pfn);
in which we can just call tdx_pamt_get(pfn) based on the mapping level?
Similar can be done for kvm_tdp_mmu_map_private_pfn() which already takes
the 'pfn' as parameter.
For the split path, we obviously can also know the 'pfn' from the huge SPTE.
I kinda do wish we could get rid of the new 'struct tdx_pamt_cache' pool if
possible.
Anything I missed?
^ permalink raw reply
* Re: [RFC PATCH v5 20/45] KVM: x86/mmu: Allocate/free S-EPT pages using tdx_{alloc,free}_control_page()
From: Yan Zhao @ 2026-02-09 9:25 UTC (permalink / raw)
To: Sean Christopherson
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Kiryl Shutsemau, Paolo Bonzini, linux-kernel, linux-coco, kvm,
Kai Huang, Rick Edgecombe, Vishal Annapurve, Ackerley Tng,
Sagi Shahar, Binbin Wu, Xiaoyao Li, Isaku Yamahata
In-Reply-To: <aYYCOiMvWfSJR1AL@google.com>
On Fri, Feb 06, 2026 at 07:01:14AM -0800, Sean Christopherson wrote:
> > (1) alloc
> > tdx_alloc_control_page
> > __tdx_alloc_control_page
> > __tdx_pamt_get
> > spin_lock(&pamt_lock) ==> under process context
> > spin_unlock(&pamt_lock)
> >
> > (2) free
> > tdp_mmu_free_sp_rcu_callback
> > tdp_mmu_free_sp
> > kvm_x86_call(free_external_sp)
> > tdx_free_control_page
> > __tdx_free_control_page
> > __tdx_pamt_put
> > spin_lock(&pamt_lock) ==> under softirq context
> > spin_unlock(&pamt_lock)
> >
> > So, invoking __tdx_pamt_put() in the RCU callback triggers deadlock warning
> > (see the bottom for details).
>
> Hrm. I can think of two options. Option #1 would be to use a raw spinlock and
> disable IRQs:
>
> diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> index 823ec092b4e4..6348085d7dcb 100644
> --- a/arch/x86/virt/vmx/tdx/tdx.c
> +++ b/arch/x86/virt/vmx/tdx/tdx.c
> @@ -2246,7 +2246,7 @@ static u64 tdh_phymem_pamt_remove(u64 pfn, u64 *pamt_pa_array)
> }
>
> /* Serializes adding/removing PAMT memory */
> -static DEFINE_SPINLOCK(pamt_lock);
> +static DEFINE_RAW_SPINLOCK(pamt_lock);
>
> /* Bump PAMT refcount for the given page and allocate PAMT memory if needed */
> int __tdx_pamt_get(u64 pfn, struct tdx_pamt_cache *cache)
> @@ -2272,7 +2272,7 @@ int __tdx_pamt_get(u64 pfn, struct tdx_pamt_cache *cache)
> if (ret)
> goto out_free;
>
> - scoped_guard(spinlock, &pamt_lock) {
> + scoped_guard(raw_spinlock_irqsave, &pamt_lock) {
> /*
> * Lost race to other tdx_pamt_add(). Other task has already allocated
> * PAMT memory for the HPA.
> @@ -2348,7 +2348,7 @@ void __tdx_pamt_put(u64 pfn)
> if (!atomic_dec_and_test(pamt_refcount))
> return;
>
> - scoped_guard(spinlock, &pamt_lock) {
> + scoped_guard(raw_spinlock_irqsave, &pamt_lock) {
> /* Lost race with tdx_pamt_get(). */
> if (atomic_read(pamt_refcount))
> return;
This option can get rid of the warning.
However, given the pamt_lock is a global lock, which may be acquired even in the
softirq context, not sure if this irq disabled version is good.
For your reference, I measured some test data by concurrently launching and
destroying 4 TDs for 3 rounds:
t0 ---------------------
scoped_guard(spinlock, &pamt_lock) { |->T1=t1-t0 |
t1 ---------- |
... |
t2 ---------- |->T3=t4-t0
tdh_phymem_pamt_add/remove() |->T2=t3-t2 |
t3 ---------- |
... |
t4 ---------------------
}
(1) for __tdx_pamt_get()
avg us min us max us
------|---------------------------
T1 | 4 0 69
T2 | 4 2 18
T3 | 10 3 83
(2) for__tdx_pamt_put()
avg us min us max us
------|---------------------------
T1 | 0 0 5
T2 | 2 1 11
T3 | 3 2 15
> Option #2 would be to immediately free the page in tdx_sept_reclaim_private_sp(),
> so that pages that freed via handle_removed_pt() don't defer freeing the S-EPT
> page table (which, IIUC, is safe since the TDX-Module forces TLB flushes and exits).
>
> I really, really don't like this option (if it even works).
I don't like its asymmetry with tdx_sept_link_private_spt().
However, do you think it would be good to have the PAMT pages of the sept pages
allocated from (*topup_private_mapping_cache) [1]?
private_mapping could also include non-leaf mappings.
So, we could invoke tdx_pamt_get() in tdx_sept_link_private_spt() for symmetry.
[1] https://lore.kernel.org/all/aYZ2qft-akOYwkOk@google.com/
> diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
> index ae7b9beb3249..4726011ad624 100644
> --- a/arch/x86/kvm/vmx/tdx.c
> +++ b/arch/x86/kvm/vmx/tdx.c
> @@ -2014,7 +2014,15 @@ static void tdx_sept_reclaim_private_sp(struct kvm *kvm, gfn_t gfn,
> */
> if (KVM_BUG_ON(is_hkid_assigned(to_kvm_tdx(kvm)), kvm) ||
> tdx_reclaim_page(virt_to_page(sp->external_spt)))
> - sp->external_spt = NULL;
> + goto out;
> +
> + /*
> + * Immediately free the control page, as the TDX subsystem doesn't
> + * support freeing pages from RCU callbacks.
> + */
> + tdx_free_control_page((unsigned long)sp->external_spt);
> +out:
> + sp->external_spt = NULL;
> }
>
> void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
> --
^ permalink raw reply
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Edgecombe, Rick P @ 2026-02-06 23:19 UTC (permalink / raw)
To: seanjc@google.com
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Huang, Kai,
Li, Xiaoyao, Zhao, Yan Y, dave.hansen@linux.intel.com,
linux-kernel@vger.kernel.org, kas@kernel.org,
binbin.wu@linux.intel.com, pbonzini@redhat.com, mingo@redhat.com,
Yamahata, Isaku, ackerleytng@google.com, tglx@kernel.org,
sagis@google.com, bp@alien8.de, Annapurve, Vishal, x86@kernel.org
In-Reply-To: <aYZ2qft-akOYwkOk@google.com>
On Fri, 2026-02-06 at 15:18 -0800, Sean Christopherson wrote:
> How about?
>
> (*topup_private_mapping_cache)
>
> Because it's not just "private memory" it's specifically the mapping. E.g. for
> the hugepage split case, the primary memory is already assigned and mapped into
> the guest, but a topup is still needed because KVM is creating a new/different
> mapping.
Sure.
^ permalink raw reply
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Sean Christopherson @ 2026-02-06 23:18 UTC (permalink / raw)
To: Rick P Edgecombe
Cc: Yan Y Zhao, kvm@vger.kernel.org, linux-coco@lists.linux.dev,
Xiaoyao Li, Kai Huang, dave.hansen@linux.intel.com,
kas@kernel.org, binbin.wu@linux.intel.com, mingo@redhat.com,
pbonzini@redhat.com, ackerleytng@google.com,
linux-kernel@vger.kernel.org, Isaku Yamahata, sagis@google.com,
tglx@kernel.org, bp@alien8.de, Vishal Annapurve, x86@kernel.org
In-Reply-To: <b3ad6d9cce83681f548b35881ebad0c5bb4fed23.camel@intel.com>
On Fri, Feb 06, 2026, Rick P Edgecombe wrote:
> On Fri, 2026-02-06 at 08:03 -0800, Sean Christopherson wrote:
> > > If this external cache is for PAMT pages allocation for guest pages only,
> > > here
> > > the min count should be 1 instead of PT64_ROOT_MAX_LEVEL?
> >
> > Oh! Right. Hmm, with that in mind, it seems like topup_external_cache()
> > isn't
> > quite the right interace. It's not at all clear that, unlike the other
> > caches,
> > the DPAMT cache isn't tied to the page tables, it's tied to the physical
> > memory
> > being mapped into the guest.
> >
> > At the very least, it seems like we should drop the @min parameter?
> >
> > int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu);
> >
> > Though if someone has a name that better captures what the cache is used for,
> > without bleeding too many details into common x86...
>
> From the TDX perspective we have 4 types of pages that are needed to service
> faults:
> 1. "Control pages" (i.e. external page tables themselves)
> 2. Private guest memory pages
> 3. DPAMT backing pages for control pages
> 4. DPAMT backing pages for private pages
>
> (3) is totally hidden now, but we need a hook to allocate (4). But from core
> MMU's perspective we hide the existence of DPAMT backing pages. So we don't want
> to leak that concept.
Heh, there is no way around that. Common KVM needs to know that the cache is
tied to mapping a page into the guest, otherwise the parameters don't make any
sense whatsoever. All we can do is minimize the bleeding.
> The page we need is kind of like something to "prepare" the private page before
> installing it. It actually isn't that related to the mirror/external concept. So
> if we separate it from "external" and make it about installing private guest
> memory, it fits better conceptually I think. But it could be a bit confusing for
> other types of VMs who have to trace to see if anything special is happening
> inside the op for their private memory. In that case it could be:
>
> (*topup_private_mem_prepare_cache)(struct kvm_vcpu *vcpu)
topup + prepare is redundant and confusing.
> The core MMU doesn't know about DPAMT backing pages, but it does know about the
> set_external_spte op that consumes this cache. So how about the slightly
> misleading:
>
> (*topup_set_external_spte_cache)(struct kvm_vcpu *vcpu)
I really, really, want to avoid "SPTE", because the cache isn't for the SPTE in
any way, it's for the memory that's _pointed_ at by the SPTE. And the confusion
is exactly what prompted this thread: I forgot that it's not every SPTE in the
chain that needs DPAMT backing, it's only the page that's being mapped into the
guest.
How about?
(*topup_private_mapping_cache)
Because it's not just "private memory" it's specifically the mapping. E.g. for
the hugepage split case, the primary memory is already assigned and mapped into
the guest, but a topup is still needed because KVM is creating a new/different
mapping.
> It is easier for other VM types to ignore, and not that semantically wrong from
> what is happening on the TDX side.
^ permalink raw reply
* [PATCH v3 2/2] KVM: SEV: Restrict userspace return codes for KVM_HC_MAP_GPA_RANGE
From: Sagi Shahar @ 2026-02-06 22:28 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Dave Hansen, Kiryl Shutsemau,
Rick Edgecombe
Cc: Thomas Gleixner, Borislav Petkov, H. Peter Anvin, Michael Roth,
Tom Lendacky, x86, kvm, linux-kernel, linux-coco, Sagi Shahar
In-Reply-To: <20260206222829.3758171-1-sagis@google.com>
To align with the updated TDX api that allows userspace to request
that guests retry MAP_GPA operations, make sure that userspace is only
returning EINVAL or EAGAIN as possible error codes.
Signed-off-by: Sagi Shahar <sagis@google.com>
---
arch/x86/kvm/svm/sev.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index f59c65abe3cf..5f78e4c3eb5d 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3722,9 +3722,13 @@ static int snp_rmptable_psmash(kvm_pfn_t pfn)
static int snp_complete_psc_msr(struct kvm_vcpu *vcpu)
{
+ u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret);
struct vcpu_svm *svm = to_svm(vcpu);
- if (vcpu->run->hypercall.ret)
+ if (!kvm_is_valid_map_gpa_range_ret(hypercall_ret))
+ return -EINVAL;
+
+ if (hypercall_ret)
set_ghcb_msr(svm, GHCB_MSR_PSC_RESP_ERROR);
else
set_ghcb_msr(svm, GHCB_MSR_PSC_RESP);
@@ -3815,10 +3819,14 @@ static void __snp_complete_one_psc(struct vcpu_svm *svm)
static int snp_complete_one_psc(struct kvm_vcpu *vcpu)
{
+ u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret);
struct vcpu_svm *svm = to_svm(vcpu);
struct psc_buffer *psc = svm->sev_es.ghcb_sa;
- if (vcpu->run->hypercall.ret) {
+ if (!kvm_is_valid_map_gpa_range_ret(hypercall_ret))
+ return -EINVAL;
+
+ if (hypercall_ret) {
snp_complete_psc(svm, VMGEXIT_PSC_ERROR_GENERIC);
return 1; /* resume guest */
}
--
2.53.0.rc2.204.g2597b5adb4-goog
^ permalink raw reply related
* [PATCH v3 1/2] KVM: TDX: Allow userspace to return errors to guest for MAPGPA
From: Sagi Shahar @ 2026-02-06 22:28 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Dave Hansen, Kiryl Shutsemau,
Rick Edgecombe
Cc: Thomas Gleixner, Borislav Petkov, H. Peter Anvin, Michael Roth,
Tom Lendacky, x86, kvm, linux-kernel, linux-coco,
Vishal Annapurve, Sagi Shahar
In-Reply-To: <20260206222829.3758171-1-sagis@google.com>
From: Vishal Annapurve <vannapurve@google.com>
MAPGPA request from TDX VMs gets split into chunks by KVM using a loop
of userspace exits until the complete range is handled.
In some cases userspace VMM might decide to break the MAPGPA operation
and continue it later. For example: in the case of intrahost migration
userspace might decide to continue the MAPGPA operation after the
migration is completed.
Allow userspace to signal to TDX guests that the MAPGPA operation should
be retried the next time the guest is scheduled.
This is potentially a breaking change since if userspace sets
hypercall.ret to a value other than EBUSY or EINVAL an EINVAL error code
will be returned to userspace. As of now QEMU never sets hypercall.ret
to a non-zero value after handling KVM_EXIT_HYPERCALL so this change
should be safe.
Signed-off-by: Vishal Annapurve <vannapurve@google.com>
Co-developed-by: Sagi Shahar <sagis@google.com>
Signed-off-by: Sagi Shahar <sagis@google.com>
---
Documentation/virt/kvm/api.rst | 3 +++
arch/x86/kvm/vmx/tdx.c | 15 +++++++++++++--
arch/x86/kvm/x86.h | 6 ++++++
3 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 01a3abef8abb..9978cd9d897e 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -8679,6 +8679,9 @@ block sizes is exposed in KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES as a
This capability, if enabled, will cause KVM to exit to userspace
with KVM_EXIT_HYPERCALL exit reason to process some hypercalls.
+Userspace may fail the hypercall by setting hypercall.ret to EINVAL
+or may request the hypercall to be retried the next time the guest run
+by setting hypercall.ret to EAGAIN.
Calling KVM_CHECK_EXTENSION for this capability will return a bitmask
of hypercalls that can be configured to exit to userspace.
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 2d7a4d52ccfb..056a44b9d78b 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1186,10 +1186,21 @@ static void __tdx_map_gpa(struct vcpu_tdx *tdx);
static int tdx_complete_vmcall_map_gpa(struct kvm_vcpu *vcpu)
{
+ u64 hypercall_ret = READ_ONCE(vcpu->run->hypercall.ret);
struct vcpu_tdx *tdx = to_tdx(vcpu);
- if (vcpu->run->hypercall.ret) {
- tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
+ if (hypercall_ret) {
+ if (hypercall_ret == EAGAIN) {
+ tdvmcall_set_return_code(vcpu, TDVMCALL_STATUS_RETRY);
+ } else if (vcpu->run->hypercall.ret == EINVAL) {
+ tdvmcall_set_return_code(
+ vcpu, TDVMCALL_STATUS_INVALID_OPERAND);
+ } else {
+ WARN_ON_ONCE(
+ kvm_is_valid_map_gpa_range_ret(hypercall_ret));
+ return -EINVAL;
+ }
+
tdx->vp_enter_args.r11 = tdx->map_gpa_next;
return 1;
}
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index fdab0ad49098..3d464d12423a 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -706,6 +706,12 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,
unsigned int port, void *data, unsigned int count,
int in);
+static inline bool kvm_is_valid_map_gpa_range_ret(u64 hypercall_ret)
+{
+ return !hypercall_ret || hypercall_ret == EINVAL ||
+ hypercall_ret == EAGAIN;
+}
+
static inline bool user_exit_on_hypercall(struct kvm *kvm, unsigned long hc_nr)
{
return kvm->arch.hypercall_exit_enabled & BIT(hc_nr);
--
2.53.0.rc2.204.g2597b5adb4-goog
^ permalink raw reply related
* [PATCH v3 0/2] Extend KVM_HC_MAP_GPA_RANGE api to allow retry
From: Sagi Shahar @ 2026-02-06 22:28 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Dave Hansen, Kiryl Shutsemau,
Rick Edgecombe
Cc: Thomas Gleixner, Borislav Petkov, H. Peter Anvin, Michael Roth,
Tom Lendacky, x86, kvm, linux-kernel, linux-coco, Sagi Shahar
In some cases, userspace might decide to split MAP_GPA requests and
retry them the next time the guest runs. One common case is MAP_GPA
requests received right before intrahost migration when userspace
might decide to complete the request after the migration is complete
to reduce blackout time.
This is v3 of the series, v1[1] and v2[2] were posted as standalone
patches.
Changes from v2:
* Rebased on top of v6.19-rc8.
* Updated documentation.
* Restricted SNP error codes to match TDX restrictions.
[1] https://lore.kernel.org/kvm/20260114003015.1386066-1-sagis@google.com/
[2] https://lore.kernel.org/lkml/20260115225238.2837449-1-sagis@google.com/
Sagi Shahar (1):
KVM: SEV: Restrict userspace return codes for KVM_HC_MAP_GPA_RANGE
Vishal Annapurve (1):
KVM: TDX: Allow userspace to return errors to guest for MAPGPA
Documentation/virt/kvm/api.rst | 3 +++
arch/x86/kvm/svm/sev.c | 12 ++++++++++--
arch/x86/kvm/vmx/tdx.c | 15 +++++++++++++--
arch/x86/kvm/x86.h | 6 ++++++
4 files changed, 32 insertions(+), 4 deletions(-)
--
2.53.0.rc2.204.g2597b5adb4-goog
^ permalink raw reply
* [PATCH] crypto: ccp - allow callers to use HV-Fixed page API when SEV is disabled
From: Ashish Kalra @ 2026-02-06 21:26 UTC (permalink / raw)
To: thomas.lendacky, john.allen, herbert, davem, bp
Cc: linux-crypto, linux-kernel, linux-coco
From: Ashish Kalra <ashish.kalra@amd.com>
When SEV is disabled, the HV-Fixed page allocation call fails, which in
turn causes SFS initialization to fail.
Fix the HV-Fixed API so callers (for example, SFS) can use it even when
SEV is disabled by performing normal page allocation and freeing.
Fixes: e09701dcdd9c ("crypto: ccp - Add new HV-Fixed page allocation/free API")
Cc: stable@vger.kernel.org
Signed-off-by: Ashish Kalra <ashish.kalra@amd.com>
---
drivers/crypto/ccp/sev-dev.c | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c
index 1cdadddb744e..0d90b5f6a454 100644
--- a/drivers/crypto/ccp/sev-dev.c
+++ b/drivers/crypto/ccp/sev-dev.c
@@ -1105,15 +1105,12 @@ struct page *snp_alloc_hv_fixed_pages(unsigned int num_2mb_pages)
{
struct psp_device *psp_master = psp_get_master_device();
struct snp_hv_fixed_pages_entry *entry;
- struct sev_device *sev;
unsigned int order;
struct page *page;
- if (!psp_master || !psp_master->sev_data)
+ if (!psp_master)
return NULL;
- sev = psp_master->sev_data;
-
order = get_order(PMD_SIZE * num_2mb_pages);
/*
@@ -1126,7 +1123,8 @@ struct page *snp_alloc_hv_fixed_pages(unsigned int num_2mb_pages)
* This API uses SNP_INIT_EX to transition allocated pages to HV_Fixed
* page state, fail if SNP is already initialized.
*/
- if (sev->snp_initialized)
+ if (psp_master->sev_data &&
+ ((struct sev_device *)psp_master->sev_data)->snp_initialized)
return NULL;
/* Re-use freed pages that match the request */
@@ -1162,7 +1160,7 @@ void snp_free_hv_fixed_pages(struct page *page)
struct psp_device *psp_master = psp_get_master_device();
struct snp_hv_fixed_pages_entry *entry, *nentry;
- if (!psp_master || !psp_master->sev_data)
+ if (!psp_master)
return;
/*
--
2.34.1
^ permalink raw reply related
* Re: [RFC PATCH v5 22/45] KVM: TDX: Get/put PAMT pages when (un)mapping private memory
From: Edgecombe, Rick P @ 2026-02-06 19:27 UTC (permalink / raw)
To: seanjc@google.com, Zhao, Yan Y
Cc: kvm@vger.kernel.org, linux-coco@lists.linux.dev, Li, Xiaoyao,
Huang, Kai, dave.hansen@linux.intel.com, kas@kernel.org,
binbin.wu@linux.intel.com, mingo@redhat.com, pbonzini@redhat.com,
ackerleytng@google.com, linux-kernel@vger.kernel.org,
Yamahata, Isaku, sagis@google.com, tglx@kernel.org, bp@alien8.de,
Annapurve, Vishal, x86@kernel.org
In-Reply-To: <aYYQ7Vx95ZrsqwCv@google.com>
On Fri, 2026-02-06 at 08:03 -0800, Sean Christopherson wrote:
> > If this external cache is for PAMT pages allocation for guest pages only,
> > here
> > the min count should be 1 instead of PT64_ROOT_MAX_LEVEL?
>
> Oh! Right. Hmm, with that in mind, it seems like topup_external_cache()
> isn't
> quite the right interace. It's not at all clear that, unlike the other
> caches,
> the DPAMT cache isn't tied to the page tables, it's tied to the physical
> memory
> being mapped into the guest.
>
> At the very least, it seems like we should drop the @min parameter?
>
> int (*topup_external_cache)(struct kvm *kvm, struct kvm_vcpu *vcpu);
>
> Though if someone has a name that better captures what the cache is used for,
> without bleeding too many details into common x86...
From the TDX perspective we have 4 types of pages that are needed to service
faults:
1. "Control pages" (i.e. external page tables themselves)
2. Private guest memory pages
3. DPAMT backing pages for control pages
4. DPAMT backing pages for private pages
(3) is totally hidden now, but we need a hook to allocate (4). But from core
MMU's perspective we hide the existence of DPAMT backing pages. So we don't want
to leak that concept.
The page we need is kind of like something to "prepare" the private page before
installing it. It actually isn't that related to the mirror/external concept. So
if we separate it from "external" and make it about installing private guest
memory, it fits better conceptually I think. But it could be a bit confusing for
other types of VMs who have to trace to see if anything special is happening
inside the op for their private memory. In that case it could be:
(*topup_private_mem_prepare_cache)(struct kvm_vcpu *vcpu)
The core MMU doesn't know about DPAMT backing pages, but it does know about the
set_external_spte op that consumes this cache. So how about the slightly
misleading:
(*topup_set_external_spte_cache)(struct kvm_vcpu *vcpu)
It is easier for other VM types to ignore, and not that semantically wrong from
what is happening on the TDX side.
^ permalink raw reply
* Re: [PATCH v2 0/2] x86/virt/tdx: Print TDX module version to dmesg
From: Edgecombe, Rick P @ 2026-02-06 18:35 UTC (permalink / raw)
To: Li, Xiaoyao, Hansen, Dave, kas@kernel.org, Verma, Vishal L,
dave.hansen@linux.intel.com
Cc: Gao, Chao, x86@kernel.org, bp@alien8.de, Huang, Kai,
hpa@zytor.com, mingo@redhat.com, Williams, Dan J,
tglx@linutronix.de, kvm@vger.kernel.org,
linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org
In-Reply-To: <2235a1cf-7ccf-4134-80b5-8056537c6d33@intel.com>
On Fri, 2026-02-06 at 07:10 -0800, Dave Hansen wrote:
> Is there any other TDX stuff that needs to get picked up at the same
> time that's been languishing?
Xiaoyao is going to send a rebase of this after RC1:
https://lore.kernel.org/kvm/20250715091312.563773-1-xiaoyao.li@intel.com/
It has pretty wide agreement, and ack's from the KVM side. Also, we have
internal branches that are carrying forms of it. So if we merge it now we can
have less dependencies later.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox