[PATCH v4 06/17] KVM: x86/mmu: Allocate DPAMT pages for vCPU-induced page split

[Yan Zhao <yan.y.zhao@intel.com>]

From: Sean Christopherson <seanjc@google.com>

Extend the TDP MMU to allocate Dynamic PAMT backing pages (DPAMT pages) for
vCPU-induced huge page splits in mirror roots when DPAMT is enabled.

Leverage the .topup_external_cache() interface to topup the DPAMT cache
when allocating a new child page table for splitting. The DPAMT cache is
currently a per-vCPU thread-local list. When a vCPU-induced page split
occurs, DPAMT pages can be drawn locklessly from the list.

Pass min_nr_spts as 1 to .topup_external_cache(), indicating there's one
new S-EPT page table page. So, tdx_topup_external_pamt_cache() will
allocate DPAMT page pairs for both the newly added S-EPT page table page
and the demoted guest private page.

tdp_mmu_alloc_sp_for_split() is currently not reachable from a non-vCPU
context for mirror roots, since dirty page tracking is not yet allowed on
mirror roots. So, simply add a WARN if tdx_topup_external_pamt_cache() is
invoked under a non-vCPU context.

Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yan Zhao <yan.y.zhao@intel.com>
---
v4: new patch.
---
 arch/x86/kvm/mmu/tdp_mmu.c | 24 +++++++++++++++---------
 arch/x86/kvm/vmx/tdx.c     |  3 +++
 2 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index f3311317a63a..472419963a19 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1475,21 +1475,27 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool is_mirror_sp)
 		return NULL;
 
 	sp->spt = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
-	if (!sp->spt) {
-		kmem_cache_free(mmu_page_header_cache, sp);
-		return NULL;
-	}
+	if (!sp->spt)
+		goto err_spt;
 
 	if (is_mirror_sp) {
 		sp->external_spt = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
-		if (!sp->external_spt) {
-			free_page((unsigned long)sp->spt);
-			kmem_cache_free(mmu_page_header_cache, sp);
-			return NULL;
-		}
+		if (!sp->external_spt)
+			goto err_external_spt;
+
+		if (kvm_x86_call(topup_external_cache)(kvm_get_running_vcpu(), 1))
+			goto err_external_split;
 	}
 
 	return sp;
+
+err_external_split:
+	free_page((unsigned long)sp->external_spt);
+err_external_spt:
+	free_page((unsigned long)sp->spt);
+err_spt:
+	kmem_cache_free(mmu_page_header_cache, sp);
+	return NULL;
 }
 
 /* Note, the caller is responsible for initializing @sp. */
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index 11792a490330..3dcddf1b48c5 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1630,6 +1630,9 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)
 
 static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr_spts)
 {
+	if (WARN_ON_ONCE(!vcpu))
+		return -EIO;
+
 	/*
 	 * Minus one page to exclude the root SPT, but plus one page for a
 	 * possible 4KB private mapping.
-- 
2.43.2