From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>
To: "linux-coco@lists.linux.dev" <linux-coco@lists.linux.dev>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"yilun.xu@linux.intel.com" <yilun.xu@linux.intel.com>,
"x86@kernel.org" <x86@kernel.org>
Cc: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"Li, Xiaoyao" <xiaoyao.li@intel.com>,
"Hansen, Dave" <dave.hansen@intel.com>,
"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
"Hunter, Adrian" <adrian.hunter@intel.com>,
"tony.lindgren@linux.intel.com" <tony.lindgren@linux.intel.com>,
"kas@kernel.org" <kas@kernel.org>,
"Xu, Yilun" <yilun.xu@intel.com>,
"artem.bityutskiy@linux.intel.com"
<artem.bityutskiy@linux.intel.com>,
"nik.borisov@suse.com" <nik.borisov@suse.com>,
"Mehta, Sohil" <sohil.mehta@intel.com>,
"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
"Gao, Chao" <chao.gao@intel.com>,
"Fang, Peter" <peter.fang@intel.com>,
"Maloor, Kishen" <kishen.maloor@intel.com>
Subject: Re: [PATCH v5 3/6] x86/virt/tdx: Add extra memory to TDX module for the extensions
Date: Wed, 18 Nov 2026 22:18:39 +0000 [thread overview]
Message-ID: <f9c89d6da82cd737bcb7d72eb6ca5f5bf121be2b.camel@intel.com> (raw)
In-Reply-To: <20261111-tdx-module-ext-v5-3-c74f764abc3d@linux.intel.com>
On Wed, 2026-11-11 at 11:48 +0800, Xu Yilun wrote:
> TDX module extensions need memory for their execution environment to
> serve SEAMCALL leafs. The TDX architecture implements the extensions
> in such a way that they use the memory outside of SEAM range, so the
> kernel should add the memory upfront at initialization time.
>
> Introduce a new memory adding process backed by a new SEAMCALL leaf
> TDH.EXT.MEM.ADD. The kernel queries TDX module how much memory needed,
> allocates it, adds it to the module, and never gets it back. Note that
> the process requires Dynamic PAMT backing.
The process requires it, or the memory does?
> A subsequent patch adds the
> support. This doesn't break bisectability cause the extensions can't be
^because?
> enabled yet without a feature that uses them.
Not clear whether the TDX module needs a new feature or the kernel does.
>
> The TDX module accepts the memory in the form of an HPA array. This
> array is passed via a single 64-bit SEAMCALL leaf parameter, which
> encodes two values: the PFN of the container page holding the array, and
> the number of entries in the array. Create a helper to encode this
> format and name it after the TDX module term: HPA_LIST_INFO.
>
> TDX module extensions consume tens of megabytes memory that will never
> be returned to the host. Use contiguous page allocation to isolate these
> large blocks entirely. This is a simple way to avoid permanent memory
> fragmentation: requiring the full size to be contiguous is more
> expensive than needed but should be good during the boot time. Print the
> allocation amount on TDX module extensions initialization for
> visibility.
>
> The TDX module provides a metadata field "required_pages" to indicate
> the required memory size. This metadata is meaningful only after the
> add-on feature configuration. Don't read and cache the metadata in
> tdx_sysinfo. Instead read it right before allocating memory for the
> extensions, when it is guaranteed to be valid.
>
> The original changelog was written by hand. AI was used to help correct
> grammatical errors and polish individual sentences for better
> readability.
>
> Signed-off-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
> ---
> An alternative solution is to use a loop that gives memory on a memory
> error code - TDX_EXT_MEMORY_POOL_REQUIRED, add one page per iteration
> until TDH.EXT.INIT succeeds. Something like:
>
> do {
> ret = tdh_sys_init();
> if (ret == TDX_EXT_MEMORY_POOL_REQUIRED)
> tdh_ext_mem_add(); //single page
> } while (ret == TDX_EXT_MEMORY_POOL_REQUIRED);
>
> This approach is slightly simpler as we don't have to query the module
> for the total memory, no memory pre-allocation or segmentation math.
> But allocating a single 4K page per iteration may cause permanent memory
> fragmentation.
>
> v5:
> - Add the DPAMT & bisectability note
> - Shorten the "required_pages field" paragraph a little bit
> - Collect Reviewed-by tags
> - Better clarify the AI usage
>
> v4:
> - Rename memory_pool_required_pages to required_pages (Rick)
> - Rename local var sysinfo_ext to ext
> - Collect Reviewed-by tags
>
> v3:
> - Merge remaining pre-check operations from the previous patch (Rick)
> - Don't save the metadata in tdx_sysinfo (Rick)
> - Tweak the code comment for memory_pool_required_pages == 0 (Chao)
> - Re-phrase the code comment for struct tdx_hpa_list (Tony)
> - Put the error handling comments on top of memory add loop, to avoid
> moving them around when DPAMT support is added in the following
> patch.
> - Print memory size before adding memory; it won't be freed afterward,
> whether the operation succeeds or fails.
> - Collect Reviewed-by tags
>
> v2:
> - Add code comments for container page allocation (Kiryl)
> - Adjust comments/changelog for contiguous allocation of extensions
> memory (Kiryl)
> - s/PFN array/HPA array in commit log (Kiryl)
> - State the alternative solution (Rick)
> - Add code comment for struct tdx_hpa_list
>
> v1:
> - Fix return value for SEAMCALL wrappers (Chao)
> - Print SEAMCALL error code for SEAMCALL wrappers (Xiaoyao)
> - Rename local vars to make the ext memory adding loop clear (Rick)
> - Remove input parameters for tdx_ext_mem_setup() (Kevin)
> - Add a Macro for tdh_hpa_list size.
> - Change the SEAMALL wrapper parameter type,
> struct page *hpa_list => struct tdx_hpa_list *hpa_list
> - changelog & code comments
> ---
> arch/x86/include/asm/tdx.h | 1 +
> arch/x86/include/asm/tdx_global_metadata.h | 4 +
> arch/x86/virt/vmx/tdx/tdx.h | 1 +
> arch/x86/virt/vmx/tdx/tdx.c | 138
> ++++++++++++++++++++++++++++
> arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 14 +++
> 5 files changed, 158 insertions(+)
>
> diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
> index 1fcfaed515fe..89e7628796b0 100644
> --- a/arch/x86/include/asm/tdx.h
> +++ b/arch/x86/include/asm/tdx.h
> @@ -37,6 +37,7 @@
> #define TDX_FEATURES0_TD_PRESERVING BIT_ULL(1)
> #define TDX_FEATURES0_NO_RBP_MOD BIT_ULL(18)
> #define TDX_FEATURES0_DYNAMIC_PAMT BIT_ULL(36)
> +#define TDX_FEATURES0_EXT BIT_ULL(39)
>
> #ifndef __ASSEMBLER__
>
> diff --git a/arch/x86/include/asm/tdx_global_metadata.h
> b/arch/x86/include/asm/tdx_global_metadata.h
> index 8a3cc1a2a41e..f18e63b671c2 100644
> --- a/arch/x86/include/asm/tdx_global_metadata.h
> +++ b/arch/x86/include/asm/tdx_global_metadata.h
> @@ -47,6 +47,10 @@ struct tdx_sys_info_handoff {
> u16 module_hv;
> };
>
> +struct tdx_sys_info_ext {
> + u32 required_pages;
> +};
> +
> struct tdx_sys_info {
> struct tdx_sys_info_version version;
> struct tdx_sys_info_features features;
> diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h
> index db209541d3cd..2adc5d31d258 100644
> --- a/arch/x86/virt/vmx/tdx/tdx.h
> +++ b/arch/x86/virt/vmx/tdx/tdx.h
> @@ -50,6 +50,7 @@
> #define TDH_SYS_UPDATE 53
> #define TDH_PHYMEM_PAMT_ADD 58
> #define TDH_PHYMEM_PAMT_REMOVE 59
> +#define TDH_EXT_MEM_ADD 61
> #define TDH_SYS_DISABLE 69
>
> /* TDX page types */
> diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> index 3bfe78b4b538..865d8089d343 100644
> --- a/arch/x86/virt/vmx/tdx/tdx.c
> +++ b/arch/x86/virt/vmx/tdx/tdx.c
> @@ -1210,6 +1210,140 @@ static __init int init_tdmrs(struct tdmr_info_list
> *tdmr_list)
> return 0;
> }
>
> +#define TDX_HPA_LIST_MAX_NR_PAGES (PAGE_SIZE / sizeof(u64))
> +
> +/*
> + * TDX module in-memory ABI to specify the memory being added to the TDX
> + * module. An array of HPAs, each element points to a 4K page.
> + */
> +struct tdx_hpa_list {
> + u64 phys[TDX_HPA_LIST_MAX_NR_PAGES];
> +};
> +
> +static_assert(sizeof(struct tdx_hpa_list) == PAGE_SIZE);
> +
> +#define HPA_LIST_INFO_FIRST_ENTRY GENMASK_U64(11, 3)
> +#define HPA_LIST_INFO_PFN GENMASK_U64(51, 12)
> +#define HPA_LIST_INFO_LAST_ENTRY GENMASK_U64(63, 55)
> +
> +static __init u64 to_hpa_list_info(struct tdx_hpa_list *hpa_list,
> + unsigned int nr_pages)
> +{
> + return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
> + FIELD_PREP(HPA_LIST_INFO_PFN, PFN_DOWN(__pa(hpa_list))) |
> + FIELD_PREP(HPA_LIST_INFO_LAST_ENTRY, nr_pages - 1);
> +}
> +
> +static __init int tdx_ext_mem_add(struct tdx_hpa_list *hpa_list,
> + unsigned int nr_pages)
> +{
> + struct tdx_module_args args = {
> + .rcx = to_hpa_list_info(hpa_list, nr_pages),
> + };
> + u64 ret;
> +
> + do {
> + /*
> + * The TDX module overwrites RCX to track progress when this
> + * SEAMCALL leaf is interrupted. Use seamcall_ret() to save
> and
> + * pass the updated value back on retry.
> + */
> + ret = seamcall_ret(TDH_EXT_MEM_ADD, &args);
> + } while (ret == TDX_INTERRUPTED_RESUMABLE);
> +
> + if (ret != TDX_SUCCESS) {
> + pr_err("TDH.EXT.MEM.ADD failed: 0x%016llx\n", ret);
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +
> +static __init int tdx_ext_mem_setup(void)
> +{
> + struct tdx_hpa_list *hpa_list;
> + struct tdx_sys_info_ext ext;
> + unsigned int added_pages;
> + struct page *page;
> + int ret;
> +
> + ret = get_tdx_sys_info_ext(&ext);
> + if (ret)
> + return ret;
> +
> + /*
> + * Skip the memory setup if no memory is required. This may happen
> when
> + * no add-on features requiring TDX module extensions are configured
> + * via TDH.SYS.CONFIG.
> + */
> + if (!ext.required_pages)
> + return 0;
> +
> + /*
> + * Allocate the container page for the HPA_LIST. tdx_hpa_list is
> + * guaranteed to be page-sized by static_assert(), so kzalloc()
> + * guarantees the page alignment.
> + */
> + hpa_list = kzalloc_obj(*hpa_list);
> + if (!hpa_list)
> + return -ENOMEM;
> +
> + /*
> + * Memory for TDX module extensions is never reclaimed and can be
> tens
> + * of megabytes. Allocating a physically contiguous chunk is a simple
> + * way to avoid permanent memory fragmentation: requiring the full
> size
> + * to be contiguous is more expensive than needed but should be good
> + * during the boot time.
> + */
> + page = alloc_contig_pages(ext.required_pages, GFP_KERNEL,
> + numa_mem_id(), &node_online_map);
> + if (!page) {
> + ret = -ENOMEM;
> + goto out_free_hpa_list;
> + }
> +
> + /* Print the amount so users know the cost. */
> + pr_info("%lu KB allocated for TDX module extensions\n",
> + ext.required_pages * PAGE_SIZE / 1024);
I wonder if this line is absolutely required for a minimal set?
> +
> + /*
> + * Following TDX operations shouldn't fail, and if they do, things
> are
> + * broken enough that complex error handling isn't worth it.
> + * Intentionally leak all pages on failure, including un-added pages.
> + */
> + added_pages = 0;
> + while (added_pages < ext.required_pages) {
> + unsigned int chunk_pages = min(ext.required_pages -
> added_pages,
> + TDX_HPA_LIST_MAX_NR_PAGES);
> + struct page *chunk = page + added_pages;
> + unsigned int i;
> +
> + for (i = 0; i < chunk_pages; i++)
> + hpa_list->phys[i] = page_to_phys(chunk + i);
> +
> + ret = tdx_ext_mem_add(hpa_list, chunk_pages);
> + if (ret) {
> + WARN(1, "TDX module rejected memory for extensions,
> stranded all pages\n");
> + break;
> + }
> +
> + added_pages += chunk_pages;
> + }
> +
> +out_free_hpa_list:
> + kfree(hpa_list);
> +
> + return ret;
> +}
> +
> +static __init int init_tdx_module_extensions(void)
> +{
> + if (!(tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_EXT))
> + return 0;
> +
> + return tdx_ext_mem_setup();
> +}
> +
> static __init int init_tdx_module(void)
> {
> int ret;
> @@ -1268,6 +1402,10 @@ static __init int init_tdx_module(void)
> if (ret)
> goto err_reset_pamts;
>
> + ret = init_tdx_module_extensions();
> + if (ret)
> + goto err_reset_pamts;
> +
> pr_info("%lu KB allocated for PAMT\n",
> tdmrs_count_pamt_kb(&tdx_tdmr_list));
>
> out_put_tdxmem:
> diff --git a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> b/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> index 98ebf17aab1c..9ab4d021017f 100644
> --- a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> +++ b/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> @@ -125,6 +125,20 @@ static int get_tdx_sys_info_handoff(struct
> tdx_sys_info_handoff *sysinfo_handoff
> return 0;
> }
>
> +static __init int get_tdx_sys_info_ext(struct tdx_sys_info_ext *sysinfo_ext)
> +{
> + int ret;
> + u64 val;
> +
> + ret = read_sys_metadata_field(0x3100000200000000, &val);
> + if (ret)
> + return ret;
> +
> + sysinfo_ext->required_pages = val;
> +
> + return 0;
> +}
> +
> static __init int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
> {
> int ret = 0;
>
next prev parent reply other threads:[~2026-11-18 22:18 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 3:48 [PATCH v5 0/6] Enable TDX module extensions Xu Yilun
2026-11-11 3:48 ` [PATCH v5 1/6] x86/virt/tdx: Move TDH.SYS.CONFIG operations into a wrapper Xu Yilun
2026-11-13 15:06 ` Nikolay Borisov
2026-11-18 18:51 ` Edgecombe, Rick P
2026-11-11 3:48 ` [PATCH v5 2/6] x86/virt/tdx: Configure add-on features on TDX module init Xu Yilun
2026-11-11 3:48 ` [PATCH v5 3/6] x86/virt/tdx: Add extra memory to TDX module for the extensions Xu Yilun
2026-11-17 3:06 ` Alexey Kardashevskiy
2026-11-18 2:32 ` Xu Yilun
2026-11-18 22:18 ` Edgecombe, Rick P [this message]
2026-11-11 3:48 ` [PATCH v5 4/6] x86/virt/tdx: Support DPAMT when adding memory " Xu Yilun
2026-11-11 3:48 ` [PATCH v5 5/6] x86/virt/tdx: Make TDX module initialize " Xu Yilun
2026-11-13 16:13 ` Nikolay Borisov
2026-11-18 22:44 ` Edgecombe, Rick P
2026-11-11 3:48 ` [PATCH v5 6/6] x86/virt/tdx: Re-initialize the extensions on runtime TDX module update Xu Yilun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f9c89d6da82cd737bcb7d72eb6ca5f5bf121be2b.camel@intel.com \
--to=rick.p.edgecombe@intel.com \
--cc=adrian.hunter@intel.com \
--cc=artem.bityutskiy@linux.intel.com \
--cc=baolu.lu@linux.intel.com \
--cc=chao.gao@intel.com \
--cc=dave.hansen@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=kas@kernel.org \
--cc=kishen.maloor@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-coco@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=nik.borisov@suse.com \
--cc=peter.fang@intel.com \
--cc=sohil.mehta@intel.com \
--cc=tony.lindgren@linux.intel.com \
--cc=x86@kernel.org \
--cc=xiaoyao.li@intel.com \
--cc=yilun.xu@intel.com \
--cc=yilun.xu@linux.intel.com \
--cc=zhenzhong.duan@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox