From: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>
To: "linux-coco@lists.linux.dev" <linux-coco@lists.linux.dev>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"yilun.xu@linux.intel.com" <yilun.xu@linux.intel.com>,
"x86@kernel.org" <x86@kernel.org>
Cc: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
"Li, Xiaoyao" <xiaoyao.li@intel.com>,
"Hansen, Dave" <dave.hansen@intel.com>,
"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
"Hunter, Adrian" <adrian.hunter@intel.com>,
"tony.lindgren@linux.intel.com" <tony.lindgren@linux.intel.com>,
"kas@kernel.org" <kas@kernel.org>,
"Xu, Yilun" <yilun.xu@intel.com>,
"artem.bityutskiy@linux.intel.com"
<artem.bityutskiy@linux.intel.com>,
"nik.borisov@suse.com" <nik.borisov@suse.com>,
"Mehta, Sohil" <sohil.mehta@intel.com>,
"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
"Gao, Chao" <chao.gao@intel.com>,
"Fang, Peter" <peter.fang@intel.com>,
"Maloor, Kishen" <kishen.maloor@intel.com>
Subject: Re: [PATCH v5 3/6] x86/virt/tdx: Add extra memory to TDX module for the extensions
Date: Wed, 18 Nov 2026 22:18:39 +0000 [thread overview]
Message-ID: <f9c89d6da82cd737bcb7d72eb6ca5f5bf121be2b.camel@intel.com> (raw)
In-Reply-To: <20261111-tdx-module-ext-v5-3-c74f764abc3d@linux.intel.com>
On Wed, 2026-11-11 at 11:48 +0800, Xu Yilun wrote:
> TDX module extensions need memory for their execution environment to
> serve SEAMCALL leafs. The TDX architecture implements the extensions
> in such a way that they use the memory outside of SEAM range, so the
> kernel should add the memory upfront at initialization time.
>
> Introduce a new memory adding process backed by a new SEAMCALL leaf
> TDH.EXT.MEM.ADD. The kernel queries TDX module how much memory needed,
> allocates it, adds it to the module, and never gets it back. Note that
> the process requires Dynamic PAMT backing.
The process requires it, or the memory does?
> A subsequent patch adds the
> support. This doesn't break bisectability cause the extensions can't be
^because?
> enabled yet without a feature that uses them.
Not clear whether the TDX module needs a new feature or the kernel does.
>
> The TDX module accepts the memory in the form of an HPA array. This
> array is passed via a single 64-bit SEAMCALL leaf parameter, which
> encodes two values: the PFN of the container page holding the array, and
> the number of entries in the array. Create a helper to encode this
> format and name it after the TDX module term: HPA_LIST_INFO.
>
> TDX module extensions consume tens of megabytes memory that will never
> be returned to the host. Use contiguous page allocation to isolate these
> large blocks entirely. This is a simple way to avoid permanent memory
> fragmentation: requiring the full size to be contiguous is more
> expensive than needed but should be good during the boot time. Print the
> allocation amount on TDX module extensions initialization for
> visibility.
>
> The TDX module provides a metadata field "required_pages" to indicate
> the required memory size. This metadata is meaningful only after the
> add-on feature configuration. Don't read and cache the metadata in
> tdx_sysinfo. Instead read it right before allocating memory for the
> extensions, when it is guaranteed to be valid.
>
> The original changelog was written by hand. AI was used to help correct
> grammatical errors and polish individual sentences for better
> readability.
>
> Signed-off-by: Xu Yilun <yilun.xu@linux.intel.com>
> Reviewed-by: Tony Lindgren <tony.lindgren@linux.intel.com>
> Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
> ---
> An alternative solution is to use a loop that gives memory on a memory
> error code - TDX_EXT_MEMORY_POOL_REQUIRED, add one page per iteration
> until TDH.EXT.INIT succeeds. Something like:
>
> do {
> ret = tdh_sys_init();
> if (ret == TDX_EXT_MEMORY_POOL_REQUIRED)
> tdh_ext_mem_add(); //single page
> } while (ret == TDX_EXT_MEMORY_POOL_REQUIRED);
>
> This approach is slightly simpler as we don't have to query the module
> for the total memory, no memory pre-allocation or segmentation math.
> But allocating a single 4K page per iteration may cause permanent memory
> fragmentation.
>
> v5:
> - Add the DPAMT & bisectability note
> - Shorten the "required_pages field" paragraph a little bit
> - Collect Reviewed-by tags
> - Better clarify the AI usage
>
> v4:
> - Rename memory_pool_required_pages to required_pages (Rick)
> - Rename local var sysinfo_ext to ext
> - Collect Reviewed-by tags
>
> v3:
> - Merge remaining pre-check operations from the previous patch (Rick)
> - Don't save the metadata in tdx_sysinfo (Rick)
> - Tweak the code comment for memory_pool_required_pages == 0 (Chao)
> - Re-phrase the code comment for struct tdx_hpa_list (Tony)
> - Put the error handling comments on top of memory add loop, to avoid
> moving them around when DPAMT support is added in the following
> patch.
> - Print memory size before adding memory; it won't be freed afterward,
> whether the operation succeeds or fails.
> - Collect Reviewed-by tags
>
> v2:
> - Add code comments for container page allocation (Kiryl)
> - Adjust comments/changelog for contiguous allocation of extensions
> memory (Kiryl)
> - s/PFN array/HPA array in commit log (Kiryl)
> - State the alternative solution (Rick)
> - Add code comment for struct tdx_hpa_list
>
> v1:
> - Fix return value for SEAMCALL wrappers (Chao)
> - Print SEAMCALL error code for SEAMCALL wrappers (Xiaoyao)
> - Rename local vars to make the ext memory adding loop clear (Rick)
> - Remove input parameters for tdx_ext_mem_setup() (Kevin)
> - Add a Macro for tdh_hpa_list size.
> - Change the SEAMALL wrapper parameter type,
> struct page *hpa_list => struct tdx_hpa_list *hpa_list
> - changelog & code comments
> ---
> arch/x86/include/asm/tdx.h | 1 +
> arch/x86/include/asm/tdx_global_metadata.h | 4 +
> arch/x86/virt/vmx/tdx/tdx.h | 1 +
> arch/x86/virt/vmx/tdx/tdx.c | 138
> ++++++++++++++++++++++++++++
> arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 14 +++
> 5 files changed, 158 insertions(+)
>
> diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h
> index 1fcfaed515fe..89e7628796b0 100644
> --- a/arch/x86/include/asm/tdx.h
> +++ b/arch/x86/include/asm/tdx.h
> @@ -37,6 +37,7 @@
> #define TDX_FEATURES0_TD_PRESERVING BIT_ULL(1)
> #define TDX_FEATURES0_NO_RBP_MOD BIT_ULL(18)
> #define TDX_FEATURES0_DYNAMIC_PAMT BIT_ULL(36)
> +#define TDX_FEATURES0_EXT BIT_ULL(39)
>
> #ifndef __ASSEMBLER__
>
> diff --git a/arch/x86/include/asm/tdx_global_metadata.h
> b/arch/x86/include/asm/tdx_global_metadata.h
> index 8a3cc1a2a41e..f18e63b671c2 100644
> --- a/arch/x86/include/asm/tdx_global_metadata.h
> +++ b/arch/x86/include/asm/tdx_global_metadata.h
> @@ -47,6 +47,10 @@ struct tdx_sys_info_handoff {
> u16 module_hv;
> };
>
> +struct tdx_sys_info_ext {
> + u32 required_pages;
> +};
> +
> struct tdx_sys_info {
> struct tdx_sys_info_version version;
> struct tdx_sys_info_features features;
> diff --git a/arch/x86/virt/vmx/tdx/tdx.h b/arch/x86/virt/vmx/tdx/tdx.h
> index db209541d3cd..2adc5d31d258 100644
> --- a/arch/x86/virt/vmx/tdx/tdx.h
> +++ b/arch/x86/virt/vmx/tdx/tdx.h
> @@ -50,6 +50,7 @@
> #define TDH_SYS_UPDATE 53
> #define TDH_PHYMEM_PAMT_ADD 58
> #define TDH_PHYMEM_PAMT_REMOVE 59
> +#define TDH_EXT_MEM_ADD 61
> #define TDH_SYS_DISABLE 69
>
> /* TDX page types */
> diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> index 3bfe78b4b538..865d8089d343 100644
> --- a/arch/x86/virt/vmx/tdx/tdx.c
> +++ b/arch/x86/virt/vmx/tdx/tdx.c
> @@ -1210,6 +1210,140 @@ static __init int init_tdmrs(struct tdmr_info_list
> *tdmr_list)
> return 0;
> }
>
> +#define TDX_HPA_LIST_MAX_NR_PAGES (PAGE_SIZE / sizeof(u64))
> +
> +/*
> + * TDX module in-memory ABI to specify the memory being added to the TDX
> + * module. An array of HPAs, each element points to a 4K page.
> + */
> +struct tdx_hpa_list {
> + u64 phys[TDX_HPA_LIST_MAX_NR_PAGES];
> +};
> +
> +static_assert(sizeof(struct tdx_hpa_list) == PAGE_SIZE);
> +
> +#define HPA_LIST_INFO_FIRST_ENTRY GENMASK_U64(11, 3)
> +#define HPA_LIST_INFO_PFN GENMASK_U64(51, 12)
> +#define HPA_LIST_INFO_LAST_ENTRY GENMASK_U64(63, 55)
> +
> +static __init u64 to_hpa_list_info(struct tdx_hpa_list *hpa_list,
> + unsigned int nr_pages)
> +{
> + return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
> + FIELD_PREP(HPA_LIST_INFO_PFN, PFN_DOWN(__pa(hpa_list))) |
> + FIELD_PREP(HPA_LIST_INFO_LAST_ENTRY, nr_pages - 1);
> +}
> +
> +static __init int tdx_ext_mem_add(struct tdx_hpa_list *hpa_list,
> + unsigned int nr_pages)
> +{
> + struct tdx_module_args args = {
> + .rcx = to_hpa_list_info(hpa_list, nr_pages),
> + };
> + u64 ret;
> +
> + do {
> + /*
> + * The TDX module overwrites RCX to track progress when this
> + * SEAMCALL leaf is interrupted. Use seamcall_ret() to save
> and
> + * pass the updated value back on retry.
> + */
> + ret = seamcall_ret(TDH_EXT_MEM_ADD, &args);
> + } while (ret == TDX_INTERRUPTED_RESUMABLE);
> +
> + if (ret != TDX_SUCCESS) {
> + pr_err("TDH.EXT.MEM.ADD failed: 0x%016llx\n", ret);
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +
> +static __init int tdx_ext_mem_setup(void)
> +{
> + struct tdx_hpa_list *hpa_list;
> + struct tdx_sys_info_ext ext;
> + unsigned int added_pages;
> + struct page *page;
> + int ret;
> +
> + ret = get_tdx_sys_info_ext(&ext);
> + if (ret)
> + return ret;
> +
> + /*
> + * Skip the memory setup if no memory is required. This may happen
> when
> + * no add-on features requiring TDX module extensions are configured
> + * via TDH.SYS.CONFIG.
> + */
> + if (!ext.required_pages)
> + return 0;
> +
> + /*
> + * Allocate the container page for the HPA_LIST. tdx_hpa_list is
> + * guaranteed to be page-sized by static_assert(), so kzalloc()
> + * guarantees the page alignment.
> + */
> + hpa_list = kzalloc_obj(*hpa_list);
> + if (!hpa_list)
> + return -ENOMEM;
> +
> + /*
> + * Memory for TDX module extensions is never reclaimed and can be
> tens
> + * of megabytes. Allocating a physically contiguous chunk is a simple
> + * way to avoid permanent memory fragmentation: requiring the full
> size
> + * to be contiguous is more expensive than needed but should be good
> + * during the boot time.
> + */
> + page = alloc_contig_pages(ext.required_pages, GFP_KERNEL,
> + numa_mem_id(), &node_online_map);
> + if (!page) {
> + ret = -ENOMEM;
> + goto out_free_hpa_list;
> + }
> +
> + /* Print the amount so users know the cost. */
> + pr_info("%lu KB allocated for TDX module extensions\n",
> + ext.required_pages * PAGE_SIZE / 1024);
I wonder if this line is absolutely required for a minimal set?
> +
> + /*
> + * Following TDX operations shouldn't fail, and if they do, things
> are
> + * broken enough that complex error handling isn't worth it.
> + * Intentionally leak all pages on failure, including un-added pages.
> + */
> + added_pages = 0;
> + while (added_pages < ext.required_pages) {
> + unsigned int chunk_pages = min(ext.required_pages -
> added_pages,
> + TDX_HPA_LIST_MAX_NR_PAGES);
> + struct page *chunk = page + added_pages;
> + unsigned int i;
> +
> + for (i = 0; i < chunk_pages; i++)
> + hpa_list->phys[i] = page_to_phys(chunk + i);
> +
> + ret = tdx_ext_mem_add(hpa_list, chunk_pages);
> + if (ret) {
> + WARN(1, "TDX module rejected memory for extensions,
> stranded all pages\n");
> + break;
> + }
> +
> + added_pages += chunk_pages;
> + }
> +
> +out_free_hpa_list:
> + kfree(hpa_list);
> +
> + return ret;
> +}
> +
> +static __init int init_tdx_module_extensions(void)
> +{
> + if (!(tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_EXT))
> + return 0;
> +
> + return tdx_ext_mem_setup();
> +}
> +
> static __init int init_tdx_module(void)
> {
> int ret;
> @@ -1268,6 +1402,10 @@ static __init int init_tdx_module(void)
> if (ret)
> goto err_reset_pamts;
>
> + ret = init_tdx_module_extensions();
> + if (ret)
> + goto err_reset_pamts;
> +
> pr_info("%lu KB allocated for PAMT\n",
> tdmrs_count_pamt_kb(&tdx_tdmr_list));
>
> out_put_tdxmem:
> diff --git a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> b/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> index 98ebf17aab1c..9ab4d021017f 100644
> --- a/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> +++ b/arch/x86/virt/vmx/tdx/tdx_global_metadata.c
> @@ -125,6 +125,20 @@ static int get_tdx_sys_info_handoff(struct
> tdx_sys_info_handoff *sysinfo_handoff
> return 0;
> }
>
> +static __init int get_tdx_sys_info_ext(struct tdx_sys_info_ext *sysinfo_ext)
> +{
> + int ret;
> + u64 val;
> +
> + ret = read_sys_metadata_field(0x3100000200000000, &val);
> + if (ret)
> + return ret;
> +
> + sysinfo_ext->required_pages = val;
> +
> + return 0;
> +}
> +
> static __init int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
> {
> int ret = 0;
>
next prev parent reply other threads:[~2026-11-18 22:18 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-11 3:48 [PATCH v5 0/6] Enable TDX module extensions Xu Yilun
2026-11-11 3:48 ` [PATCH v5 1/6] x86/virt/tdx: Move TDH.SYS.CONFIG operations into a wrapper Xu Yilun
2026-11-13 15:06 ` Nikolay Borisov
2026-11-18 18:51 ` Edgecombe, Rick P
2026-11-19 3:31 ` Xu Yilun
2026-11-11 3:48 ` [PATCH v5 2/6] x86/virt/tdx: Configure add-on features on TDX module init Xu Yilun
2026-11-11 3:48 ` [PATCH v5 3/6] x86/virt/tdx: Add extra memory to TDX module for the extensions Xu Yilun
2026-11-17 3:06 ` Alexey Kardashevskiy
2026-11-18 2:32 ` Xu Yilun
2026-11-18 22:18 ` Edgecombe, Rick P [this message]
2026-11-19 4:50 ` Xu Yilun
2026-11-11 3:48 ` [PATCH v5 4/6] x86/virt/tdx: Support DPAMT when adding memory " Xu Yilun
2026-11-11 3:48 ` [PATCH v5 5/6] x86/virt/tdx: Make TDX module initialize " Xu Yilun
2026-11-13 16:13 ` Nikolay Borisov
2026-11-18 22:44 ` Edgecombe, Rick P
2026-11-19 5:20 ` Xu Yilun
2026-11-11 3:48 ` [PATCH v5 6/6] x86/virt/tdx: Re-initialize the extensions on runtime TDX module update Xu Yilun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f9c89d6da82cd737bcb7d72eb6ca5f5bf121be2b.camel@intel.com \
--to=rick.p.edgecombe@intel.com \
--cc=adrian.hunter@intel.com \
--cc=artem.bityutskiy@linux.intel.com \
--cc=baolu.lu@linux.intel.com \
--cc=chao.gao@intel.com \
--cc=dave.hansen@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=kas@kernel.org \
--cc=kishen.maloor@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-coco@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=nik.borisov@suse.com \
--cc=peter.fang@intel.com \
--cc=sohil.mehta@intel.com \
--cc=tony.lindgren@linux.intel.com \
--cc=x86@kernel.org \
--cc=xiaoyao.li@intel.com \
--cc=yilun.xu@intel.com \
--cc=yilun.xu@linux.intel.com \
--cc=zhenzhong.duan@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.