Re: [PATCH v8 8/9] x86/virt/tdx: Reduce TDMR's reserved areas by using CMRs to find memory holes

["Huang, Kai" <kai.huang@intel.com>]

On Wed, 2024-12-04 at 14:22 +0000, Huang, Kai wrote:
> On Thu, 2024-11-14 at 00:57 +1300, Kai Huang wrote:
> > A TDX module initialization failure was reported on a Emerald Rapids
> > platform [*]:
> > 
> >   virt/tdx: initialization failed: TDMR [0x0, 0x80000000): reserved areas exhausted.
> >   virt/tdx: module initialization failed (-28)
> > 
> > As part of initializing the TDX module, the kernel informs the TDX
> > module of all "TDX-usable memory regions" using an array of TDX defined
> > structure "TD Memory Region" (TDMR).  Each TDMR must be in 1GB aligned
> > and in 1GB granularity, and all "non-TDX-usable memory holes" within a
> > given TDMR are marked as "reserved areas".  The TDX module reports a
> > maximum number of reserved areas that can be supported per TDMR (16).
> > 
> > The kernel builds the "TDX-usable memory regions" based on memblocks
> > (which reflects e820), and uses this list to find all "reserved areas"
> > for each TDMR.
> > 
> > It turns out that the kernel's view of memory holes is too fine grained
> > and sometimes exceeds the number of holes that the TDX module can track
> > per TDMR [1], resulting in the above failure.
> > 
> > Thankfully the module also lists memory that is potentially convertible
> > in a list of "Convertible Memory Regions" (CMRs).  That coarser grained
> > CMR list tends to track usable memory in the memory map even if it might
> > be reserved for host usage like 'ACPI data' [2].
> > 
> > Use that list to relax what the kernel considers unusable memory.  If it
> > falls in a CMR no need to instantiate a hole, and rely on the fact that
> > kernel will keep what it considers 'reserved' out of the page allocator.
> > 
> 
> Hi Dave,
> 
> I realized there's one issue if we change to use CMRs to fill up reserved areas
> for TDMRs after some internal discussion with Dan:
> 
> Currently we requires all memory regions in memblocks.memory to be TDX
> convertible memory at the time of initializing the TDX module to make module
> initialization successful.  We build a list of those memory regions as a list of
> "TDX memory blocks", and use them to construct the TDMRs to configure the
> module.  But we don't explicitly check those memory regions against CMRs to make
> sure they are truly TDX convertible, instead we depend on TDH.SYS.CONFIG to
> catch any non-CMR memory regions that end up to the TDX memory blocks.
> 
> This works fine because currently we use those TDX memory blocks to fill up the
> reserved areas of TDMRs, i.e., any non-CMR regions in TDX memory blocks will end
> up to "non-reserved areas" in TDMR(s), and this will cause TDH.SYS.CONFIG to
> fail.
> 
> After we change to using CMRs to fill up reserved areas of TDMRs, then all non-
> CMR regions will be marked as "reserved areas", regardless whether there is any
> non-CMR memory region in the TDX memory blocks.  This will result in TDX module
> initialization being successful while there are non-CMR pages going into page
> allocator.
> 
> To avoid this, we need to explicitly check all the TDX memory blocks against
> CMRs to make sure they are actually TDX convertible, before using CMRs to fill
> up reserved areas.
> 
> I ended up with below incremental diff with some additional text for the
> changelog.
> 
> Do you have any comments?
> 

Hi Dave,

I sent out the updated patch as "PATCH v8 8.1/9".  I appreciate if you can
review.

Hi Dan,

I didn't remove your Reviewed-by because I think the additional code added
should be straightforward.  But please let me know if you have any concern.

Thanks!