[PATCH v3 00/18] ASI: HVM per-vCPU areas and sparse-view directmap

[PATCH v3 00/18] ASI: HVM per-vCPU areas and sparse-view directmap

[George Dunlap]

In an attempt to reach something useful for XenServer sooner, I've
changed the approach somewhat.

The previous approach implemented sparse direct-map and intra-domain
isolation everywhere, for all types of guests.  Implementing
intra-domain isolation for PV guests introduces a large amount of
complexity, due to the ABI of Xen running on guest pagetables.

Start from the other direction instead, and focus just on getting ASI
and intra-domain isolation for HVM guests.

This series looks like this:

- A fix that should be considered for backport (01).

- Implement per-vCPU per-domain area and per-vCPU mapcaches for HVM
guests.  (02-11).  This is much simpler since each HVM vcpu already
has its own monitor table.

- Maintain two "views" of the directmap (12-18): A full directmap, and
a sparse directmap.  The full directmap is what we have now; the
sparse directmap, which removes non-xenheap heap memory.

The key patch to get feedback on is patch 13/18.

Note that at the end of this series, PV guests can be given the sparse
direct-map; this isolates them from other domains, but doesn't isolate
vCPUs within a domain.

An advantage of this is that it allows us to easily turn the sparse
directmap off for dom0 (or other system domains).  Some quick tests on
my local test rig show that network throughput from dom0 to a guest
drops 31% when the sparse directmap is enabled for dom0 vs not.

The rest of the series would look like this:

- Implement a per-vCPU state area, and move key bits of guest state into it.

- Implement per-pCPU stacks.  In this mode, per-pCPU stacks would only
be for HVM guests using the "sparse directmap".  Until we get per-vCPU
root pagetables for PV guests, it's not practical to prevent a PV
guest from seeing all the stacks.

A dependency graph of the above functionality can be found here:

https://xenbits.xenproject.org/people/gdunlap/asi-series-deps.html

Extending the intra-domain isolation to PV guests would involve
implementing per-vCPU root pagetables for PV guests.  There are draft
implementations of this, but they are quite complicated.

George Dunlap (9):
  x86/PoD: map one page at a time in p2m_pod_zero_check()
  x86/domain_page: don't unmap a mapcache entry that is not present
  x86/mm: let Xen's page-table walker operate on a given root
  x86/mm: build and maintain a sparse view of the directmap
  x86/domain_page: key the directmap fast paths on the loaded view
  x86/mm: flush owned pages before reuse with the sparse directmap view
  x86/domain_page: keep the maphash's slots, not their mappings
  x86/mm: run guest contexts on the sparse directmap view
  x86/spec-ctrl: wire the sparse directmap view into the asi= option

Roger Pau Monné (9):
  x86/mm: purge unneeded destroy_perdomain_mapping()
  x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain
    areas
  x86/domain_page: drop redundant create_perdomain_mapping() call
  x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain
    areas
  x86/spec-ctrl: introduce Address Space Isolation command line option
  x86/hvm: introduce per-vCPU L3 page-table
  x86/domain_page: prepare the mapcache for per-vCPU accounting
  x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu
  x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains

 docs/misc/xen-command-line.pandoc           |  49 ++
 xen/arch/x86/Kconfig                        |   1 +
 xen/arch/x86/Makefile                       |   1 +
 xen/arch/x86/domain.c                       |  21 +-
 xen/arch/x86/domain_page.c                  | 311 ++++++++---
 xen/arch/x86/hvm/hvm.c                      |   6 -
 xen/arch/x86/include/asm/domain.h           |  62 ++-
 xen/arch/x86/include/asm/mm.h               |  26 +-
 xen/arch/x86/include/asm/sparse-directmap.h |  98 ++++
 xen/arch/x86/include/asm/spec_ctrl.h        |   2 +
 xen/arch/x86/mm.c                           | 259 +++++++--
 xen/arch/x86/mm/hap/hap.c                   |   2 +-
 xen/arch/x86/mm/p2m-pod.c                   |  72 ++-
 xen/arch/x86/mm/shadow/hvm.c                |   2 +-
 xen/arch/x86/mm/shadow/multi.c              |   5 +-
 xen/arch/x86/pv/dom0_build.c                |   6 +-
 xen/arch/x86/pv/domain.c                    |  14 +-
 xen/arch/x86/setup.c                        |   7 +
 xen/arch/x86/sparse-directmap.c             | 570 ++++++++++++++++++++
 xen/arch/x86/spec_ctrl.c                    | 194 ++++++-
 xen/arch/x86/x86_64/mm.c                    |  10 +-
 xen/common/Kconfig                          |  18 +
 xen/common/page_alloc.c                     |  55 +-
 xen/include/xen/mm.h                        |  41 ++
 24 files changed, 1615 insertions(+), 217 deletions(-)
 create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
 create mode 100644 xen/arch/x86/sparse-directmap.c

-- 
2.55.0

[PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[George Dunlap]

p2m_pod_zero_check() maps every page of its batch that passes its
checks, up to POD_SWEEP_STRIDE (16) of them, and keeps them mapped
across the p2m lookups and updates which follow, each of which maps
page-table pages of its own.  Its callers hold mappings too: the p2m
lookup which found the entry to populate still has its page-table page
mapped, and a PV mmu_update() the L1 page it is writing.

map_domain_page() cannot always provide that many at once.  It uses the
mapcache on behalf of PV vCPUs only: in debug builds for every page, in
release builds for pages beyond the reach of the directmap in PV guests'
page-tables (5TiB, 3.5TiB with CONFIG_BIGMEM).  The mapcache has
MAPCACHE_VCPU_ENTRIES (16) slots per vCPU of the domain, shared by its
vCPUs, and a mapping which finds no slot free or reclaimable hits the
BUG_ON() in map_domain_page(): the host crashes.

The sweep runs in the context of whichever vCPU populates a PoD entry
while the domain's PoD cache is empty.  The guest's own vCPUs, being
HVM, map through the directmap; a PV vCPU gets there when its domain
maps or copies the guest's memory: a device model in dom0 or in a stub
domain, a backend doing grant copies.  A PV domain with a single vCPU,
as a device-model stub domain or a dom0 booted with dom0_max_vcpus=1
is, has 16 slots, and a sweep over 16 candidate pages needs more.

The sweep has mapped its whole batch at once since 9ad9a1609fbe ("PoD
memory 5/9: emergency scan"), when x86-64 reached all memory through the
directmap.  The per-vCPU budget came with the mapcache's return to
x86-64 in 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al").

Map one page at a time instead, as p2m_pod_zero_check_superpage()
already does: map a page for the quick check and unmap it again, and
map it anew for the full check after the p2m update and the TLB flush.
The checks and their order are unchanged.  Each page is mapped twice,
which costs little next to scanning it, and the error paths no longer
have mappings to undo.

Fixes: 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al")
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
NB this issue was raised in review and verified only by code
inspection; it hasn't been demonstrated.
---
 xen/arch/x86/mm/p2m-pod.c | 72 +++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 37 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
index 4602c32cff..0b0beca15a 100644
--- a/xen/arch/x86/mm/p2m-pod.c
+++ b/xen/arch/x86/mm/p2m-pod.c
@@ -901,7 +901,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
 {
     mfn_t mfns[POD_SWEEP_STRIDE];
     p2m_type_t types[POD_SWEEP_STRIDE];
-    unsigned long *map[POD_SWEEP_STRIDE];
+    bool check[POD_SWEEP_STRIDE];
+    const unsigned long *map;
     struct domain *d = p2m->domain;
     unsigned int i, j, max_ref = 1;
 
@@ -911,7 +912,12 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
     if ( paging_mode_shadow(d) )
         max_ref++;
 
-    /* First, get the gfn list, translate to mfns, and map the pages. */
+    /*
+     * First, get the gfn list, translate to mfns, and pick the pages to
+     * check.  They are mapped one at a time below, as in
+     * p2m_pod_zero_check_superpage(): a vCPU can hold only a few transient
+     * mappings at once, and the p2m lookups and updates here need some too.
+     */
     for ( i = 0; i < count; i++ )
     {
         p2m_access_t a;
@@ -921,18 +927,17 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
 
         /*
          * If this is ram, and not a pagetable or a special page, and
-         * probably not mapped elsewhere, map it; otherwise, skip.
+         * probably not mapped elsewhere, check it; otherwise, skip.
          */
-        map[i] = NULL;
+        check[i] = false;
         if ( p2m_is_ram(types[i]) )
         {
             const struct page_info *pg = mfn_to_page(mfns[i]);
 
-            if ( !is_special_page(pg) &&
-                 (pg->count_info & PGC_allocated) &&
-                 !(pg->count_info & PGC_shadowed_pt) &&
-                 ((pg->count_info & PGC_count_mask) <= max_ref) )
-                map[i] = map_domain_page(mfns[i]);
+            check[i] = !is_special_page(pg) &&
+                       (pg->count_info & PGC_allocated) &&
+                       !(pg->count_info & PGC_shadowed_pt) &&
+                       ((pg->count_info & PGC_count_mask) <= max_ref);
         }
     }
 
@@ -942,18 +947,24 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
      */
     for ( i = 0; i < count; i++ )
     {
-        if ( !map[i] )
+        if ( !check[i] )
             continue;
 
         /* Quick zero-check */
+        map = map_domain_page(mfns[i]);
         for ( j = 0; j < 16; j++ )
-            if ( *(map[i] + j) != 0 )
-                goto skip;
+            if ( map[j] != 0 )
+                break;
+        unmap_domain_page(map);
 
         /* Try to remove the page, restoring old mapping if it fails. */
-        if ( p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
+        if ( j < 16 ||
+             p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
                            p2m_populate_on_demand, p2m->default_access) )
-            goto skip;
+        {
+            check[i] = false;
+            continue;
+        }
 
         /*
          * See if the page was successfully unmapped.  (Allow one refcount
@@ -961,6 +972,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
          */
         if ( (mfn_to_page(mfns[i])->count_info & PGC_count_mask) > 1 )
         {
+            check[i] = false;
+
             /*
              * If the previous p2m_set_entry call succeeded, this one shouldn't
              * be able to fail.  If it does, crashing the domain should be safe.
@@ -970,14 +983,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
             {
                 ASSERT_UNREACHABLE();
                 domain_crash(d);
-                goto out_unmap;
+                return;
             }
-
-        skip:
-            unmap_domain_page(map[i]);
-            map[i] = NULL;
-
-            continue;
         }
     }
 
@@ -986,22 +993,20 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
     /* Now check each page for real */
     for ( i = 0; i < count; i++ )
     {
-        if ( !map[i] )
+        if ( !check[i] )
             continue;
 
-        for ( j = 0; j < (PAGE_SIZE / sizeof(*map[i])); j++ )
-            if ( *(map[i] + j) != 0 )
+        map = map_domain_page(mfns[i]);
+        for ( j = 0; j < (PAGE_SIZE / sizeof(*map)); j++ )
+            if ( map[j] != 0 )
                 break;
-
-        unmap_domain_page(map[i]);
-
-        map[i] = NULL;
+        unmap_domain_page(map);
 
         /*
          * See comment in p2m_pod_zero_check_superpage() re gnttab
          * check timing.
          */
-        if ( j < (PAGE_SIZE / sizeof(*map[i])) )
+        if ( j < (PAGE_SIZE / sizeof(*map)) )
         {
             /*
              * If the previous p2m_set_entry call succeeded, this one shouldn't
@@ -1012,14 +1017,7 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
             {
                 ASSERT_UNREACHABLE();
                 domain_crash(d);
- out_unmap:
-                /*
-                 * Something went wrong, probably crashing the domain.  Unmap
-                 * everything and return.
-                 */
-                for ( i = 0; i < count; i++ )
-                    if ( map[i] )
-                        unmap_domain_page(map[i]);
+                return;
             }
         }
         else
-- 
2.55.0

Re: [PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> p2m_pod_zero_check() maps every page of its batch that passes its
> checks, up to POD_SWEEP_STRIDE (16) of them, and keeps them mapped
> across the p2m lookups and updates which follow, each of which maps
> page-table pages of its own.  Its callers hold mappings too: the p2m
> lookup which found the entry to populate still has its page-table page
> mapped, and a PV mmu_update() the L1 page it is writing.
> 
> map_domain_page() cannot always provide that many at once.  It uses the
> mapcache on behalf of PV vCPUs only: in debug builds for every page, in
> release builds for pages beyond the reach of the directmap in PV guests'
> page-tables (5TiB, 3.5TiB with CONFIG_BIGMEM).  The mapcache has
> MAPCACHE_VCPU_ENTRIES (16) slots per vCPU of the domain, shared by its
> vCPUs, and a mapping which finds no slot free or reclaimable hits the
> BUG_ON() in map_domain_page(): the host crashes.
> 
> The sweep runs in the context of whichever vCPU populates a PoD entry
> while the domain's PoD cache is empty.  The guest's own vCPUs, being
> HVM, map through the directmap; a PV vCPU gets there when its domain
> maps or copies the guest's memory: a device model in dom0 or in a stub
> domain, a backend doing grant copies.  A PV domain with a single vCPU,
> as a device-model stub domain or a dom0 booted with dom0_max_vcpus=1
> is, has 16 slots, and a sweep over 16 candidate pages needs more.
> 
> The sweep has mapped its whole batch at once since 9ad9a1609fbe ("PoD
> memory 5/9: emergency scan"), when x86-64 reached all memory through the
> directmap.  The per-vCPU budget came with the mapcache's return to
> x86-64 in 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al").
> 
> Map one page at a time instead, as p2m_pod_zero_check_superpage()
> already does: map a page for the quick check and unmap it again, and
> map it anew for the full check after the p2m update and the TLB flush.
> The checks and their order are unchanged.  Each page is mapped twice,
> which costs little next to scanning it, and the error paths no longer
> have mappings to undo.
> 
> Fixes: 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al")
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Reviewed-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[Andrew Cooper]

On 07/10/2026 11:40 am, George Dunlap wrote:
> p2m_pod_zero_check() maps every page of its batch that passes its
> checks, up to POD_SWEEP_STRIDE (16) of them, and keeps them mapped
> across the p2m lookups and updates which follow, each of which maps
> page-table pages of its own.  Its callers hold mappings too: the p2m
> lookup which found the entry to populate still has its page-table page
> mapped, and a PV mmu_update() the L1 page it is writing.
>
> map_domain_page() cannot always provide that many at once.  It uses the
> mapcache on behalf of PV vCPUs only: in debug builds for every page, in
> release builds for pages beyond the reach of the directmap in PV guests'
> page-tables (5TiB, 3.5TiB with CONFIG_BIGMEM).  The mapcache has
> MAPCACHE_VCPU_ENTRIES (16) slots per vCPU of the domain, shared by its
> vCPUs, and a mapping which finds no slot free or reclaimable hits the
> BUG_ON() in map_domain_page(): the host crashes.
>
> The sweep runs in the context of whichever vCPU populates a PoD entry
> while the domain's PoD cache is empty.  The guest's own vCPUs, being
> HVM, map through the directmap; a PV vCPU gets there when its domain
> maps or copies the guest's memory: a device model in dom0 or in a stub
> domain, a backend doing grant copies.  A PV domain with a single vCPU,
> as a device-model stub domain or a dom0 booted with dom0_max_vcpus=1
> is, has 16 slots, and a sweep over 16 candidate pages needs more.
>
> The sweep has mapped its whole batch at once since 9ad9a1609fbe ("PoD
> memory 5/9: emergency scan"), when x86-64 reached all memory through the
> directmap.  The per-vCPU budget came with the mapcache's return to
> x86-64 in 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al").
>
> Map one page at a time instead, as p2m_pod_zero_check_superpage()
> already does: map a page for the quick check and unmap it again, and
> map it anew for the full check after the p2m update and the TLB flush.
> The checks and their order are unchanged.  Each page is mapped twice,
> which costs little next to scanning it, and the error paths no longer
> have mappings to undo.
>
> Fixes: 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al")
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> NB this issue was raised in review and verified only by code
> inspection; it hasn't been demonstrated.
> ---
>  xen/arch/x86/mm/p2m-pod.c | 72 +++++++++++++++++++--------------------
>  1 file changed, 35 insertions(+), 37 deletions(-)
>
> diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
> index 4602c32cff..0b0beca15a 100644
> --- a/xen/arch/x86/mm/p2m-pod.c
> +++ b/xen/arch/x86/mm/p2m-pod.c
> @@ -901,7 +901,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>  {
>      mfn_t mfns[POD_SWEEP_STRIDE];
>      p2m_type_t types[POD_SWEEP_STRIDE];
> -    unsigned long *map[POD_SWEEP_STRIDE];
> +    bool check[POD_SWEEP_STRIDE];
> +    const unsigned long *map;
>      struct domain *d = p2m->domain;
>      unsigned int i, j, max_ref = 1;

I agree with getting rid of the map batching, but your LLM has done a
poor job doing so.

An array of 16 bools is excessive.  You want "unsigned int check = 0;"
here, so ...

>  
> @@ -911,7 +912,12 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>      if ( paging_mode_shadow(d) )
>          max_ref++;
>  
> -    /* First, get the gfn list, translate to mfns, and map the pages. */
> +    /*
> +     * First, get the gfn list, translate to mfns, and pick the pages to
> +     * check.  They are mapped one at a time below, as in
> +     * p2m_pod_zero_check_superpage(): a vCPU can hold only a few transient
> +     * mappings at once, and the p2m lookups and updates here need some too.
> +     */
>      for ( i = 0; i < count; i++ )
>      {
>          p2m_access_t a;
> @@ -921,18 +927,17 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>  
>          /*
>           * If this is ram, and not a pagetable or a special page, and
> -         * probably not mapped elsewhere, map it; otherwise, skip.
> +         * probably not mapped elsewhere, check it; otherwise, skip.
>           */
> -        map[i] = NULL;
> +        check[i] = false;

... this can be dropped, and ...

>          if ( p2m_is_ram(types[i]) )
>          {
>              const struct page_info *pg = mfn_to_page(mfns[i]);
>  
> -            if ( !is_special_page(pg) &&
> -                 (pg->count_info & PGC_allocated) &&
> -                 !(pg->count_info & PGC_shadowed_pt) &&
> -                 ((pg->count_info & PGC_count_mask) <= max_ref) )
> -                map[i] = map_domain_page(mfns[i]);
> +            check[i] = !is_special_page(pg) &&
> +                       (pg->count_info & PGC_allocated) &&
> +                       !(pg->count_info & PGC_shadowed_pt) &&
> +                       ((pg->count_info & PGC_count_mask) <= max_ref);

... this can be put back to it's older if() form, with `check |= (1U <<
i);` instead, so ...

>          }
>      }
>  
> @@ -942,18 +947,24 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>       */
>      for ( i = 0; i < count; i++ )
>      {
> -        if ( !map[i] )
> +        if ( !check[i] )
>              continue;

... this and ...

>  
>          /* Quick zero-check */
> +        map = map_domain_page(mfns[i]);
>          for ( j = 0; j < 16; j++ )
> -            if ( *(map[i] + j) != 0 )
> -                goto skip;
> +            if ( map[j] != 0 )
> +                break;
> +        unmap_domain_page(map);
>  
>          /* Try to remove the page, restoring old mapping if it fails. */
> -        if ( p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
> +        if ( j < 16 ||
> +             p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
>                             p2m_populate_on_demand, p2m->default_access) )
> -            goto skip;
> +        {
> +            check[i] = false;
> +            continue;
> +        }
>  
>          /*
>           * See if the page was successfully unmapped.  (Allow one refcount
> @@ -961,6 +972,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>           */
>          if ( (mfn_to_page(mfns[i])->count_info & PGC_count_mask) > 1 )
>          {
> +            check[i] = false;
> +
>              /*
>               * If the previous p2m_set_entry call succeeded, this one shouldn't
>               * be able to fail.  If it does, crashing the domain should be safe.
> @@ -970,14 +983,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>              {
>                  ASSERT_UNREACHABLE();
>                  domain_crash(d);
> -                goto out_unmap;
> +                return;
>              }
> -
> -        skip:
> -            unmap_domain_page(map[i]);
> -            map[i] = NULL;
> -
> -            continue;
>          }
>      }
>  
> @@ -986,22 +993,20 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>      /* Now check each page for real */
>      for ( i = 0; i < count; i++ )
>      {
> -        if ( !map[i] )
> +        if ( !check[i] )
>              continue;

... this can turn into for_each_set_bit() loops.

~Andrew

Re: [PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[Andrew Cooper]

On 07/10/2026 11:40 am, George Dunlap wrote:
> diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
> index 4602c32cff..0b0beca15a 100644
> --- a/xen/arch/x86/mm/p2m-pod.c
> +++ b/xen/arch/x86/mm/p2m-pod.c
> @@ -911,7 +912,12 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>      if ( paging_mode_shadow(d) )
>          max_ref++;
>  
> -    /* First, get the gfn list, translate to mfns, and map the pages. */
> +    /*
> +     * First, get the gfn list, translate to mfns, and pick the pages to
> +     * check.  They are mapped one at a time below, as in
> +     * p2m_pod_zero_check_superpage(): a vCPU can hold only a few transient
> +     * mappings at once, and the p2m lookups and updates here need some too.
> +     */

Actually, I think this wants almost entirely dropping.   Yes, "and maps
the pages." needs editing, and should probably say "and check for being
a plausible candidate" or so, which was missing previously.

The rest is a justification of why we do mapping in the same way we do
it generally, and not useful to have in the source code.

~Andrew

Re: [PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[George Dunlap]

On Wed, Oct 7, 2026 at 1:38 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
> On 07/10/2026 11:40 am, George Dunlap wrote:
> > diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
> > index 4602c32cff..0b0beca15a 100644
> > --- a/xen/arch/x86/mm/p2m-pod.c
> > +++ b/xen/arch/x86/mm/p2m-pod.c
> > @@ -911,7 +912,12 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
> >      if ( paging_mode_shadow(d) )
> >          max_ref++;
> >
> > -    /* First, get the gfn list, translate to mfns, and map the pages. */
> > +    /*
> > +     * First, get the gfn list, translate to mfns, and pick the pages to
> > +     * check.  They are mapped one at a time below, as in
> > +     * p2m_pod_zero_check_superpage(): a vCPU can hold only a few transient
> > +     * mappings at once, and the p2m lookups and updates here need some too.
> > +     */
>
> Actually, I think this wants almost entirely dropping.   Yes, "and maps
> the pages." needs editing, and should probably say "and check for being
> a plausible candidate" or so, which was missing previously.
>
> The rest is a justification of why we do mapping in the same way we do
> it generally, and not useful to have in the source code.

Yes, sorry about the verbose comment, I forgot to go back and give
this a more detailed review.  I'll trim back the comment.

Re the bool array: I assumed the compiler would turn that into a
bitmask.  That not being the case, I'll switch to bitops.

Thanks,
 -George

Re: [PATCH v3 01/18] x86/PoD: map one page at a time in p2m_pod_zero_check()

[Andrew Cooper]

On 08/10/2026 12:09 pm, George Dunlap wrote:
> On Wed, Oct 7, 2026 at 1:38 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 07/10/2026 11:40 am, George Dunlap wrote:
>>> diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
>>> index 4602c32cff..0b0beca15a 100644
>>> --- a/xen/arch/x86/mm/p2m-pod.c
>>> +++ b/xen/arch/x86/mm/p2m-pod.c
>>> @@ -911,7 +912,12 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>>>      if ( paging_mode_shadow(d) )
>>>          max_ref++;
>>>
>>> -    /* First, get the gfn list, translate to mfns, and map the pages. */
>>> +    /*
>>> +     * First, get the gfn list, translate to mfns, and pick the pages to
>>> +     * check.  They are mapped one at a time below, as in
>>> +     * p2m_pod_zero_check_superpage(): a vCPU can hold only a few transient
>>> +     * mappings at once, and the p2m lookups and updates here need some too.
>>> +     */
>> Actually, I think this wants almost entirely dropping.   Yes, "and maps
>> the pages." needs editing, and should probably say "and check for being
>> a plausible candidate" or so, which was missing previously.
>>
>> The rest is a justification of why we do mapping in the same way we do
>> it generally, and not useful to have in the source code.
> Yes, sorry about the verbose comment, I forgot to go back and give
> this a more detailed review.  I'll trim back the comment.

Thanks.

>
> Re the bool array: I assumed the compiler would turn that into a
> bitmask.  That not being the case, I'll switch to bitops.

I'm not aware of any optimisation pass which could make that transform.

https://godbolt.org/z/d76qW4Tzz

GCC will convert the (1 << x) expressions into BT{,S,R} when it can see
they're all in range, but it can't turn the bool array into a bitmap.

~Andrew

[PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

We want to change per-domain mappings to be per-vCPU mappings.  In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.

There are two calls made from domain-scoped contexts; both calls turn
out to be unnecessary:

- destroy_perdomain_mapping() is not logically the undo of
  create_perdomain_mapping(), as the name and its use in
  hvm_domain_initialise() suggest.  create_ allocates a per-domain L3,
  but destroy_ tears down mappings without freeing it; and since the
  call here passes nr == 0, it tears down nothing at all.  The
  per-domain L3 page is actually freed by free_perdomain_mappings(),
  which hvm_domain_initialise()'s caller, arch_domain_create(),
  already invokes on its failure path.

- The call in pv_domain_destroy() is redundant at both of its call
  sites.  arch_domain_destroy() unconditionally calls
  free_perdomain_mappings(), which tears down the same entries and
  additionally frees the page-table structures; and the other caller,
  the error path of pv_domain_initialise(), returns into
  arch_domain_create()'s failure path, which invokes
  free_perdomain_mappings() as well.

One vCPU-scoped caller, pv_destroy_gdt_ldt_l1tab(), is covered the
same way: a vCPU is only ever destroyed together with its domain, or
on the failure path of creating it, and both routes end in
free_perdomain_mappings(); the GDT/LDT slots hold no area-owned pages,
so there is nothing else for the call to do.  While we're getting rid
of unneccessary callers, drop this one too.

Note that free_compat_arg_xlat() keeps its explicit teardown: in
addition to vCPU teardown, it serves switch_compat()'s undo path,
where the domain lives on as a 64-bit one and the argument-translation
pages have to go.  Add a comment to make sure it isn't missed.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- Note that pv_domain_destroy()'s other call site, the error path of
  pv_domain_initialise(), is covered the same way (Jan).
- Also drop pv_destroy_gdt_ldt_l1tab(), the per-vCPU teardown call
  (Jan), and comment on why free_compat_arg_xlat() keeps its own.  The
  Reviewed-by tags given for the previous form are dropped.

Changes in v2:
- Added to the series

Changes since the previously posted version:
- Reworked the commit message to make it more clear how it fits in
  with the larger series.  No functional change.
---
 xen/arch/x86/hvm/hvm.c   |  1 -
 xen/arch/x86/pv/domain.c | 10 ----------
 xen/arch/x86/x86_64/mm.c |  5 +++++
 3 files changed, 5 insertions(+), 11 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 192309c2fc..3924c1b204 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -730,7 +730,6 @@ int hvm_domain_initialise(struct domain *d,
     XFREE(d->arch.hvm.irq);
  fail0:
     hvm_destroy_cacheattr_region_list(d);
-    destroy_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0);
  fail:
     hvm_domain_relinquish_resources(d);
     XFREE(d->arch.hvm.io_handler);
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 03262a4952..07ea4bb4d2 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -332,12 +332,6 @@ static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
                                     NULL);
 }
 
-static void pv_destroy_gdt_ldt_l1tab(struct vcpu *v)
-{
-    destroy_perdomain_mapping(v->domain, GDT_VIRT_START(v),
-                              1U << GDT_LDT_VCPU_SHIFT);
-}
-
 void pv_vcpu_destroy(struct vcpu *v)
 {
     if ( is_pv_32bit_vcpu(v) )
@@ -346,7 +340,6 @@ void pv_vcpu_destroy(struct vcpu *v)
         release_compat_l4(v);
     }
 
-    pv_destroy_gdt_ldt_l1tab(v);
     XFREE(v->arch.pv.trap_ctxt);
 }
 
@@ -394,9 +387,6 @@ void pv_domain_destroy(struct domain *d)
 {
     pv_l1tf_domain_destroy(d);
 
-    destroy_perdomain_mapping(d, GDT_LDT_VIRT_START,
-                              GDT_LDT_MBYTES << (20 - PAGE_SHIFT));
-
     XFREE(d->arch.pv.cpuidmasks);
 
     FREE_XENHEAP_PAGE(d->arch.pv.gdt_ldt_l1tab);
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 8eadab7933..2a1805964d 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -737,6 +737,11 @@ int setup_compat_arg_xlat(struct vcpu *v)
                                     NULL, NIL(struct page_info *));
 }
 
+/*
+ * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
+ * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
+ * there, so the translation area has to go right away.
+ */
 void free_compat_arg_xlat(struct vcpu *v)
 {
     destroy_perdomain_mapping(v->domain, ARG_XLAT_START(v),
-- 
2.55.0

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:35AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> We want to change per-domain mappings to be per-vCPU mappings.  In
> preparation for that, we want to arrange that
> destroy_perdomain_mapping() work either with a single perdomain area,
> or with a per-vCPU perdomain area.
> 
> There are two calls made from domain-scoped contexts; both calls turn
> out to be unnecessary:
> 
> - destroy_perdomain_mapping() is not logically the undo of
>   create_perdomain_mapping(), as the name and its use in
>   hvm_domain_initialise() suggest.  create_ allocates a per-domain L3,
>   but destroy_ tears down mappings without freeing it; and since the
>   call here passes nr == 0, it tears down nothing at all.  The
>   per-domain L3 page is actually freed by free_perdomain_mappings(),
>   which hvm_domain_initialise()'s caller, arch_domain_create(),
>   already invokes on its failure path.
> 
> - The call in pv_domain_destroy() is redundant at both of its call
>   sites.  arch_domain_destroy() unconditionally calls
>   free_perdomain_mappings(), which tears down the same entries and
>   additionally frees the page-table structures; and the other caller,
>   the error path of pv_domain_initialise(), returns into
>   arch_domain_create()'s failure path, which invokes
>   free_perdomain_mappings() as well.
> 
> One vCPU-scoped caller, pv_destroy_gdt_ldt_l1tab(), is covered the
> same way: a vCPU is only ever destroyed together with its domain, or
> on the failure path of creating it, and both routes end in
> free_perdomain_mappings(); the GDT/LDT slots hold no area-owned pages,
> so there is nothing else for the call to do.  While we're getting rid
> of unneccessary callers, drop this one too.
> 
> Note that free_compat_arg_xlat() keeps its explicit teardown: in
> addition to vCPU teardown, it serves switch_compat()'s undo path,
> where the domain lives on as a 64-bit one and the argument-translation
> pages have to go.  Add a comment to make sure it isn't missed.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - Note that pv_domain_destroy()'s other call site, the error path of
>   pv_domain_initialise(), is covered the same way (Jan).
> - Also drop pv_destroy_gdt_ldt_l1tab(), the per-vCPU teardown call
>   (Jan), and comment on why free_compat_arg_xlat() keeps its own.  The
>   Reviewed-by tags given for the previous form are dropped.
> 
> Changes in v2:
> - Added to the series
> 
> Changes since the previously posted version:
> - Reworked the commit message to make it more clear how it fits in
>   with the larger series.  No functional change.
> ---
>  xen/arch/x86/hvm/hvm.c   |  1 -
>  xen/arch/x86/pv/domain.c | 10 ----------
>  xen/arch/x86/x86_64/mm.c |  5 +++++
>  3 files changed, 5 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
> index 192309c2fc..3924c1b204 100644
> --- a/xen/arch/x86/hvm/hvm.c
> +++ b/xen/arch/x86/hvm/hvm.c
> @@ -730,7 +730,6 @@ int hvm_domain_initialise(struct domain *d,
>      XFREE(d->arch.hvm.irq);
>   fail0:
>      hvm_destroy_cacheattr_region_list(d);
> -    destroy_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0);
>   fail:
>      hvm_domain_relinquish_resources(d);
>      XFREE(d->arch.hvm.io_handler);
> diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
> index 03262a4952..07ea4bb4d2 100644
> --- a/xen/arch/x86/pv/domain.c
> +++ b/xen/arch/x86/pv/domain.c
> @@ -332,12 +332,6 @@ static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
>                                      NULL);
>  }
>  
> -static void pv_destroy_gdt_ldt_l1tab(struct vcpu *v)
> -{
> -    destroy_perdomain_mapping(v->domain, GDT_VIRT_START(v),
> -                              1U << GDT_LDT_VCPU_SHIFT);
> -}
> -
>  void pv_vcpu_destroy(struct vcpu *v)
>  {
>      if ( is_pv_32bit_vcpu(v) )
> @@ -346,7 +340,6 @@ void pv_vcpu_destroy(struct vcpu *v)
>          release_compat_l4(v);
>      }
>  
> -    pv_destroy_gdt_ldt_l1tab(v);
>      XFREE(v->arch.pv.trap_ctxt);
>  }
>  
> @@ -394,9 +387,6 @@ void pv_domain_destroy(struct domain *d)
>  {
>      pv_l1tf_domain_destroy(d);
>  
> -    destroy_perdomain_mapping(d, GDT_LDT_VIRT_START,
> -                              GDT_LDT_MBYTES << (20 - PAGE_SHIFT));
> -
>      XFREE(d->arch.pv.cpuidmasks);
>  
>      FREE_XENHEAP_PAGE(d->arch.pv.gdt_ldt_l1tab);
> diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
> index 8eadab7933..2a1805964d 100644
> --- a/xen/arch/x86/x86_64/mm.c
> +++ b/xen/arch/x86/x86_64/mm.c
> @@ -737,6 +737,11 @@ int setup_compat_arg_xlat(struct vcpu *v)
>                                      NULL, NIL(struct page_info *));
>  }
>  
> +/*
> + * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
> + * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
> + * there, so the translation area has to go right away.

I'm unsure we want to be that verbose, but if so I would mention PV
explicitly, possible: "the PV domain lives ..."

Otherwise LGTM, but I don't think I can RB or Ack it being also the
author.

Thanks, Roger.

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[Jan Beulich]

On 07.10.2026 16:06, Roger Pau Monné wrote:
> Otherwise LGTM, but I don't think I can RB or Ack it being also the
> author.

You can, as per earlier discussions of the topic. George's S-o-b effectively
implies an R-b on the code he's taken unchanged. Your R-b then covers the
parts he has changed (and the overall result). That fulfills the 2-pairs-of-
eyes requirement.

Jan

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[George Dunlap]

On Wed, Oct 7, 2026 at 3:24 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 07.10.2026 16:06, Roger Pau Monné wrote:
> > Otherwise LGTM, but I don't think I can RB or Ack it being also the
> > author.
>
> You can, as per earlier discussions of the topic. George's S-o-b effectively
> implies an R-b on the code he's taken unchanged. Your R-b then covers the
> parts he has changed (and the overall result). That fulfills the 2-pairs-of-
> eyes requirement.

Yes -- relevant section from MAINTAINERS quoted below.  So basically,
the tags should look like:

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger.pau@amd.com>

OK to add your R-b, Roger, pending comment tweaks?

 -Georeg

8<---

In the case where two people collaborate on a patch, at least one of
whom is a maintainer -- typically where one maintainer will do an
early version of the patch, and another maintainer will pick it up and
revise it -- there should be two Signed-off-by's and one Acked-by or
Reviewed-by; with the maintainer who did the most recent change
sending the patch, and an Acked-by or Reviewed-by coming from the
maintainer who did not most recently edit the patch.  This satisfies
the requirement #2 because a) the Signed-off-by of the sender approves
the final version of the patch; including all parts of the patch that
the sender did not write b) the Reviewed-by approves the final version
of the patch, including all patches that the reviewer did not write.
Thus all code in the patch has been approved by someone who did not
write it.

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> --- a/xen/arch/x86/x86_64/mm.c
> +++ b/xen/arch/x86/x86_64/mm.c
> @@ -737,6 +737,11 @@ int setup_compat_arg_xlat(struct vcpu *v)
>                                      NULL, NIL(struct page_info *));
>  }
>  
> +/*
> + * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
> + * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
> + * there, so the translation area has to go right away.
> + */

Is it really "has to"? The xlat area is simply unused for 64-bit guests,
so it would be merely "needlessly occupying resources" if we deferred
freeing until domain destruction. Preferably with a respective adjustment
(and perhaps also the adjustment suggested by Roger):
Reviewed-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[Roger Pau Monné]

On Thu, Oct 08, 2026 at 09:55:46AM +0200, Jan Beulich wrote:
> On 07.10.2026 12:40, George Dunlap wrote:
> > --- a/xen/arch/x86/x86_64/mm.c
> > +++ b/xen/arch/x86/x86_64/mm.c
> > @@ -737,6 +737,11 @@ int setup_compat_arg_xlat(struct vcpu *v)
> >                                      NULL, NIL(struct page_info *));
> >  }
> >  
> > +/*
> > + * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
> > + * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
> > + * there, so the translation area has to go right away.
> > + */
> 
> Is it really "has to"? The xlat area is simply unused for 64-bit guests,
> so it would be merely "needlessly occupying resources" if we deferred
> freeing until domain destruction.

I wondered the same, but pv_vcpu_destroy() won't free the xlat are if
is_pv_32bit_vcpu() returns false, which would be the case if
switch_compat() fails.  All this could be adjusted, but it seemed more
churn than benefit.

Roger.

Re: [PATCH v3 02/18] x86/mm: purge unneeded destroy_perdomain_mapping()

[George Dunlap]

On Thu, Oct 8, 2026 at 9:31 AM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Thu, Oct 08, 2026 at 09:55:46AM +0200, Jan Beulich wrote:
> > On 07.10.2026 12:40, George Dunlap wrote:
> > > --- a/xen/arch/x86/x86_64/mm.c
> > > +++ b/xen/arch/x86/x86_64/mm.c
> > > @@ -737,6 +737,11 @@ int setup_compat_arg_xlat(struct vcpu *v)
> > >                                      NULL, NIL(struct page_info *));
> > >  }
> > >
> > > +/*
> > > + * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
> > > + * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
> > > + * there, so the translation area has to go right away.
> > > + */
> >
> > Is it really "has to"? The xlat area is simply unused for 64-bit guests,
> > so it would be merely "needlessly occupying resources" if we deferred
> > freeing until domain destruction.
>
> I wondered the same, but pv_vcpu_destroy() won't free the xlat are if
> is_pv_32bit_vcpu() returns false, which would be the case if
> switch_compat() fails.  All this could be adjusted, but it seemed more
> churn than benefit.

But free_perdomain_mappings() would still free them; so as Jan says,
it's just about hanging on to unnecessary resources.

I was really on the fence about the comment at all; but I think it
should probably go.  There's little chance of someone removing it
later without checking, and even if they did, the only result would be
a few pages held unnecessarily; in the common case that would be
almost immediately.  (It's theoretically possible for a toolstack to
keep a 64-bit domain around after the 32-bit domain creation failed,
but it would be very strange.)

 -George

[PATCH v3 03/18] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

We want to change per-domain mappings to be per-vCPU mappings.  In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.

The one remaining caller, free_compat_arg_xlat(), is already in a vCPU
context, so we just need to change the parameter from a domain pointer
to a vCPU pointer.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
  HVM-only series): free_compat_arg_xlat() is the only caller left.
- Drop the linear-page-table fast path altogether (Jan): its only user
  would have been guest-driven LDT teardown, and without it the
  function does not depend on the linear mapping being available.

Changes in v2:
- Added to the series

Changes since the previously posted version:
- The fast path is gone (v3), and with it the previously posted
  version's sync_local_execstate() call.
- Reword and retitle for clarity (was: "x86/mm: switch
  destroy_perdomain_mapping() parameter from domain to vCPU").
---
 xen/arch/x86/include/asm/mm.h | 2 +-
 xen/arch/x86/mm.c             | 3 ++-
 xen/arch/x86/x86_64/mm.c      | 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 2254a7e3fe..b19b1ae53f 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -606,7 +606,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
 int create_perdomain_mapping(struct domain *d, unsigned long va,
                              unsigned int nr, l1_pgentry_t **pl1tab,
                              struct page_info **ppg);
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
                                unsigned int nr);
 void free_perdomain_mappings(struct domain *d);
 
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index b158742408..769b9b3016 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6334,10 +6334,11 @@ int create_perdomain_mapping(struct domain *d, unsigned long va,
     return rc;
 }
 
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
                                unsigned int nr)
 {
     const l3_pgentry_t *l3tab, *pl3e;
+    const struct domain *d = v->domain;
 
     ASSERT(va >= PERDOMAIN_VIRT_START &&
            va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 2a1805964d..8c576acdc7 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -744,7 +744,7 @@ int setup_compat_arg_xlat(struct vcpu *v)
  */
 void free_compat_arg_xlat(struct vcpu *v)
 {
-    destroy_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+    destroy_perdomain_mapping(v, ARG_XLAT_START(v),
                               PFN_UP(COMPAT_ARG_XLAT_SIZE));
 }
 
-- 
2.55.0

Re: [PATCH v3 03/18] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:36AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> We want to change per-domain mappings to be per-vCPU mappings.  In
> preparation for that, we want to arrange that
> destroy_perdomain_mapping() work either with a single perdomain area,
> or with a per-vCPU perdomain area.
> 
> The one remaining caller, free_compat_arg_xlat(), is already in a vCPU
> context, so we just need to change the parameter from a domain pointer
> to a vCPU pointer.

Without the linear mapping usage this is now a non-functional change,
I would possibly state that on the commit message.

> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Reviewed-by: Roger Pau Monné <roger@xenproject.org>

Thanks, Roger.

Re: [PATCH v3 03/18] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas

[George Dunlap]

On Wed, Oct 7, 2026 at 4:58 PM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:36AM +0100, George Dunlap wrote:
> > From: Roger Pau Monné <roger.pau@citrix.com>
> >
> > We want to change per-domain mappings to be per-vCPU mappings.  In
> > preparation for that, we want to arrange that
> > destroy_perdomain_mapping() work either with a single perdomain area,
> > or with a per-vCPU perdomain area.
> >
> > The one remaining caller, free_compat_arg_xlat(), is already in a vCPU
> > context, so we just need to change the parameter from a domain pointer
> > to a vCPU pointer.
>
> Without the linear mapping usage this is now a non-functional change,
> I would possibly state that on the commit message.

Ack.

> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> > Signed-off-by: George Dunlap <gwd@xenproject.org>
>
> Reviewed-by: Roger Pau Monné <roger@xenproject.org>

Thanks,
 -George

[PATCH v3 04/18] x86/domain_page: drop redundant create_perdomain_mapping() call

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

We want to change per-domain mappings to be per-vCPU mappings.  In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.

One of the current calls in a domain context turns out to be
unnecessary:

mapcache_domain_init() pre-plumbs L1 tables over the whole
inuse/garbage bitmap range (sized for the full MAPCACHE_ENTRIES
capacity), without populating any data pages.  The plumbing is
redundant: mapcache_vcpu_init() installs the bitmap pages the domain
will actually use with populating calls, which allocate any missing
page-table structure on demand -- and since d->max_vcpus is fixed
before any vCPU is created, the range those calls cover never grows.
The pre-plumbed tail beyond it backs virtual addresses that are never
populated at all.

In debug builds, and on hosts with memory hotplug or with more memory
than the PV directmap covers, the call was also what first allocated a
PV domain's per-domain L3.  Release builds on other hosts already
allocated it only with the first vCPU.

Drop the call; now we always allocate only with the first vCPU.

With the only fallible operation gone, mapcache_domain_init() becomes
void, and arch_domain_create()'s error handling for it goes away.

The per-domain-mapping work in mapcache_vcpu_init() stays where it is
rather than moving into mapcache_domain_init() along with this change,
ready for a later patch which introduces per-vCPU mapcaches.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
  HVM-only series); no functional difference.
- Say why the per-vCPU part stays in mapcache_vcpu_init() (Jan).
- Say that the call also first allocated a PV domain's per-domain L3 in
  some builds, which all builds now do with the first vCPU.

Changes in v2:
- Added to the series (split out of the following patch).

Changes since the previously posted version:
- Split out of "x86/mm: switch {create,destroy}_perdomain_mapping()
  domain parameter to vCPU", where the removal was folded into the
  parameter switch without its own rationale.
---
 xen/arch/x86/domain.c             | 3 +--
 xen/arch/x86/domain_page.c        | 8 ++------
 xen/arch/x86/include/asm/domain.h | 2 +-
 3 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 996b50af7a..85cedc3d03 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -908,8 +908,7 @@ int arch_domain_create(struct domain *d,
     }
     else if ( is_pv_domain(d) )
     {
-        if ( (rc = mapcache_domain_init(d)) != 0 )
-            goto fail;
+        mapcache_domain_init(d);
 
         if ( (rc = pv_domain_initialise(d)) != 0 )
             goto fail;
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 72c00194f3..3c90acd2bc 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -223,7 +223,7 @@ void unmap_domain_page(const void *ptr)
     local_irq_restore(flags);
 }
 
-int mapcache_domain_init(struct domain *d)
+void mapcache_domain_init(struct domain *d)
 {
     struct mapcache_domain *dcache = &d->arch.pv.mapcache;
     unsigned int bitmap_pages;
@@ -232,7 +232,7 @@ int mapcache_domain_init(struct domain *d)
 
 #ifdef NDEBUG
     if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
-        return 0;
+        return;
 #endif
 
     BUILD_BUG_ON(MAPCACHE_VIRT_END + PAGE_SIZE * (3 +
@@ -244,10 +244,6 @@ int mapcache_domain_init(struct domain *d)
                       (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
 
     spin_lock_init(&dcache->lock);
-
-    return create_perdomain_mapping(d, (unsigned long)dcache->inuse,
-                                    2 * bitmap_pages + 1,
-                                    NIL(l1_pgentry_t *), NULL);
 }
 
 int mapcache_vcpu_init(struct vcpu *v)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 6cb76781e4..c32dec793a 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -89,7 +89,7 @@ struct mapcache_domain {
     unsigned long *garbage;
 };
 
-int mapcache_domain_init(struct domain *d);
+void mapcache_domain_init(struct domain *d);
 int mapcache_vcpu_init(struct vcpu *v);
 
 /* x86/64: toggle guest between kernel and user modes. */
-- 
2.55.0

Re: [PATCH v3 04/18] x86/domain_page: drop redundant create_perdomain_mapping() call

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:37AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> We want to change per-domain mappings to be per-vCPU mappings.  In
> preparation for that, we want to arrange that
> create_perdomain_mapping() work either with a single perdomain area,
> or with a per-vCPU perdomain area.
> 
> One of the current calls in a domain context turns out to be
> unnecessary:
> 
> mapcache_domain_init() pre-plumbs L1 tables over the whole
> inuse/garbage bitmap range (sized for the full MAPCACHE_ENTRIES
> capacity), without populating any data pages.  The plumbing is
> redundant: mapcache_vcpu_init() installs the bitmap pages the domain
> will actually use with populating calls, which allocate any missing
> page-table structure on demand -- and since d->max_vcpus is fixed
> before any vCPU is created, the range those calls cover never grows.
> The pre-plumbed tail beyond it backs virtual addresses that are never
> populated at all.
> 
> In debug builds, and on hosts with memory hotplug or with more memory
> than the PV directmap covers, the call was also what first allocated a
> PV domain's per-domain L3.  Release builds on other hosts already
> allocated it only with the first vCPU.
> 
> Drop the call; now we always allocate only with the first vCPU.
> 
> With the only fallible operation gone, mapcache_domain_init() becomes
> void, and arch_domain_create()'s error handling for it goes away.
> 
> The per-domain-mapping work in mapcache_vcpu_init() stays where it is
> rather than moving into mapcache_domain_init() along with this change,
> ready for a later patch which introduces per-vCPU mapcaches.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Reviewed-by: Roger Pau Monné <roger@xenproject.org>

Thanks, Roger.

[PATCH v3 05/18] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

We want to change per-domain mappings to be per-vCPU mappings.  In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.

Most of the remaining callers are already in a vCPU context.  This is
no accident: the perdomain area has always been laid out in per-vCPU
slices -- each vCPU has its own GDT/LDT window, its own
COMPAT_ARG_XLAT pages, its own window of mapcache entries -- and each
vCPU's slice is set up as that vCPU is created.  For these callers, we
just need to change the parameter from a domain pointer to a vCPU
pointer.  Once the perdomain area itself becomes per-vCPU, the same
calls will populate the owning vCPU's own area rather than slices of a
shared one.

One exception is the call in hvm_domain_initialise().  An HVM vCPU's
monitor table is created during vCPU initialisation, and
init_xen_l4_slots() stamps the perdomain slot into it at that point --
far earlier than for PV, where the Xen slots are written only once
guest page tables are built.  hvm_domain_initialise() therefore had an
explicit create_perdomain_mapping() call just to make the perdomain
root exist ahead of that.  Move it to arch_vcpu_create(), covering HVM
and PV alike.  With a single shared area, the call allocates at most
once per domain; but once each vCPU has its own perdomain area, this
is the call that will allocate every vCPU's root before any page
tables referencing it are built.  vCPU creation is
where the call must end up; move it there directly.  For PV guests
nothing observable changes: the root was already being created during
vCPU creation as a side effect (by mapcache_vcpu_init(), or failing
that pv_create_gdt_ldt_l1tab()); it now merely becomes explicit.

Note that we cannot yet do a parallel movement of
free_perdomain_mappings(): the per-domain page-table hierarchy is
still a single domain-wide structure shared by all vCPUs, so tearing
it down from a per-vCPU path would pull the mappings out from under
sibling vCPUs (e.g.  on a partially failed, retryable
XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
destruction to free a partially set up hierarchy.  Teardown will move
to vCPU scope only once the structure itself becomes per-vCPU.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
  HVM-only series): the capture parameters stay, as the PV GDT/LDT
  stash still uses them.

Changes in v2:
- Added to the series

Changes since the previously posted version:
- Keep free_perdomain_mappings() (and hence perdomain teardown)
  domain-scoped.
- Keep the idle domain without a perdomain area.
- Retitle (was: "x86/mm: switch {create,destroy}_perdomain_mapping()
  domain parameter to vCPU"); destroy_perdomain_mapping() was switched
  in a separate patch.
- Split the removal of mapcache_domain_init()'s redundant
  create_perdomain_mapping() call into its own (preceding) patch.
---
 xen/arch/x86/domain.c         | 10 ++++++++++
 xen/arch/x86/domain_page.c    |  6 +++---
 xen/arch/x86/hvm/hvm.c        |  5 -----
 xen/arch/x86/include/asm/mm.h |  2 +-
 xen/arch/x86/mm.c             |  3 ++-
 xen/arch/x86/pv/domain.c      |  2 +-
 xen/arch/x86/x86_64/mm.c      |  2 +-
 7 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 85cedc3d03..b670cea52b 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -517,6 +517,16 @@ int arch_vcpu_create(struct vcpu *v)
 
     if ( !is_idle_domain(d) )
     {
+        /*
+         * Make sure the per-domain L3 exists ahead of any consumer (e.g.
+         * init_xen_l4_slots() for the HVM monitor tables): with
+         * create_perdomain_mapping() taking a vCPU this can no longer be
+         * done when creating the domain.
+         */
+        rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
+        if ( rc )
+            return rc;
+
         paging_vcpu_init(v);
 
         if ( (rc = vcpu_init_fpu(v)) != 0 )
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 3c90acd2bc..9119f735f5 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -260,15 +260,15 @@ int mapcache_vcpu_init(struct vcpu *v)
     if ( ents > dcache->entries )
     {
         /* Populate page tables. */
-        int rc = create_perdomain_mapping(d, MAPCACHE_VIRT_START, ents,
+        int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
                                           NIL(l1_pgentry_t *), NULL);
 
         /* Populate bit maps. */
         if ( !rc )
-            rc = create_perdomain_mapping(d, (unsigned long)dcache->inuse,
+            rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
                                           nr, NULL, NIL(struct page_info *));
         if ( !rc )
-            rc = create_perdomain_mapping(d, (unsigned long)dcache->garbage,
+            rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
                                           nr, NULL, NIL(struct page_info *));
 
         if ( rc )
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 3924c1b204..5572728db3 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -620,10 +620,6 @@ int hvm_domain_initialise(struct domain *d,
     INIT_LIST_HEAD(&d->arch.hvm.mmcfg_regions);
     INIT_LIST_HEAD(&d->arch.hvm.msix_tables);
 
-    rc = create_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0, NULL, NULL);
-    if ( rc )
-        goto fail;
-
     hvm_init_cacheattr_region_list(d);
 
     rc = paging_enable(d, PG_refcounts|PG_translate|PG_external);
@@ -730,7 +726,6 @@ int hvm_domain_initialise(struct domain *d,
     XFREE(d->arch.hvm.irq);
  fail0:
     hvm_destroy_cacheattr_region_list(d);
- fail:
     hvm_domain_relinquish_resources(d);
     XFREE(d->arch.hvm.io_handler);
     XFREE(d->arch.hvm.pl_time);
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index b19b1ae53f..87bbe1a6f5 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -603,7 +603,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
 #define NIL(type) ((type *)-sizeof(type))
 #define IS_NIL(ptr) (!((uintptr_t)(ptr) + sizeof(*(ptr))))
 
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
                              unsigned int nr, l1_pgentry_t **pl1tab,
                              struct page_info **ppg);
 void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 769b9b3016..ca22107cd3 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6211,10 +6211,11 @@ static bool perdomain_l1e_needs_freeing(l1_pgentry_t l1e)
            (_PAGE_PRESENT | _PAGE_AVAIL0);
 }
 
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
                              unsigned int nr, l1_pgentry_t **pl1tab,
                              struct page_info **ppg)
 {
+    struct domain *d = v->domain;
     struct page_info *pg;
     l3_pgentry_t *l3tab;
     l2_pgentry_t *l2tab;
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 07ea4bb4d2..9877771cd4 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -326,7 +326,7 @@ int switch_compat(struct domain *d)
 
 static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
 {
-    return create_perdomain_mapping(v->domain, GDT_VIRT_START(v),
+    return create_perdomain_mapping(v, GDT_VIRT_START(v),
                                     1U << GDT_LDT_VCPU_SHIFT,
                                     v->domain->arch.pv.gdt_ldt_l1tab,
                                     NULL);
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 8c576acdc7..0f9e0469ef 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -732,7 +732,7 @@ void __init zap_low_mappings(void)
 
 int setup_compat_arg_xlat(struct vcpu *v)
 {
-    return create_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+    return create_perdomain_mapping(v, ARG_XLAT_START(v),
                                     PFN_UP(COMPAT_ARG_XLAT_SIZE),
                                     NULL, NIL(struct page_info *));
 }
-- 
2.55.0

Re: [PATCH v3 05/18] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:38AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> We want to change per-domain mappings to be per-vCPU mappings.  In
> preparation for that, we want to arrange that
> create_perdomain_mapping() work either with a single perdomain area,
> or with a per-vCPU perdomain area.
> 
> Most of the remaining callers are already in a vCPU context.  This is
> no accident: the perdomain area has always been laid out in per-vCPU
> slices -- each vCPU has its own GDT/LDT window, its own
> COMPAT_ARG_XLAT pages, its own window of mapcache entries -- and each
> vCPU's slice is set up as that vCPU is created.  For these callers, we
> just need to change the parameter from a domain pointer to a vCPU
> pointer.  Once the perdomain area itself becomes per-vCPU, the same
> calls will populate the owning vCPU's own area rather than slices of a
> shared one.
> 
> One exception is the call in hvm_domain_initialise().  An HVM vCPU's
> monitor table is created during vCPU initialisation, and
> init_xen_l4_slots() stamps the perdomain slot into it at that point --
> far earlier than for PV, where the Xen slots are written only once
> guest page tables are built.  hvm_domain_initialise() therefore had an
> explicit create_perdomain_mapping() call just to make the perdomain
> root exist ahead of that.  Move it to arch_vcpu_create(), covering HVM
> and PV alike.  With a single shared area, the call allocates at most
> once per domain; but once each vCPU has its own perdomain area, this
> is the call that will allocate every vCPU's root before any page
> tables referencing it are built.  vCPU creation is
> where the call must end up; move it there directly.

I think it's clear from the paragraph above the call must be moved
there, there's likely no need to re-iterate that fact.

> For PV guests
> nothing observable changes: the root was already being created during
> vCPU creation as a side effect (by mapcache_vcpu_init(), or failing
> that pv_create_gdt_ldt_l1tab()); it now merely becomes explicit.
> 
> Note that we cannot yet do a parallel movement of
> free_perdomain_mappings(): the per-domain page-table hierarchy is
> still a single domain-wide structure shared by all vCPUs, so tearing
> it down from a per-vCPU path would pull the mappings out from under
> sibling vCPUs (e.g.  on a partially failed, retryable
> XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
> destruction to free a partially set up hierarchy.  Teardown will move
> to vCPU scope only once the structure itself becomes per-vCPU.

Even then - we will still need to support non-ASI (PV domains) not
using a per-vCPU slot, and hence the call can't be moved as-is just
when per-vCPU support is available?

> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - Rebased onto staging without the GDT/LDT conversions (for the
>   HVM-only series): the capture parameters stay, as the PV GDT/LDT
>   stash still uses them.
> 
> Changes in v2:
> - Added to the series
> 
> Changes since the previously posted version:
> - Keep free_perdomain_mappings() (and hence perdomain teardown)
>   domain-scoped.
> - Keep the idle domain without a perdomain area.
> - Retitle (was: "x86/mm: switch {create,destroy}_perdomain_mapping()
>   domain parameter to vCPU"); destroy_perdomain_mapping() was switched
>   in a separate patch.
> - Split the removal of mapcache_domain_init()'s redundant
>   create_perdomain_mapping() call into its own (preceding) patch.
> ---
>  xen/arch/x86/domain.c         | 10 ++++++++++
>  xen/arch/x86/domain_page.c    |  6 +++---
>  xen/arch/x86/hvm/hvm.c        |  5 -----
>  xen/arch/x86/include/asm/mm.h |  2 +-
>  xen/arch/x86/mm.c             |  3 ++-
>  xen/arch/x86/pv/domain.c      |  2 +-
>  xen/arch/x86/x86_64/mm.c      |  2 +-
>  7 files changed, 18 insertions(+), 12 deletions(-)
> 
> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> index 85cedc3d03..b670cea52b 100644
> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -517,6 +517,16 @@ int arch_vcpu_create(struct vcpu *v)
>  
>      if ( !is_idle_domain(d) )
>      {
> +        /*
> +         * Make sure the per-domain L3 exists ahead of any consumer (e.g.
> +         * init_xen_l4_slots() for the HVM monitor tables): with
> +         * create_perdomain_mapping() taking a vCPU this can no longer be
> +         * done when creating the domain.

I think the comment here is not very useful - it's not helpful that
the comment here references the previous position of the call.  It
only makes sense in the context of this change, but not for someone
looking at the code after this commit has landed.  I would terminate
the commit at the ":".

The code itself looks good to me.

Thanks, Roger.

Re: [PATCH v3 05/18] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas

[George Dunlap]

On Wed, Oct 7, 2026 at 5:32 PM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:38AM +0100, George Dunlap wrote:
> > From: Roger Pau Monné <roger.pau@citrix.com>
> >
> > We want to change per-domain mappings to be per-vCPU mappings.  In
> > preparation for that, we want to arrange that
> > create_perdomain_mapping() work either with a single perdomain area,
> > or with a per-vCPU perdomain area.
> >
> > Most of the remaining callers are already in a vCPU context.  This is
> > no accident: the perdomain area has always been laid out in per-vCPU
> > slices -- each vCPU has its own GDT/LDT window, its own
> > COMPAT_ARG_XLAT pages, its own window of mapcache entries -- and each
> > vCPU's slice is set up as that vCPU is created.  For these callers, we
> > just need to change the parameter from a domain pointer to a vCPU
> > pointer.  Once the perdomain area itself becomes per-vCPU, the same
> > calls will populate the owning vCPU's own area rather than slices of a
> > shared one.
> >
> > One exception is the call in hvm_domain_initialise().  An HVM vCPU's
> > monitor table is created during vCPU initialisation, and
> > init_xen_l4_slots() stamps the perdomain slot into it at that point --
> > far earlier than for PV, where the Xen slots are written only once
> > guest page tables are built.  hvm_domain_initialise() therefore had an
> > explicit create_perdomain_mapping() call just to make the perdomain
> > root exist ahead of that.  Move it to arch_vcpu_create(), covering HVM
> > and PV alike.  With a single shared area, the call allocates at most
> > once per domain; but once each vCPU has its own perdomain area, this
> > is the call that will allocate every vCPU's root before any page
> > tables referencing it are built.  vCPU creation is
> > where the call must end up; move it there directly.
>
> I think it's clear from the paragraph above the call must be moved
> there, there's likely no need to re-iterate that fact.
>
> > For PV guests
> > nothing observable changes: the root was already being created during
> > vCPU creation as a side effect (by mapcache_vcpu_init(), or failing
> > that pv_create_gdt_ldt_l1tab()); it now merely becomes explicit.
> >
> > Note that we cannot yet do a parallel movement of
> > free_perdomain_mappings(): the per-domain page-table hierarchy is
> > still a single domain-wide structure shared by all vCPUs, so tearing
> > it down from a per-vCPU path would pull the mappings out from under
> > sibling vCPUs (e.g.  on a partially failed, retryable
> > XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
> > destruction to free a partially set up hierarchy.  Teardown will move
> > to vCPU scope only once the structure itself becomes per-vCPU.
>
> Even then - we will still need to support non-ASI (PV domains) not
> using a per-vCPU slot, and hence the call can't be moved as-is just
> when per-vCPU support is available?
>
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> > Signed-off-by: George Dunlap <gwd@xenproject.org>
> > ---
> > Changes in v3:
> > - Rebased onto staging without the GDT/LDT conversions (for the
> >   HVM-only series): the capture parameters stay, as the PV GDT/LDT
> >   stash still uses them.
> >
> > Changes in v2:
> > - Added to the series
> >
> > Changes since the previously posted version:
> > - Keep free_perdomain_mappings() (and hence perdomain teardown)
> >   domain-scoped.
> > - Keep the idle domain without a perdomain area.
> > - Retitle (was: "x86/mm: switch {create,destroy}_perdomain_mapping()
> >   domain parameter to vCPU"); destroy_perdomain_mapping() was switched
> >   in a separate patch.
> > - Split the removal of mapcache_domain_init()'s redundant
> >   create_perdomain_mapping() call into its own (preceding) patch.
> > ---
> >  xen/arch/x86/domain.c         | 10 ++++++++++
> >  xen/arch/x86/domain_page.c    |  6 +++---
> >  xen/arch/x86/hvm/hvm.c        |  5 -----
> >  xen/arch/x86/include/asm/mm.h |  2 +-
> >  xen/arch/x86/mm.c             |  3 ++-
> >  xen/arch/x86/pv/domain.c      |  2 +-
> >  xen/arch/x86/x86_64/mm.c      |  2 +-
> >  7 files changed, 18 insertions(+), 12 deletions(-)
> >
> > diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> > index 85cedc3d03..b670cea52b 100644
> > --- a/xen/arch/x86/domain.c
> > +++ b/xen/arch/x86/domain.c
> > @@ -517,6 +517,16 @@ int arch_vcpu_create(struct vcpu *v)
> >
> >      if ( !is_idle_domain(d) )
> >      {
> > +        /*
> > +         * Make sure the per-domain L3 exists ahead of any consumer (e.g.
> > +         * init_xen_l4_slots() for the HVM monitor tables): with
> > +         * create_perdomain_mapping() taking a vCPU this can no longer be
> > +         * done when creating the domain.
>
> I think the comment here is not very useful - it's not helpful that
> the comment here references the previous position of the call.  It
> only makes sense in the context of this change, but not for someone
> looking at the code after this commit has landed.  I would terminate
> the commit at the ":".
>
> The code itself looks good to me.

Done, thanks.

 -George

Re: [PATCH v3 05/18] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> We want to change per-domain mappings to be per-vCPU mappings.  In
> preparation for that, we want to arrange that
> create_perdomain_mapping() work either with a single perdomain area,
> or with a per-vCPU perdomain area.
> 
> Most of the remaining callers are already in a vCPU context.  This is
> no accident: the perdomain area has always been laid out in per-vCPU
> slices -- each vCPU has its own GDT/LDT window, its own
> COMPAT_ARG_XLAT pages, its own window of mapcache entries -- and each
> vCPU's slice is set up as that vCPU is created.  For these callers, we
> just need to change the parameter from a domain pointer to a vCPU
> pointer.  Once the perdomain area itself becomes per-vCPU, the same
> calls will populate the owning vCPU's own area rather than slices of a
> shared one.
> 
> One exception is the call in hvm_domain_initialise().  An HVM vCPU's
> monitor table is created during vCPU initialisation, and
> init_xen_l4_slots() stamps the perdomain slot into it at that point --
> far earlier than for PV, where the Xen slots are written only once
> guest page tables are built.  hvm_domain_initialise() therefore had an
> explicit create_perdomain_mapping() call just to make the perdomain
> root exist ahead of that.  Move it to arch_vcpu_create(), covering HVM
> and PV alike.  With a single shared area, the call allocates at most
> once per domain; but once each vCPU has its own perdomain area, this
> is the call that will allocate every vCPU's root before any page
> tables referencing it are built.  vCPU creation is
> where the call must end up; move it there directly.  For PV guests
> nothing observable changes: the root was already being created during
> vCPU creation as a side effect (by mapcache_vcpu_init(), or failing
> that pv_create_gdt_ldt_l1tab()); it now merely becomes explicit.
> 
> Note that we cannot yet do a parallel movement of
> free_perdomain_mappings(): the per-domain page-table hierarchy is
> still a single domain-wide structure shared by all vCPUs, so tearing
> it down from a per-vCPU path would pull the mappings out from under
> sibling vCPUs (e.g.  on a partially failed, retryable
> XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
> destruction to free a partially set up hierarchy.  Teardown will move
> to vCPU scope only once the structure itself becomes per-vCPU.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Preferably with Roger's suggested comment adjustment:
Reviewed-by: Jan Beulich <jbeulich@suse.com>

Jan

[PATCH v3 06/18] x86/spec-ctrl: introduce Address Space Isolation command line option

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

Introduce the `asi=` command line option, and the
opt_vcpu_pt_{hwdom,hvm} knobs plus the per-domain d->arch.vcpu_pt
setting they control.  The option is introduced ahead of the
functionality it enables, so that the newly added code can be keyed on
it from the start; all knobs currently default to off, and enabling any
of them taints the boot with a "not functional, development purposes
only" warning.

The mechanisms this option controls apply to HVM domains: an HVM vCPU
already runs on its own monitor table, so a per-vCPU per-domain area
needs no work at context switch.  d->arch.vcpu_pt is therefore only
ever set for HVM domains, a PVH hardware domain included; PV domains,
a PV hardware domain included, are unaffected.

The boot log gains "ASI features for ..." lines for Dom0 and HVM
domains, so hardware-domain-only configurations remain visible.

Further per-mechanism tokens arrive with their mechanisms in later
patches.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- HVM only: drop the PV knob and its interaction with XPTI.  vCPU-PT
  applies to HVM domains and to a PVH hardware domain; the Dom0 boot
  log line says None for a PV one.

Changes in v2:
- Added to the series

Changes since the previously posted version:
- Include the hardware domain in the development warning and the boot
   log summary.
- Make the opt_vcpu_pt_* knobs plain booleans preinitialised to false,
   dropping the late -1 resolution.
- Documentation: mention possible protection against unmitigated
   attacks, and state that hvm= does not affect the hardware domain.
- Rewrite the commit message.
---
 docs/misc/xen-command-line.pandoc    | 26 +++++++++
 xen/arch/x86/include/asm/domain.h    |  6 +++
 xen/arch/x86/include/asm/spec_ctrl.h |  2 +
 xen/arch/x86/spec_ctrl.c             | 79 ++++++++++++++++++++++++++++
 4 files changed, 113 insertions(+)

diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index b2c94ae56d..785e099c0a 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -202,6 +202,32 @@ to appropriate auditing by Xen.  Argo is disabled by default.
     This option is disabled by default, to protect domains from a DoS by a
     buggy or malicious other domain spamming the ring.
 
+### asi (x86)
+> `= List of [ <bool>, hvm=<bool>, vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
+
+> Default: `false`
+
+Offers control over whether the hypervisor will engage in Address Space
+Isolation, by not having potentially sensitive information permanently mapped
+in the VMM page-tables.  Using this option might avoid the need to apply
+mitigations for certain speculative related attacks, at the cost of mapping
+sensitive information on-demand.  It might also offer some protection against
+unmitigated speculation-related attacks.
+
+The mechanisms currently implemented apply to HVM guests, including a PVH
+hardware domain.  PV guests, including a PV hardware domain, are unaffected.
+
+* `hvm=` enables the features for HVM guests other than the hardware domain,
+  which follows the whole-feature forms (the plain boolean, or an un-suffixed
+  `vcpu-pt=<bool>`).
+
+**WARNING: manual de-selection of enabled options will invalidate any
+protection offered by the feature.  The fine grained options provided below
+are meant to be used for debugging purposes only.**
+
+* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
+  each vCPU's monitor table maps a region private to that vCPU.
+
 ### asid (x86)
 > `= <boolean>`
 
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index c32dec793a..b192c86bc1 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -477,6 +477,12 @@ struct arch_domain
     /* Don't unconditionally inject #GP for unhandled MSRs. */
     bool msr_relaxed;
 
+    /*
+     * Give each vCPU its own per-domain area (HVM only: each vCPU already
+     * runs on its own monitor table).
+     */
+    bool vcpu_pt;
+
     /* Emulated devices enabled bitmap. */
     uint32_t emulation_flags;
 } __cacheline_aligned;
diff --git a/xen/arch/x86/include/asm/spec_ctrl.h b/xen/arch/x86/include/asm/spec_ctrl.h
index 8f82533c41..b62229ab5d 100644
--- a/xen/arch/x86/include/asm/spec_ctrl.h
+++ b/xen/arch/x86/include/asm/spec_ctrl.h
@@ -87,6 +87,8 @@ extern uint8_t default_scf;
 
 extern int8_t opt_xpti_hwdom, opt_xpti_domu;
 
+extern bool opt_vcpu_pt_hwdom, opt_vcpu_pt_hvm;
+
 extern bool cpu_has_bug_l1tf;
 extern int8_t opt_pv_l1tf_hwdom, opt_pv_l1tf_domu;
 extern bool opt_bp_spec_reduce;
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index bc8538a56f..999e41aefa 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -86,6 +86,13 @@ bool __ro_after_init opt_bp_spec_reduce = true;
 
 static bool __initdata opt_ibpb_alt;
 
+/*
+ * Give each vCPU of an HVM domain its own per-domain area (vCPU-PT).  Off by
+ * default until the feature is complete.
+ */
+bool __ro_after_init opt_vcpu_pt_hvm;
+bool __ro_after_init opt_vcpu_pt_hwdom;
+
 static int __init cf_check parse_spec_ctrl(const char *s)
 {
     const char *ss;
@@ -487,6 +494,62 @@ static int __init cf_check parse_pv_l1tf(const char *s)
 }
 custom_param("pv-l1tf", parse_pv_l1tf);
 
+static int __init cf_check parse_asi(const char *s)
+{
+    const char *ss;
+    int val, rc = 0;
+
+    /* Interpret 'asi' alone in its positive boolean form. */
+    if ( *s == '\0' )
+        opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+
+    do {
+        ss = strchr(s, ',');
+        if ( !ss )
+            ss = strchr(s, '\0');
+
+        val = parse_bool(s, ss);
+        switch ( val )
+        {
+        case 0:
+        case 1:
+            opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+            break;
+
+        default:
+            if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+                opt_vcpu_pt_hvm = val;
+            else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
+            {
+                switch ( val )
+                {
+                case 1:
+                case 0:
+                    opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+                    break;
+
+                case -2:
+                    s += strlen("vcpu-pt=");
+                    if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+                        opt_vcpu_pt_hvm = val;
+                    else
+                default:
+                        rc = -EINVAL;
+                    break;
+                }
+            }
+            else if ( *s )
+                rc = -EINVAL;
+            break;
+        }
+
+        s = ss + 1;
+    } while ( *ss );
+
+    return rc;
+}
+custom_param("asi", parse_asi);
+
 static void __init print_details(enum ind_thunk thunk)
 {
     unsigned int _7d0 = 0, _7d2 = 0, e8b = 0, e21a = 0, e21c = 0, max = 0, tmp;
@@ -680,6 +743,14 @@ static void __init print_details(enum ind_thunk thunk)
            opt_pv_l1tf_hwdom ? "enabled"  : "disabled",
            opt_pv_l1tf_domu  ? "enabled"  : "disabled");
 #endif
+
+    /* vCPU-PT is implemented for HVM only: a PV Dom0 gets none of it. */
+    printk("  ASI features for Dom0:%s\n",
+           opt_dom0_pvh && opt_vcpu_pt_hwdom ? " vCPU-PT" : " None");
+#ifdef CONFIG_HVM
+    printk("  ASI features for HVM VMs:%s\n",
+           opt_vcpu_pt_hvm                   ? " vCPU-PT" : " None");
+#endif
 }
 
 static bool __init check_smt_enabled(void)
@@ -1866,6 +1937,9 @@ void spec_ctrl_init_domain(struct domain *d)
     if ( pv )
         d->arch.pv.xpti = is_hardware_domain(d) ? opt_xpti_hwdom
                                                 : opt_xpti_domu;
+
+    d->arch.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_vcpu_pt_hwdom
+                                                    : opt_vcpu_pt_hvm);
 }
 
 void __init init_speculation_mitigations(void)
@@ -2158,6 +2232,11 @@ void __init init_speculation_mitigations(void)
          hw_smt_enabled && default_xen_spec_ctrl )
         setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
 
+    if ( opt_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
+        warning_add(
+            "Address Space Isolation is not functional, this option is\n"
+            "intended to be used only for development purposes.\n");
+
     xpti_init_default();
 
     l1tf_calculations();
-- 
2.55.0

Re: [PATCH v3 06/18] x86/spec-ctrl: introduce Address Space Isolation command line option

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:39AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> Introduce the `asi=` command line option, and the
> opt_vcpu_pt_{hwdom,hvm} knobs plus the per-domain d->arch.vcpu_pt
> setting they control.  The option is introduced ahead of the
> functionality it enables, so that the newly added code can be keyed on
> it from the start; all knobs currently default to off, and enabling any
> of them taints the boot with a "not functional, development purposes
> only" warning.

I would possibly avoid using "taint" above, as we have a taint
mechanism and it might create confusion in the context here.  Using
ASI doesn't taint the hypervisor.

> 
> The mechanisms this option controls apply to HVM domains: an HVM vCPU
> already runs on its own monitor table, so a per-vCPU per-domain area
> needs no work at context switch.  d->arch.vcpu_pt is therefore only
> ever set for HVM domains, a PVH hardware domain included; PV domains,
> a PV hardware domain included, are unaffected.
> 
> The boot log gains "ASI features for ..." lines for Dom0 and HVM
> domains, so hardware-domain-only configurations remain visible.
> 
> Further per-mechanism tokens arrive with their mechanisms in later
> patches.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - HVM only: drop the PV knob and its interaction with XPTI.  vCPU-PT
>   applies to HVM domains and to a PVH hardware domain; the Dom0 boot
>   log line says None for a PV one.
> 
> Changes in v2:
> - Added to the series
> 
> Changes since the previously posted version:
> - Include the hardware domain in the development warning and the boot
>    log summary.
> - Make the opt_vcpu_pt_* knobs plain booleans preinitialised to false,
>    dropping the late -1 resolution.
> - Documentation: mention possible protection against unmitigated
>    attacks, and state that hvm= does not affect the hardware domain.
> - Rewrite the commit message.
> ---
>  docs/misc/xen-command-line.pandoc    | 26 +++++++++
>  xen/arch/x86/include/asm/domain.h    |  6 +++
>  xen/arch/x86/include/asm/spec_ctrl.h |  2 +
>  xen/arch/x86/spec_ctrl.c             | 79 ++++++++++++++++++++++++++++
>  4 files changed, 113 insertions(+)
> 
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index b2c94ae56d..785e099c0a 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -202,6 +202,32 @@ to appropriate auditing by Xen.  Argo is disabled by default.
>      This option is disabled by default, to protect domains from a DoS by a
>      buggy or malicious other domain spamming the ring.
>  
> +### asi (x86)

AFAICT the sparse direct map series contained support for ARM also, I
assume this is being dropped from the content pushed (at least in this
round).  I think we will possibly duplicate the "asi" command line
option between architectures?

> +> `= List of [ <bool>, hvm=<bool>, vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> +
> +> Default: `false`
> +
> +Offers control over whether the hypervisor will engage in Address Space
> +Isolation, by not having potentially sensitive information permanently mapped
> +in the VMM page-tables.  Using this option might avoid the need to apply
> +mitigations for certain speculative related attacks, at the cost of mapping
> +sensitive information on-demand.  It might also offer some protection against
> +unmitigated speculation-related attacks.
> +
> +The mechanisms currently implemented apply to HVM guests, including a PVH
> +hardware domain.  PV guests, including a PV hardware domain, are unaffected.

In the above you mix "guests" and "domains", I would rather use
"domains" consistently: "... apply to HVM domains, including a PVH
hardware domain ...".  Same for the instances below.

> +
> +* `hvm=` enables the features for HVM guests other than the hardware domain,
> +  which follows the whole-feature forms (the plain boolean, or an un-suffixed
> +  `vcpu-pt=<bool>`).
> +
> +**WARNING: manual de-selection of enabled options will invalidate any
> +protection offered by the feature.  The fine grained options provided below
> +are meant to be used for debugging purposes only.**
> +
> +* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
> +  each vCPU's monitor table maps a region private to that vCPU.
> +
>  ### asid (x86)
>  > `= <boolean>`
>  
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index c32dec793a..b192c86bc1 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -477,6 +477,12 @@ struct arch_domain
>      /* Don't unconditionally inject #GP for unhandled MSRs. */
>      bool msr_relaxed;
>  
> +    /*
> +     * Give each vCPU its own per-domain area (HVM only: each vCPU already
> +     * runs on its own monitor table).
> +     */
> +    bool vcpu_pt;

Do we want to place this inside of an "asi" structure, and possibly
make it a single bit bitfield?  I think we expect to add more boolean
ASI related options, so might be nice to have them packed together,
ie:

/* Fine grained ASI related options. */
struct {
    bool vcpu_pt : 1;
} asi;

Thanks, Roger.

Re: [PATCH v3 06/18] x86/spec-ctrl: introduce Address Space Isolation command line option

[George Dunlap]

On Wed, Oct 7, 2026 at 5:52 PM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:39AM +0100, George Dunlap wrote:
> > From: Roger Pau Monné <roger.pau@citrix.com>
> >
> > Introduce the `asi=` command line option, and the
> > opt_vcpu_pt_{hwdom,hvm} knobs plus the per-domain d->arch.vcpu_pt
> > setting they control.  The option is introduced ahead of the
> > functionality it enables, so that the newly added code can be keyed on
> > it from the start; all knobs currently default to off, and enabling any
> > of them taints the boot with a "not functional, development purposes
> > only" warning.
>
> I would possibly avoid using "taint" above, as we have a taint
> mechanism and it might create confusion in the context here.  Using
> ASI doesn't taint the hypervisor.

Ack

> > diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> > index b2c94ae56d..785e099c0a 100644
> > --- a/docs/misc/xen-command-line.pandoc
> > +++ b/docs/misc/xen-command-line.pandoc
> > @@ -202,6 +202,32 @@ to appropriate auditing by Xen.  Argo is disabled by default.
> >      This option is disabled by default, to protect domains from a DoS by a
> >      buggy or malicious other domain spamming the ring.
> >
> > +### asi (x86)
>
> AFAICT the sparse direct map series contained support for ARM also, I
> assume this is being dropped from the content pushed (at least in this
> round).  I think we will possibly duplicate the "asi" command line
> option between architectures?

Yes, I think that's the plan; there are a number of other options we
duplicate.  The exact options and what they mean will probably differ
somewhat as well.  No point asking someone trying to use ASI on ARM to
filter out all the stuff about PV, shadow, and XPTI, for instance.

> > +> `= List of [ <bool>, hvm=<bool>, vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> > +
> > +> Default: `false`
> > +
> > +Offers control over whether the hypervisor will engage in Address Space
> > +Isolation, by not having potentially sensitive information permanently mapped
> > +in the VMM page-tables.  Using this option might avoid the need to apply
> > +mitigations for certain speculative related attacks, at the cost of mapping
> > +sensitive information on-demand.  It might also offer some protection against
> > +unmitigated speculation-related attacks.
> > +
> > +The mechanisms currently implemented apply to HVM guests, including a PVH
> > +hardware domain.  PV guests, including a PV hardware domain, are unaffected.
>
> In the above you mix "guests" and "domains", I would rather use
> "domains" consistently: "... apply to HVM domains, including a PVH
> hardware domain ...".  Same for the instances below.

Ack

> > diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> > index c32dec793a..b192c86bc1 100644
> > --- a/xen/arch/x86/include/asm/domain.h
> > +++ b/xen/arch/x86/include/asm/domain.h
> > @@ -477,6 +477,12 @@ struct arch_domain
> >      /* Don't unconditionally inject #GP for unhandled MSRs. */
> >      bool msr_relaxed;
> >
> > +    /*
> > +     * Give each vCPU its own per-domain area (HVM only: each vCPU already
> > +     * runs on its own monitor table).
> > +     */
> > +    bool vcpu_pt;
>
> Do we want to place this inside of an "asi" structure, and possibly
> make it a single bit bitfield?  I think we expect to add more boolean
> ASI related options, so might be nice to have them packed together,
> ie:
>
> /* Fine grained ASI related options. */
> struct {
>     bool vcpu_pt : 1;
> } asi;

Yes, that's nicer.  A tiny bit of churn but worth it I think.

Thanks,
 -George

Re: [PATCH v3 06/18] x86/spec-ctrl: introduce Address Space Isolation command line option

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> --- a/xen/arch/x86/spec_ctrl.c
> +++ b/xen/arch/x86/spec_ctrl.c
> @@ -86,6 +86,13 @@ bool __ro_after_init opt_bp_spec_reduce = true;
>  
>  static bool __initdata opt_ibpb_alt;
>  
> +/*
> + * Give each vCPU of an HVM domain its own per-domain area (vCPU-PT).  Off by
> + * default until the feature is complete.
> + */
> +bool __ro_after_init opt_vcpu_pt_hvm;
> +bool __ro_after_init opt_vcpu_pt_hwdom;

I'm a little worried of this name, which doesn't distinguish PV and PVH. Yes,
the variable name is already long-ish, yet I think inserting "pvh" is going
to help.

> @@ -487,6 +494,62 @@ static int __init cf_check parse_pv_l1tf(const char *s)
>  }
>  custom_param("pv-l1tf", parse_pv_l1tf);
>  
> +static int __init cf_check parse_asi(const char *s)
> +{
> +    const char *ss;
> +    int val, rc = 0;
> +
> +    /* Interpret 'asi' alone in its positive boolean form. */
> +    if ( *s == '\0' )
> +        opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
> +
> +    do {
> +        ss = strchr(s, ',');
> +        if ( !ss )
> +            ss = strchr(s, '\0');
> +
> +        val = parse_bool(s, ss);
> +        switch ( val )
> +        {
> +        case 0:
> +        case 1:
> +            opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> +            break;
> +
> +        default:
> +            if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> +                opt_vcpu_pt_hvm = val;
> +            else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
> +            {
> +                switch ( val )
> +                {
> +                case 1:
> +                case 0:
> +                    opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> +                    break;
> +
> +                case -2:
> +                    s += strlen("vcpu-pt=");
> +                    if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> +                        opt_vcpu_pt_hvm = val;
> +                    else
> +                default:
> +                        rc = -EINVAL;
> +                    break;

While personally I like the reduced redundancy from placing case labels
like the default: one here, Misra rule 16.2 wants us to not use such.
Note that as written there's no violation, yet iirc Andrew keeps asking
for figure braces in such constructs. I think we're better off not
introducing new instances (the rule is marked clean, so with the braces
added we simply can't, unless the file is excluded from scanning).

Jan

Re: [PATCH v3 06/18] x86/spec-ctrl: introduce Address Space Isolation command line option

[George Dunlap]

On Thu, Oct 8, 2026 at 9:24 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 07.10.2026 12:40, George Dunlap wrote:
> > --- a/xen/arch/x86/spec_ctrl.c
> > +++ b/xen/arch/x86/spec_ctrl.c
> > @@ -86,6 +86,13 @@ bool __ro_after_init opt_bp_spec_reduce = true;
> >
> >  static bool __initdata opt_ibpb_alt;
> >
> > +/*
> > + * Give each vCPU of an HVM domain its own per-domain area (vCPU-PT).  Off by
> > + * default until the feature is complete.
> > + */
> > +bool __ro_after_init opt_vcpu_pt_hvm;
> > +bool __ro_after_init opt_vcpu_pt_hwdom;
>
> I'm a little worried of this name, which doesn't distinguish PV and PVH. Yes,
> the variable name is already long-ish, yet I think inserting "pvh" is going
> to help.

Ack

>
> > @@ -487,6 +494,62 @@ static int __init cf_check parse_pv_l1tf(const char *s)
> >  }
> >  custom_param("pv-l1tf", parse_pv_l1tf);
> >
> > +static int __init cf_check parse_asi(const char *s)
> > +{
> > +    const char *ss;
> > +    int val, rc = 0;
> > +
> > +    /* Interpret 'asi' alone in its positive boolean form. */
> > +    if ( *s == '\0' )
> > +        opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
> > +
> > +    do {
> > +        ss = strchr(s, ',');
> > +        if ( !ss )
> > +            ss = strchr(s, '\0');
> > +
> > +        val = parse_bool(s, ss);
> > +        switch ( val )
> > +        {
> > +        case 0:
> > +        case 1:
> > +            opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> > +            break;
> > +
> > +        default:
> > +            if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> > +                opt_vcpu_pt_hvm = val;
> > +            else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
> > +            {
> > +                switch ( val )
> > +                {
> > +                case 1:
> > +                case 0:
> > +                    opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> > +                    break;
> > +
> > +                case -2:
> > +                    s += strlen("vcpu-pt=");
> > +                    if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> > +                        opt_vcpu_pt_hvm = val;
> > +                    else
> > +                default:
> > +                        rc = -EINVAL;
> > +                    break;
>
> While personally I like the reduced redundancy from placing case labels
> like the default: one here, Misra rule 16.2 wants us to not use such.
> Note that as written there's no violation, yet iirc Andrew keeps asking
> for figure braces in such constructs. I think we're better off not
> introducing new instances (the rule is marked clean, so with the braces
> added we simply can't, unless the file is excluded from scanning).

OK.

Thanks,
 -George

[PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

The per-domain area is currently a single domain-wide structure: one L3,
referenced from every root page-table associated with the domain, so
every mapping in it is visible to every vCPU of the domain.  Its
contents are already laid out in per-vCPU slices (each vCPU's
COMPAT_ARG_XLAT pages, and for PV each vCPU's GDT/LDT window and
mapcache entries), but the visibility is domain-wide.  Meanwhile, much
of the per-vCPU state Xen maintains -- the VMCB, the VMX MSR load/save
areas, FPU/XSAVE state -- lives in always-mapped memory.

Allow the "per-domain" area to be per-vCPU instead ("VCPU-PT") for HVM
domains.  HVM monitor tables are already per-vCPU, so this needs nothing
at context switch: every monitor table carries its vCPU's own L3 from
creation, and the per-vCPU mappings are isolated from other running
vCPUs at once.  We will later build on this, adding a per-vCPU mapcache
and per-vCPU mapped areas, into which we can move the vCPU state
mentioned above.

Add pervcpu_l3_pg to the arch_vcpu struct, to correspond to the
perdomain_l3_pg in the domain struct.  (We retain both so that we can
switch between per-domain and per-vCPU on a domain-by-domain basis; PV
domains always use the domain-wide one.)

In {create,destroy}_perdomain_mapping(), if d->arch.vcpu_pt, use
pervcpu_l3_pg as the per-domain L3 (allocating it for a vCPU if it's
NULL, just as we allocate for a domain in !vcpu_pt mode); otherwise,
use perdomain_l3_pg.  Introduce a helper, perdomain_l3(), to
consistently choose the correct one.

Introduce free_pervcpu_mappings() to free this tree, called from
arch_vcpu_destroy() on normal teardown.  Since the vcpu structure holds
the only reference to pervcpu_l3_pg, arch_vcpu_create() must also call
it on its error paths: nothing else records the allocation once the
vcpu struct is torn down.  The domain-wide free_perdomain_mappings() is
unchanged and keeps covering non-vCPU-PT domains.

Give init_xen_l4_slots() the vCPU a table is private to, and use
perdomain_l3() to select the value to install in slot 260 when one is
given.  The HVM monitor tables and the 32-bit PV Xen-owned L4s pass
their vCPU; tables the domain's vCPUs share -- promoted PV guest L4s,
PV shadow L4s, and the PV dom0 L4 under construction -- pass NULL and
keep the domain-wide L3, which is all a PV domain has.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-fable-5-1, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- HVM only: PV domains keep the domain-wide area, so the PV parts go
  (the single-vCPU limit, the shadow refusals, the XPTI slot install).
  init_xen_l4_slots() takes the vCPU as an extra, optional argument
  rather than in place of the domain, so PV guest L4s, which no vCPU
  owns, are promoted exactly as before.
- Retitle (was: "x86/mm: introduce per-vCPU L3 page-table").
- free_compat_arg_xlat()'s comment names free_pervcpu_mappings() as the
  teardown for a vCPU-PT domain.

Changes in v2:
- Added to the series

Changes since the previously posted version:
- Expand the commit message with the motivation and the design
  rationale.
- Keep domain-wide freeing intact and introduce a vCPU-scoped
  free_pervcpu_mappings() instead of re-scoping
  free_perdomain_mappings(); fix the arch_vcpu_create() error-path
  leaks of a partially built per-vCPU hierarchy.
- Add a perdomain_l3() helper for the root selection, rather than
  open-coding the vcpu_pt choice (and testing both root pointers) at
  each site.
---
 xen/arch/x86/domain.c             | 10 +++--
 xen/arch/x86/include/asm/domain.h | 12 +++++
 xen/arch/x86/include/asm/mm.h     |  5 ++-
 xen/arch/x86/mm.c                 | 73 ++++++++++++++++++++++---------
 xen/arch/x86/mm/hap/hap.c         |  2 +-
 xen/arch/x86/mm/shadow/hvm.c      |  2 +-
 xen/arch/x86/mm/shadow/multi.c    |  5 ++-
 xen/arch/x86/pv/dom0_build.c      |  2 +-
 xen/arch/x86/pv/domain.c          |  2 +-
 xen/arch/x86/x86_64/mm.c          |  7 +--
 10 files changed, 87 insertions(+), 33 deletions(-)

diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index b670cea52b..0aba2cca2e 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -513,7 +513,7 @@ int arch_vcpu_create(struct vcpu *v)
 
     rc = mapcache_vcpu_init(v);
     if ( rc )
-        return rc;
+        goto fail_early;
 
     if ( !is_idle_domain(d) )
     {
@@ -525,12 +525,12 @@ int arch_vcpu_create(struct vcpu *v)
          */
         rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
         if ( rc )
-            return rc;
+            goto fail_early;
 
         paging_vcpu_init(v);
 
         if ( (rc = vcpu_init_fpu(v)) != 0 )
-            return rc;
+            goto fail_early;
 
         vmce_init_vcpu(v);
 
@@ -578,6 +578,8 @@ int arch_vcpu_create(struct vcpu *v)
     vcpu_destroy_fpu(v);
     xfree(v->arch.msrs);
     v->arch.msrs = NULL;
+ fail_early:
+    free_pervcpu_mappings(v);
 
     return rc;
 }
@@ -598,6 +600,8 @@ void arch_vcpu_destroy(struct vcpu *v)
         pv_vcpu_destroy(v);
     else
         ASSERT_UNREACHABLE();
+
+    free_pervcpu_mappings(v);
 }
 
 int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index b192c86bc1..a6c6eaf53a 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -335,6 +335,11 @@ struct monitor_write_data {
 
 struct arch_domain
 {
+    /*
+     * Domain-wide L3 page-table for the L4 per-domain slot, used when
+     * the domain does not use per-vCPU page-tables (!d->arch.vcpu_pt).
+     * NULL otherwise (see v->arch.pervcpu_l3_pg and perdomain_l3()).
+     */
     struct page_info *perdomain_l3_pg;
 
     /* I/O-port admin-specified access capabilities. */
@@ -691,6 +696,13 @@ struct arch_vcpu
 
     struct vcpu_msrs *msrs;
 
+    /*
+     * Per-vCPU L3 page-table for the L4 per-domain slot, used when the
+     * domain uses per-vCPU page-tables (d->arch.vcpu_pt).  NULL
+     * otherwise (see d->arch.perdomain_l3_pg and perdomain_l3()).
+     */
+    struct page_info *pervcpu_l3_pg;
+
     struct {
         bool next_interrupt_enabled;
     } monitor;
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 87bbe1a6f5..87131dd4bf 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -369,8 +369,10 @@ int devalidate_page(struct page_info *page, unsigned long type,
                          int preemptible);
 
 void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
+struct page_info *perdomain_l3(const struct vcpu *v);
 void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
-                       const struct domain *d, mfn_t sl4mfn, bool ro_mpt);
+                       const struct domain *d, const struct vcpu *v,
+                       mfn_t sl4mfn, bool ro_mpt);
 bool fill_ro_mpt(mfn_t mfn);
 void zap_ro_mpt(mfn_t mfn);
 
@@ -609,6 +611,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
 void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
                                unsigned int nr);
 void free_perdomain_mappings(struct domain *d);
+void free_pervcpu_mappings(struct vcpu *v);
 
 void __iomem *ioremap_wc(paddr_t pa, size_t len);
 
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index ca22107cd3..7258935528 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1638,6 +1638,17 @@ static int promote_l3_table(struct page_info *page)
 }
 #endif /* CONFIG_PV */
 
+/*
+ * The root of the per-domain area in use by @v: the vCPU's own L3 for a
+ * vCPU-PT domain, the domain-wide one otherwise.
+ */
+struct page_info *perdomain_l3(const struct vcpu *v)
+{
+    const struct domain *d = v->domain;
+
+    return d->arch.vcpu_pt ? v->arch.pervcpu_l3_pg : d->arch.perdomain_l3_pg;
+}
+
 /*
  * Fill an L4 with Xen entries.
  *
@@ -1645,12 +1656,16 @@ static int promote_l3_table(struct page_info *page)
  * values a guest may have left there from promote_l4_table().
  *
  * l4t, l4mfn, and d are mandatory, but l4mfn doesn't need to be the mfn under
- * *l4t.  All other parameters are optional and will either fill or zero the
- * appropriate slots.  Pagetables not shared with guests will gain the
+ * *l4t.  v names the vCPU the table is private to (an HVM monitor table, a
+ * 32-bit PV vCPU's Xen-owned L4), or is NULL for tables the domain's vCPUs
+ * share; it is mandatory for a vCPU-PT domain, whose per-domain area is
+ * per-vCPU.  All other parameters are optional and will either fill or zero
+ * the appropriate slots.  Pagetables not shared with guests will gain the
  * extended directmap.
  */
 void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
-                       const struct domain *d, mfn_t sl4mfn, bool ro_mpt)
+                       const struct domain *d, const struct vcpu *v,
+                       mfn_t sl4mfn, bool ro_mpt)
 {
     /*
      * PV vcpus need a shortened directmap.  HVM and Idle vcpus get the full
@@ -1678,8 +1693,11 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
         l4e_from_mfn(sl4mfn, __PAGE_HYPERVISOR_RW);
 
     /* Slot 260: Per-domain mappings. */
+    ASSERT(!v || v->domain == d);
+    ASSERT(v || !d->arch.vcpu_pt);
     l4t[l4_table_offset(PERDOMAIN_VIRT_START)] =
-        l4e_from_page(d->arch.perdomain_l3_pg, __PAGE_HYPERVISOR_RW);
+        l4e_from_page(v ? perdomain_l3(v) : d->arch.perdomain_l3_pg,
+                      __PAGE_HYPERVISOR_RW);
 
     /* Slot 4: Per-domain mappings mirror. */
     BUILD_BUG_ON(IS_ENABLED(CONFIG_PV32) &&
@@ -1835,7 +1853,7 @@ static int promote_l4_table(struct page_info *page)
     if ( !rc )
     {
         init_xen_l4_slots(pl4e, l4mfn,
-                          d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
+                          d, NULL, INVALID_MFN, VM_ASSIST(d, m2p_strict));
         atomic_inc(&d->arch.pv.nr_l4_pages);
     }
     unmap_domain_page(pl4e);
@@ -6216,7 +6234,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
                              struct page_info **ppg)
 {
     struct domain *d = v->domain;
-    struct page_info *pg;
+    struct page_info *pg, *l3_pg = perdomain_l3(v);
     l3_pgentry_t *l3tab;
     l2_pgentry_t *l2tab;
     l1_pgentry_t *l1tab;
@@ -6225,14 +6243,17 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
     ASSERT(va >= PERDOMAIN_VIRT_START &&
            va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
 
-    if ( !d->arch.perdomain_l3_pg )
+    if ( !l3_pg )
     {
         pg = alloc_domheap_page(d, MEMF_no_owner);
         if ( !pg )
             return -ENOMEM;
         l3tab = __map_domain_page(pg);
         clear_page(l3tab);
-        d->arch.perdomain_l3_pg = pg;
+        if ( d->arch.vcpu_pt )
+            v->arch.pervcpu_l3_pg = pg;
+        else
+            d->arch.perdomain_l3_pg = pg;
         if ( !nr )
         {
             unmap_domain_page(l3tab);
@@ -6242,7 +6263,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
     else if ( !nr )
         return 0;
     else
-        l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+        l3tab = __map_domain_page(l3_pg);
 
     ASSERT(!l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
 
@@ -6339,16 +6360,16 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
                                unsigned int nr)
 {
     const l3_pgentry_t *l3tab, *pl3e;
-    const struct domain *d = v->domain;
+    struct page_info *l3_pg = perdomain_l3(v);
 
     ASSERT(va >= PERDOMAIN_VIRT_START &&
            va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
     ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
 
-    if ( !d->arch.perdomain_l3_pg )
+    if ( !l3_pg )
         return;
 
-    l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+    l3tab = __map_domain_page(l3_pg);
     pl3e = l3tab + l3_table_offset(va);
 
     if ( l3e_get_flags(*pl3e) & _PAGE_PRESENT )
@@ -6387,16 +6408,11 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
     unmap_domain_page(l3tab);
 }
 
-void free_perdomain_mappings(struct domain *d)
+static void free_perdomain_l3(struct page_info *l3pg)
 {
-    l3_pgentry_t *l3tab;
+    l3_pgentry_t *l3tab = __map_domain_page(l3pg);
     unsigned int i;
 
-    if ( !d->arch.perdomain_l3_pg )
-        return;
-
-    l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
-
     for ( i = 0; i < PERDOMAIN_SLOTS; ++i)
         if ( l3e_get_flags(l3tab[i]) & _PAGE_PRESENT )
         {
@@ -6432,10 +6448,27 @@ void free_perdomain_mappings(struct domain *d)
         }
 
     unmap_domain_page(l3tab);
-    free_domheap_page(d->arch.perdomain_l3_pg);
+    free_domheap_page(l3pg);
+}
+
+void free_perdomain_mappings(struct domain *d)
+{
+    if ( !d->arch.perdomain_l3_pg )
+        return;
+
+    free_perdomain_l3(d->arch.perdomain_l3_pg);
     d->arch.perdomain_l3_pg = NULL;
 }
 
+void free_pervcpu_mappings(struct vcpu *v)
+{
+    if ( !v->arch.pervcpu_l3_pg )
+        return;
+
+    free_perdomain_l3(v->arch.pervcpu_l3_pg);
+    v->arch.pervcpu_l3_pg = NULL;
+}
+
 static void write_sss_token(unsigned long *ptr)
 {
     /*
diff --git a/xen/arch/x86/mm/hap/hap.c b/xen/arch/x86/mm/hap/hap.c
index 0ede4181a0..9fc59f0702 100644
--- a/xen/arch/x86/mm/hap/hap.c
+++ b/xen/arch/x86/mm/hap/hap.c
@@ -407,7 +407,7 @@ static mfn_t hap_make_monitor_table(struct vcpu *v)
     m4mfn = page_to_mfn(pg);
     l4e = map_domain_page(m4mfn);
 
-    init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+    init_xen_l4_slots(l4e, m4mfn, d, v, INVALID_MFN, false);
     unmap_domain_page(l4e);
 
     return m4mfn;
diff --git a/xen/arch/x86/mm/shadow/hvm.c b/xen/arch/x86/mm/shadow/hvm.c
index e6fb97c4b6..833e378e30 100644
--- a/xen/arch/x86/mm/shadow/hvm.c
+++ b/xen/arch/x86/mm/shadow/hvm.c
@@ -760,7 +760,7 @@ mfn_t sh_make_monitor_table(const struct vcpu *v, unsigned int shadow_levels)
      * shadow-linear mapping will either be inserted below when creating
      * lower level monitor tables, or later in sh_update_cr3().
      */
-    init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+    init_xen_l4_slots(l4e, m4mfn, d, v, INVALID_MFN, false);
 
     if ( shadow_levels < 4 )
     {
diff --git a/xen/arch/x86/mm/shadow/multi.c b/xen/arch/x86/mm/shadow/multi.c
index 7f72b4b7b2..b2017c1010 100644
--- a/xen/arch/x86/mm/shadow/multi.c
+++ b/xen/arch/x86/mm/shadow/multi.c
@@ -974,8 +974,9 @@ sh_make_shadow(struct vcpu *v, mfn_t gmfn, u32 shadow_type)
 
             BUILD_BUG_ON(sizeof(l4_pgentry_t) != sizeof(shadow_l4e_t));
 
-            init_xen_l4_slots(l4t, gmfn, d, smfn, (!is_pv_32bit_domain(d) &&
-                                                   VM_ASSIST(d, m2p_strict)));
+            init_xen_l4_slots(l4t, gmfn, d, NULL, smfn,
+                              (!is_pv_32bit_domain(d) &&
+                               VM_ASSIST(d, m2p_strict)));
             unmap_domain_page(l4t);
         }
         break;
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index ddeb144b06..bf980eba8c 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -726,7 +726,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
         l4start = l4tab = __va(mpt_alloc); mpt_alloc += PAGE_SIZE;
         clear_page(l4tab);
         init_xen_l4_slots(l4tab, _mfn(virt_to_mfn(l4start)),
-                          d, INVALID_MFN, true);
+                          d, NULL, INVALID_MFN, true);
         v->arch.guest_table = pagetable_from_paddr(__pa(l4start));
     }
     else
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 9877771cd4..c1611fdcff 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
     mfn = page_to_mfn(pg);
     l4tab = map_domain_page(mfn);
     clear_page(l4tab);
-    init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
+    init_xen_l4_slots(l4tab, mfn, v->domain, v, INVALID_MFN, false);
     unmap_domain_page(l4tab);
 
     /* This page needs to look like a pagetable so that it can be shadowed */
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 0f9e0469ef..baee0a9cfe 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -738,9 +738,10 @@ int setup_compat_arg_xlat(struct vcpu *v)
 }
 
 /*
- * Besides vCPU teardown, which free_perdomain_mappings() would cover, this
- * serves switch_compat()'s undo path: the domain lives on as a 64-bit one
- * there, so the translation area has to go right away.
+ * Besides vCPU teardown, which the per-domain area's own teardown would
+ * cover (free_perdomain_mappings(), or free_pervcpu_mappings() for a
+ * vCPU-PT domain), this serves switch_compat()'s undo path: the domain lives
+ * on as a 64-bit one there, so the translation area has to go right away.
  */
 void free_compat_arg_xlat(struct vcpu *v)
 {
-- 
2.55.0

Re: [PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:40AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> The per-domain area is currently a single domain-wide structure: one L3,

nit: s/structure/entry/ might be better?

> referenced from every root page-table associated with the domain, so
> every mapping in it is visible to every vCPU of the domain.  Its
> contents are already laid out in per-vCPU slices (each vCPU's
> COMPAT_ARG_XLAT pages, and for PV each vCPU's GDT/LDT window and
> mapcache entries), but the visibility is domain-wide.  Meanwhile, much
> of the per-vCPU state Xen maintains -- the VMCB, the VMX MSR load/save
> areas, FPU/XSAVE state -- lives in always-mapped memory.
> 
> Allow the "per-domain" area to be per-vCPU instead ("VCPU-PT") for HVM
> domains.  HVM monitor tables are already per-vCPU, so this needs nothing
> at context switch: every monitor table carries its vCPU's own L3 from
> creation, and the per-vCPU mappings are isolated from other running
> vCPUs at once.  We will later build on this, adding a per-vCPU mapcache
> and per-vCPU mapped areas, into which we can move the vCPU state
> mentioned above.
> 
> Add pervcpu_l3_pg to the arch_vcpu struct, to correspond to the
> perdomain_l3_pg in the domain struct.  (We retain both so that we can
> switch between per-domain and per-vCPU on a domain-by-domain basis; PV
> domains always use the domain-wide one.)
> 
> In {create,destroy}_perdomain_mapping(), if d->arch.vcpu_pt, use
> pervcpu_l3_pg as the per-domain L3 (allocating it for a vCPU if it's
> NULL, just as we allocate for a domain in !vcpu_pt mode); otherwise,
> use perdomain_l3_pg.  Introduce a helper, perdomain_l3(), to
> consistently choose the correct one.
> 
> Introduce free_pervcpu_mappings() to free this tree, called from
> arch_vcpu_destroy() on normal teardown.  Since the vcpu structure holds
> the only reference to pervcpu_l3_pg, arch_vcpu_create() must also call
> it on its error paths: nothing else records the allocation once the
> vcpu struct is torn down.  The domain-wide free_perdomain_mappings() is
> unchanged and keeps covering non-vCPU-PT domains.
> 
> Give init_xen_l4_slots() the vCPU a table is private to, and use

Can't you replace the domain parameter with a vcpu one, and obtain
domain as v->domain?

> perdomain_l3() to select the value to install in slot 260 when one is
> given.  The HVM monitor tables and the 32-bit PV Xen-owned L4s pass
> their vCPU; tables the domain's vCPUs share -- promoted PV guest L4s,
> PV shadow L4s, and the PV dom0 L4 under construction -- pass NULL and
> keep the domain-wide L3, which is all a PV domain has.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-fable-5-1, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - HVM only: PV domains keep the domain-wide area, so the PV parts go
>   (the single-vCPU limit, the shadow refusals, the XPTI slot install).
>   init_xen_l4_slots() takes the vCPU as an extra, optional argument
>   rather than in place of the domain, so PV guest L4s, which no vCPU
>   owns, are promoted exactly as before.

Hm, I see, so this is the reason. PV L4s are promoted ahead of being
loaded into a vCPU %cr3, and hence when init_xen_l4_slots() is called
the per-vCPU L3 to use is not clear.

> - Retitle (was: "x86/mm: introduce per-vCPU L3 page-table").
> - free_compat_arg_xlat()'s comment names free_pervcpu_mappings() as the
>   teardown for a vCPU-PT domain.
> 
> Changes in v2:
> - Added to the series
> 
> Changes since the previously posted version:
> - Expand the commit message with the motivation and the design
>   rationale.
> - Keep domain-wide freeing intact and introduce a vCPU-scoped
>   free_pervcpu_mappings() instead of re-scoping
>   free_perdomain_mappings(); fix the arch_vcpu_create() error-path
>   leaks of a partially built per-vCPU hierarchy.
> - Add a perdomain_l3() helper for the root selection, rather than
>   open-coding the vcpu_pt choice (and testing both root pointers) at
>   each site.
> ---
>  xen/arch/x86/domain.c             | 10 +++--
>  xen/arch/x86/include/asm/domain.h | 12 +++++
>  xen/arch/x86/include/asm/mm.h     |  5 ++-
>  xen/arch/x86/mm.c                 | 73 ++++++++++++++++++++++---------
>  xen/arch/x86/mm/hap/hap.c         |  2 +-
>  xen/arch/x86/mm/shadow/hvm.c      |  2 +-
>  xen/arch/x86/mm/shadow/multi.c    |  5 ++-
>  xen/arch/x86/pv/dom0_build.c      |  2 +-
>  xen/arch/x86/pv/domain.c          |  2 +-
>  xen/arch/x86/x86_64/mm.c          |  7 +--
>  10 files changed, 87 insertions(+), 33 deletions(-)
> 
> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> index b670cea52b..0aba2cca2e 100644
> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -513,7 +513,7 @@ int arch_vcpu_create(struct vcpu *v)
>  
>      rc = mapcache_vcpu_init(v);
>      if ( rc )
> -        return rc;
> +        goto fail_early;
>  
>      if ( !is_idle_domain(d) )
>      {
> @@ -525,12 +525,12 @@ int arch_vcpu_create(struct vcpu *v)
>           */
>          rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
>          if ( rc )
> -            return rc;
> +            goto fail_early;
>  
>          paging_vcpu_init(v);
>  
>          if ( (rc = vcpu_init_fpu(v)) != 0 )
> -            return rc;
> +            goto fail_early;
>  
>          vmce_init_vcpu(v);
>  
> @@ -578,6 +578,8 @@ int arch_vcpu_create(struct vcpu *v)
>      vcpu_destroy_fpu(v);
>      xfree(v->arch.msrs);
>      v->arch.msrs = NULL;
> + fail_early:
> +    free_pervcpu_mappings(v);

Do we really need the extra label?  It would seem to me like the
existing cleanup functions are harmless to execute regardless of
whether the matching initialization routine has been called.

>  
>      return rc;
>  }
> @@ -598,6 +600,8 @@ void arch_vcpu_destroy(struct vcpu *v)
>          pv_vcpu_destroy(v);
>      else
>          ASSERT_UNREACHABLE();
> +
> +    free_pervcpu_mappings(v);
>  }
>  
>  int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index b192c86bc1..a6c6eaf53a 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -335,6 +335,11 @@ struct monitor_write_data {
>  
>  struct arch_domain
>  {
> +    /*
> +     * Domain-wide L3 page-table for the L4 per-domain slot, used when
> +     * the domain does not use per-vCPU page-tables (!d->arch.vcpu_pt).
> +     * NULL otherwise (see v->arch.pervcpu_l3_pg and perdomain_l3()).
> +     */
>      struct page_info *perdomain_l3_pg;
>  
>      /* I/O-port admin-specified access capabilities. */
> @@ -691,6 +696,13 @@ struct arch_vcpu
>  
>      struct vcpu_msrs *msrs;
>  
> +    /*
> +     * Per-vCPU L3 page-table for the L4 per-domain slot, used when the
> +     * domain uses per-vCPU page-tables (d->arch.vcpu_pt).  NULL
> +     * otherwise (see d->arch.perdomain_l3_pg and perdomain_l3()).
> +     */
> +    struct page_info *pervcpu_l3_pg;
> +
>      struct {
>          bool next_interrupt_enabled;
>      } monitor;
> diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
> index 87bbe1a6f5..87131dd4bf 100644
> --- a/xen/arch/x86/include/asm/mm.h
> +++ b/xen/arch/x86/include/asm/mm.h
> @@ -369,8 +369,10 @@ int devalidate_page(struct page_info *page, unsigned long type,
>                           int preemptible);
>  
>  void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
> +struct page_info *perdomain_l3(const struct vcpu *v);
>  void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
> -                       const struct domain *d, mfn_t sl4mfn, bool ro_mpt);
> +                       const struct domain *d, const struct vcpu *v,
> +                       mfn_t sl4mfn, bool ro_mpt);
>  bool fill_ro_mpt(mfn_t mfn);
>  void zap_ro_mpt(mfn_t mfn);
>  
> @@ -609,6 +611,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
>  void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
>                                 unsigned int nr);
>  void free_perdomain_mappings(struct domain *d);
> +void free_pervcpu_mappings(struct vcpu *v);
>  
>  void __iomem *ioremap_wc(paddr_t pa, size_t len);
>  
> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
> index ca22107cd3..7258935528 100644
> --- a/xen/arch/x86/mm.c
> +++ b/xen/arch/x86/mm.c
> @@ -1638,6 +1638,17 @@ static int promote_l3_table(struct page_info *page)
>  }
>  #endif /* CONFIG_PV */
>  
> +/*
> + * The root of the per-domain area in use by @v: the vCPU's own L3 for a
> + * vCPU-PT domain, the domain-wide one otherwise.
> + */
> +struct page_info *perdomain_l3(const struct vcpu *v)
> +{
> +    const struct domain *d = v->domain;
> +
> +    return d->arch.vcpu_pt ? v->arch.pervcpu_l3_pg : d->arch.perdomain_l3_pg;
> +}
> +
>  /*
>   * Fill an L4 with Xen entries.
>   *
> @@ -1645,12 +1656,16 @@ static int promote_l3_table(struct page_info *page)
>   * values a guest may have left there from promote_l4_table().
>   *
>   * l4t, l4mfn, and d are mandatory, but l4mfn doesn't need to be the mfn under
> - * *l4t.  All other parameters are optional and will either fill or zero the
> - * appropriate slots.  Pagetables not shared with guests will gain the
> + * *l4t.  v names the vCPU the table is private to (an HVM monitor table, a
> + * 32-bit PV vCPU's Xen-owned L4), or is NULL for tables the domain's vCPUs
> + * share; it is mandatory for a vCPU-PT domain, whose per-domain area is
> + * per-vCPU.  All other parameters are optional and will either fill or zero
> + * the appropriate slots.  Pagetables not shared with guests will gain the
>   * extended directmap.
>   */
>  void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
> -                       const struct domain *d, mfn_t sl4mfn, bool ro_mpt)
> +                       const struct domain *d, const struct vcpu *v,
> +                       mfn_t sl4mfn, bool ro_mpt)
>  {
>      /*
>       * PV vcpus need a shortened directmap.  HVM and Idle vcpus get the full
> @@ -1678,8 +1693,11 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
>          l4e_from_mfn(sl4mfn, __PAGE_HYPERVISOR_RW);
>  
>      /* Slot 260: Per-domain mappings. */
> +    ASSERT(!v || v->domain == d);
> +    ASSERT(v || !d->arch.vcpu_pt);
>      l4t[l4_table_offset(PERDOMAIN_VIRT_START)] =
> -        l4e_from_page(d->arch.perdomain_l3_pg, __PAGE_HYPERVISOR_RW);
> +        l4e_from_page(v ? perdomain_l3(v) : d->arch.perdomain_l3_pg,
> +                      __PAGE_HYPERVISOR_RW);
>  
>      /* Slot 4: Per-domain mappings mirror. */
>      BUILD_BUG_ON(IS_ENABLED(CONFIG_PV32) &&
> @@ -1835,7 +1853,7 @@ static int promote_l4_table(struct page_info *page)
>      if ( !rc )
>      {
>          init_xen_l4_slots(pl4e, l4mfn,
> -                          d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
> +                          d, NULL, INVALID_MFN, VM_ASSIST(d, m2p_strict));
>          atomic_inc(&d->arch.pv.nr_l4_pages);
>      }
>      unmap_domain_page(pl4e);
> @@ -6216,7 +6234,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
>                               struct page_info **ppg)
>  {
>      struct domain *d = v->domain;
> -    struct page_info *pg;
> +    struct page_info *pg, *l3_pg = perdomain_l3(v);
>      l3_pgentry_t *l3tab;
>      l2_pgentry_t *l2tab;
>      l1_pgentry_t *l1tab;
> @@ -6225,14 +6243,17 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
>      ASSERT(va >= PERDOMAIN_VIRT_START &&
>             va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
>  
> -    if ( !d->arch.perdomain_l3_pg )
> +    if ( !l3_pg )
>      {
>          pg = alloc_domheap_page(d, MEMF_no_owner);
>          if ( !pg )
>              return -ENOMEM;
>          l3tab = __map_domain_page(pg);
>          clear_page(l3tab);
> -        d->arch.perdomain_l3_pg = pg;
> +        if ( d->arch.vcpu_pt )
> +            v->arch.pervcpu_l3_pg = pg;
> +        else
> +            d->arch.perdomain_l3_pg = pg;
>          if ( !nr )
>          {
>              unmap_domain_page(l3tab);
> @@ -6242,7 +6263,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
>      else if ( !nr )
>          return 0;
>      else
> -        l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
> +        l3tab = __map_domain_page(l3_pg);
>  
>      ASSERT(!l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
>  
> @@ -6339,16 +6360,16 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
>                                 unsigned int nr)
>  {
>      const l3_pgentry_t *l3tab, *pl3e;
> -    const struct domain *d = v->domain;
> +    struct page_info *l3_pg = perdomain_l3(v);
>  
>      ASSERT(va >= PERDOMAIN_VIRT_START &&
>             va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
>      ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
>  
> -    if ( !d->arch.perdomain_l3_pg )
> +    if ( !l3_pg )
>          return;
>  
> -    l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
> +    l3tab = __map_domain_page(l3_pg);
>      pl3e = l3tab + l3_table_offset(va);
>  
>      if ( l3e_get_flags(*pl3e) & _PAGE_PRESENT )
> @@ -6387,16 +6408,11 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
>      unmap_domain_page(l3tab);
>  }
>  
> -void free_perdomain_mappings(struct domain *d)
> +static void free_perdomain_l3(struct page_info *l3pg)
>  {
> -    l3_pgentry_t *l3tab;
> +    l3_pgentry_t *l3tab = __map_domain_page(l3pg);
>      unsigned int i;
>  
> -    if ( !d->arch.perdomain_l3_pg )
> -        return;
> -
> -    l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
> -
>      for ( i = 0; i < PERDOMAIN_SLOTS; ++i)
>          if ( l3e_get_flags(l3tab[i]) & _PAGE_PRESENT )
>          {
> @@ -6432,10 +6448,27 @@ void free_perdomain_mappings(struct domain *d)
>          }
>  
>      unmap_domain_page(l3tab);
> -    free_domheap_page(d->arch.perdomain_l3_pg);
> +    free_domheap_page(l3pg);
> +}
> +
> +void free_perdomain_mappings(struct domain *d)
> +{
> +    if ( !d->arch.perdomain_l3_pg )
> +        return;
> +
> +    free_perdomain_l3(d->arch.perdomain_l3_pg);
>      d->arch.perdomain_l3_pg = NULL;
>  }
>  
> +void free_pervcpu_mappings(struct vcpu *v)
> +{
> +    if ( !v->arch.pervcpu_l3_pg )
> +        return;
> +
> +    free_perdomain_l3(v->arch.pervcpu_l3_pg);
> +    v->arch.pervcpu_l3_pg = NULL;

We have been (recently) following the model of first setting the
pointer to NULL, and then freeing it, ie:

struct page_info *l3_pg = v->arch.pervcpu_l3_pg;

if ( !l3_pg )
    return;

v->arch.pervcpu_l3_pg = NULL;
free_perdomain_l3(l3_pg);

Like XFREE() and similar macros.  It's possibly worth using the same
pattern here.

Thanks, Roger.

Re: [PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[George Dunlap]

On Wed, Oct 7, 2026 at 8:21 PM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:40AM +0100, George Dunlap wrote:
> > From: Roger Pau Monné <roger.pau@citrix.com>
> >
> > The per-domain area is currently a single domain-wide structure: one L3,
>
> nit: s/structure/entry/ might be better?

Not sure why?  After this change, there's still a single perdomain
entry; but each vCPU has its own tree, whereas previously there was
only a single tree.  A tree is a structure.

I could change it to "tree" if you want, but I'm not sure that's any clearer.

> > Give init_xen_l4_slots() the vCPU a table is private to, and use
>
> Can't you replace the domain parameter with a vcpu one, and obtain
> domain as v->domain?
[snip]
-
> > Changes in v3:
> > - HVM only: PV domains keep the domain-wide area, so the PV parts go
> >   (the single-vCPU limit, the shadow refusals, the XPTI slot install).
> >   init_xen_l4_slots() takes the vCPU as an extra, optional argument
> >   rather than in place of the domain, so PV guest L4s, which no vCPU
> >   owns, are promoted exactly as before.
>
> Hm, I see, so this is the reason. PV L4s are promoted ahead of being
> loaded into a vCPU %cr3, and hence when init_xen_l4_slots() is called
> the per-vCPU L3 to use is not clear.

Right, so your v2 passed d->vcpu[0] in a couple of places where there
was no current cpu (e.g., dom0 creation).

I think the reason this version ended up this way is that in theory,
the toolstack could do something that would cause promote_l4_table()
to be called before the vcpus are created (by XEN_DOMCTL_max_vcpus).
If we have init_xen_l4_slots() take only a vcpu argument, we'd have to
modify that to check that d->vcpu[0] existed, and fail the call if
not; and any toolstack that did promote an l4 before creating the
vCPUs would stop working.

As it happens, libxc / libxl always call max_vcpus first.  So I think
failing the call would be OK.  Then most of the sites would just take
a single argument, and Jan's question about 32-bit PV guests just goes
away.

I'll give that a try and see how it looks, unless someone has an objection.

> > @@ -525,12 +525,12 @@ int arch_vcpu_create(struct vcpu *v)
> >           */
> >          rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
> >          if ( rc )
> > -            return rc;
> > +            goto fail_early;
> >
> >          paging_vcpu_init(v);
> >
> >          if ( (rc = vcpu_init_fpu(v)) != 0 )
> > -            return rc;
> > +            goto fail_early;
> >
> >          vmce_init_vcpu(v);
> >
> > @@ -578,6 +578,8 @@ int arch_vcpu_create(struct vcpu *v)
> >      vcpu_destroy_fpu(v);
> >      xfree(v->arch.msrs);
> >      v->arch.msrs = NULL;
> > + fail_early:
> > +    free_pervcpu_mappings(v);
>
> Do we really need the extra label?  It would seem to me like the
> existing cleanup functions are harmless to execute regardless of
> whether the matching initialization routine has been called.

Huh, just noticed that all the fail paths call all the cleanup
functions, regardless of whether they need to be cleaned up.  OK,
yeah, we can do the same thing.

> > +    free_perdomain_l3(v->arch.pervcpu_l3_pg);
> > +    v->arch.pervcpu_l3_pg = NULL;
>
> We have been (recently) following the model of first setting the
> pointer to NULL, and then freeing it, ie:
>
> struct page_info *l3_pg = v->arch.pervcpu_l3_pg;
>
> if ( !l3_pg )
>     return;
>
> v->arch.pervcpu_l3_pg = NULL;
> free_perdomain_l3(l3_pg);
>
> Like XFREE() and similar macros.  It's possibly worth using the same
> pattern here.

Ack.

 -George

Re: [PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> @@ -1835,7 +1853,7 @@ static int promote_l4_table(struct page_info *page)
>      if ( !rc )
>      {
>          init_xen_l4_slots(pl4e, l4mfn,
> -                          d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
> +                          d, NULL, INVALID_MFN, VM_ASSIST(d, m2p_strict));

Nit: Take the opportunity and re-flow the statement (i.e. move more args
onto the 1st line)?

> --- a/xen/arch/x86/pv/domain.c
> +++ b/xen/arch/x86/pv/domain.c
> @@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
>      mfn = page_to_mfn(pg);
>      l4tab = map_domain_page(mfn);
>      clear_page(l4tab);
> -    init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
> +    init_xen_l4_slots(l4tab, mfn, v->domain, v, INVALID_MFN, false);

This being PV-only code, don't you want to pass NULL here? The description
talks about this, but (a) it's not quite clear why PV32 would want handling
differently from PV64 at this point and (b) in sh_make_shadow() you do pass
NULL unconditionally.

Jan

Re: [PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[George Dunlap]

On Thu, Oct 8, 2026 at 9:40 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 07.10.2026 12:40, George Dunlap wrote:
> > @@ -1835,7 +1853,7 @@ static int promote_l4_table(struct page_info *page)
> >      if ( !rc )
> >      {
> >          init_xen_l4_slots(pl4e, l4mfn,
> > -                          d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
> > +                          d, NULL, INVALID_MFN, VM_ASSIST(d, m2p_strict));
>
> Nit: Take the opportunity and re-flow the statement (i.e. move more args
> onto the 1st line)?

Ack

> > --- a/xen/arch/x86/pv/domain.c
> > +++ b/xen/arch/x86/pv/domain.c
> > @@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
> >      mfn = page_to_mfn(pg);
> >      l4tab = map_domain_page(mfn);
> >      clear_page(l4tab);
> > -    init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
> > +    init_xen_l4_slots(l4tab, mfn, v->domain, v, INVALID_MFN, false);
>
> This being PV-only code, don't you want to pass NULL here? The description
> talks about this, but (a) it's not quite clear why PV32 would want handling
> differently from PV64 at this point and (b) in sh_make_shadow() you do pass
> NULL unconditionally.

This is actually left over from the previous iteration of the series
which supports PV guests; and apparently it would be a lot easier for
implement per-vCPU pagetables for 32-bit PV guests, since the L4
doesn't belong to the guest.  But it's outside of the scope of the
current work.

At any rate, I'm going to try passing a single vcpu, as Roger
suggested.  That restricts toolstacks to not being able to promote PV
L4s before calling XEN_DOMCTL_max_vcpus, but that seems fine to me.

 -George

Re: [PATCH v3 07/18] x86/hvm: introduce per-vCPU L3 page-table

[Jan Beulich]

On 08.10.2026 23:00, George Dunlap wrote:
> On Thu, Oct 8, 2026 at 9:40 AM Jan Beulich <jbeulich@suse.com> wrote:
>> On 07.10.2026 12:40, George Dunlap wrote:
>>> --- a/xen/arch/x86/pv/domain.c
>>> +++ b/xen/arch/x86/pv/domain.c
>>> @@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
>>>      mfn = page_to_mfn(pg);
>>>      l4tab = map_domain_page(mfn);
>>>      clear_page(l4tab);
>>> -    init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
>>> +    init_xen_l4_slots(l4tab, mfn, v->domain, v, INVALID_MFN, false);
>>
>> This being PV-only code, don't you want to pass NULL here? The description
>> talks about this, but (a) it's not quite clear why PV32 would want handling
>> differently from PV64 at this point and (b) in sh_make_shadow() you do pass
>> NULL unconditionally.
> 
> This is actually left over from the previous iteration of the series
> which supports PV guests; and apparently it would be a lot easier for
> implement per-vCPU pagetables for 32-bit PV guests, since the L4
> doesn't belong to the guest.  But it's outside of the scope of the
> current work.
> 
> At any rate, I'm going to try passing a single vcpu, as Roger
> suggested.  That restricts toolstacks to not being able to promote PV
> L4s before calling XEN_DOMCTL_max_vcpus, but that seems fine to me.

But my question was why it's not NULL that is being passed here.

Jan

[PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

Currently, only PV domains have a mapcache, which is domain-wide one.
In preparation for enabling per-vCPU mapcaches for HVM domains, split
the accounting out into a struct mapcache, embedded in struct
mapcache_domain.

Introduce has_mapcache(), for whether a vCPU has a mapcache, and
vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
caller has to take into account.  For now has_mapcache() is
is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
accounting.  map_domain_page() and unmap_domain_page() stop repeating
the is_pv_vcpu() check mapcache_current_vcpu() has made.

Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
into three helpers: mapcache_lock() takes the lock and catches up with
a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
after a wrap, and mapcache_unlock() drops the lock.  A per-vCPU
mapcache can then skip them without map_domain_page() being
re-indented.

struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
and so do the structures containing it.

No functional change.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
  mapcache for vCPU-PT domains" for review.
---
 xen/arch/x86/domain_page.c        | 155 ++++++++++++++++++++----------
 xen/arch/x86/include/asm/domain.h |  19 ++--
 2 files changed, 116 insertions(+), 58 deletions(-)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 9119f735f5..5d90f77bed 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -18,17 +18,23 @@
 #include <asm/hardirq.h>
 #include <asm/setup.h>
 
+/*
+ * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
+ * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
+ * the directmap, which covers all physical address space for them.
+ */
+static bool has_mapcache(const struct vcpu *v)
+{
+    return is_pv_vcpu(v);
+}
+
 static inline struct vcpu *mapcache_current_vcpu(void)
 {
     struct vcpu *v = this_cpu(pgtable_vcpu);
     struct vcpu *curr = current;
 
-    /*
-     * During early boot pgtable_vcpu is not set, callers must handle NULL.
-     * Non-PV domains don't have a mapcache, the directmap covers all physical
-     * address space.
-     */
-    if ( !v || !is_pv_vcpu(v) )
+    /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
+    if ( !v || !has_mapcache(v) )
         return NULL;
 
     /*
@@ -54,6 +60,55 @@ static inline struct vcpu *mapcache_current_vcpu(void)
     return ACCESS_ONCE(this_cpu(pgtable_vcpu));
 }
 
+/*
+ * The accounting for v's mapcache: the domain's, with *dcache set so the
+ * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ */
+static struct mapcache *vcpu_mapcache(struct vcpu *v,
+                                      struct mapcache_domain **dcache)
+{
+    struct domain *d = v->domain;
+
+    *dcache = &d->arch.pv.mapcache;
+    return &(*dcache)->cache;
+}
+
+/*
+ * The domain-wide mapcache's lock and TLB-epoch machinery, taken around
+ * allocating a slot: the domain's vCPUs share the mapcache's slots, and each
+ * catches up with a wrap another one caused, flushing its TLB if it has not
+ * been flushed since.
+ */
+static void mapcache_lock(struct mapcache_domain *dcache,
+                          struct mapcache_vcpu *vcache)
+{
+    spin_lock(&dcache->lock);
+
+    /* Has some other CPU caused a wrap? We must flush if so. */
+    if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
+    {
+        vcache->shadow_epoch = dcache->epoch;
+        if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
+        {
+            perfc_incr(domain_page_tlb_flush);
+            flush_tlb_local();
+        }
+    }
+}
+
+/* A wrap has just flushed this CPU's TLB: start a new epoch. */
+static void mapcache_new_epoch(struct mapcache_domain *dcache,
+                               struct mapcache_vcpu *vcache)
+{
+    vcache->shadow_epoch = ++dcache->epoch;
+    dcache->tlbflush_timestamp = tlbflush_current_time();
+}
+
+static void mapcache_unlock(struct mapcache_domain *dcache)
+{
+    spin_unlock(&dcache->lock);
+}
+
 #define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
 #define MAPCACHE_L2_ENTRIES (mapcache_l2_entry(MAPCACHE_ENTRIES - 1) + 1)
 #define MAPCACHE_L1ENT(idx) \
@@ -66,6 +121,7 @@ void *map_domain_page(mfn_t mfn)
     struct vcpu *v;
     struct mapcache_domain *dcache;
     struct mapcache_vcpu *vcache;
+    struct mapcache *cache;
     struct vcpu_maphash_entry *hashent;
 
 #ifdef NDEBUG
@@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
 #endif
 
     v = mapcache_current_vcpu();
-    if ( !v || !is_pv_vcpu(v) )
+    if ( !v )
         return mfn_to_virt(mfn_x(mfn));
 
-    dcache = &v->domain->arch.pv.mapcache;
+    cache = vcpu_mapcache(v, &dcache);
     vcache = &v->arch.pv.mapcache;
-    if ( !dcache->inuse )
+    if ( !cache->inuse )
         return mfn_to_virt(mfn_x(mfn));
 
     perfc_incr(map_domain_page_count);
@@ -90,41 +146,30 @@ void *map_domain_page(mfn_t mfn)
     if ( hashent->mfn == mfn_x(mfn) )
     {
         idx = hashent->idx;
-        ASSERT(idx < dcache->entries);
+        ASSERT(idx < cache->entries);
         hashent->refcnt++;
         ASSERT(hashent->refcnt);
         ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
         goto out;
     }
 
-    spin_lock(&dcache->lock);
-
-    /* Has some other CPU caused a wrap? We must flush if so. */
-    if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
-    {
-        vcache->shadow_epoch = dcache->epoch;
-        if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
-        {
-            perfc_incr(domain_page_tlb_flush);
-            flush_tlb_local();
-        }
-    }
+    mapcache_lock(dcache, vcache);
 
-    idx = find_next_zero_bit(dcache->inuse, dcache->entries, dcache->cursor);
-    if ( unlikely(idx >= dcache->entries) )
+    idx = find_next_zero_bit(cache->inuse, cache->entries, cache->cursor);
+    if ( unlikely(idx >= cache->entries) )
     {
         unsigned long accum = 0, prev = 0;
 
         /* /First/, clean the garbage map and update the inuse list. */
-        for ( i = 0; i < BITS_TO_LONGS(dcache->entries); i++ )
+        for ( i = 0; i < BITS_TO_LONGS(cache->entries); i++ )
         {
             accum |= prev;
-            dcache->inuse[i] &= ~xchg(&dcache->garbage[i], 0);
-            prev = ~dcache->inuse[i];
+            cache->inuse[i] &= ~xchg(&cache->garbage[i], 0);
+            prev = ~cache->inuse[i];
         }
 
-        if ( accum | (prev & BITMAP_LAST_WORD_MASK(dcache->entries)) )
-            idx = find_first_zero_bit(dcache->inuse, dcache->entries);
+        if ( accum | (prev & BITMAP_LAST_WORD_MASK(cache->entries)) )
+            idx = find_first_zero_bit(cache->inuse, cache->entries);
         else
         {
             /* Replace a hash entry instead. */
@@ -144,19 +189,18 @@ void *map_domain_page(mfn_t mfn)
                     i = 0;
             } while ( i != MAPHASH_HASHFN(mfn_x(mfn)) );
         }
-        BUG_ON(idx >= dcache->entries);
+        BUG_ON(idx >= cache->entries);
 
         /* /Second/, flush TLBs. */
         perfc_incr(domain_page_tlb_flush);
         flush_tlb_local();
-        vcache->shadow_epoch = ++dcache->epoch;
-        dcache->tlbflush_timestamp = tlbflush_current_time();
+        mapcache_new_epoch(dcache, vcache);
     }
 
-    set_bit(idx, dcache->inuse);
-    dcache->cursor = idx + 1;
+    set_bit(idx, cache->inuse);
+    cache->cursor = idx + 1;
 
-    spin_unlock(&dcache->lock);
+    mapcache_unlock(dcache);
 
     l1e_write(&MAPCACHE_L1ENT(idx), l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
 
@@ -170,6 +214,7 @@ void unmap_domain_page(const void *ptr)
     unsigned int idx;
     struct vcpu *v;
     struct mapcache_domain *dcache;
+    struct mapcache *cache;
     unsigned long va = (unsigned long)ptr, mfn, flags;
     struct vcpu_maphash_entry *hashent;
 
@@ -179,10 +224,10 @@ void unmap_domain_page(const void *ptr)
     ASSERT(va >= MAPCACHE_VIRT_START && va < MAPCACHE_VIRT_END);
 
     v = mapcache_current_vcpu();
-    ASSERT(v && is_pv_vcpu(v));
+    ASSERT(v);
 
-    dcache = &v->domain->arch.pv.mapcache;
-    ASSERT(dcache->inuse);
+    cache = vcpu_mapcache(v, &dcache);
+    ASSERT(cache->inuse);
 
     idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
     mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
@@ -205,7 +250,7 @@ void unmap_domain_page(const void *ptr)
                    hashent->mfn);
             l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
             /* /Second/, mark as garbage. */
-            set_bit(hashent->idx, dcache->garbage);
+            set_bit(hashent->idx, cache->garbage);
         }
 
         /* Add newly-freed mapping to the maphash. */
@@ -217,7 +262,7 @@ void unmap_domain_page(const void *ptr)
         /* /First/, zap the PTE. */
         l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
         /* /Second/, mark as garbage. */
-        set_bit(idx, dcache->garbage);
+        set_bit(idx, cache->garbage);
     }
 
     local_irq_restore(flags);
@@ -239,9 +284,9 @@ void mapcache_domain_init(struct domain *d)
                  2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
                  MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
     bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
-    dcache->inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
-    dcache->garbage = dcache->inuse +
-                      (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
+    dcache->cache.inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
+    dcache->cache.garbage = dcache->cache.inuse +
+                            (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
 
     spin_lock_init(&dcache->lock);
 }
@@ -249,15 +294,23 @@ void mapcache_domain_init(struct domain *d)
 int mapcache_vcpu_init(struct vcpu *v)
 {
     struct domain *d = v->domain;
-    struct mapcache_domain *dcache = &d->arch.pv.mapcache;
+    struct mapcache_domain *dcache;
+    struct mapcache *cache;
     unsigned long i;
-    unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
-    unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+    unsigned int ents, nr;
 
-    if ( !is_pv_vcpu(v) || !dcache->inuse )
+    if ( !has_mapcache(v) )
         return 0;
 
-    if ( ents > dcache->entries )
+    cache = vcpu_mapcache(v, &dcache);
+    if ( !cache->inuse )
+        /* Domain-wide mapcache disabled (fully direct-mapped build). */
+        return 0;
+
+    ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
+    nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+
+    if ( ents > cache->entries )
     {
         /* Populate page tables. */
         int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
@@ -265,16 +318,16 @@ int mapcache_vcpu_init(struct vcpu *v)
 
         /* Populate bit maps. */
         if ( !rc )
-            rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
+            rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
                                           nr, NULL, NIL(struct page_info *));
         if ( !rc )
-            rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
+            rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
                                           nr, NULL, NIL(struct page_info *));
 
         if ( rc )
             return rc;
 
-        dcache->entries = ents;
+        cache->entries = ents;
     }
 
     /* Mark all maphash entries as not in use. */
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index a6c6eaf53a..6f5873879e 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -57,6 +57,16 @@ struct trap_bounce {
     unsigned long eip;
 };
 
+struct mapcache {
+    /* The number of array entries, and a cursor into the array. */
+    unsigned int entries;
+    unsigned int cursor;
+
+    /* Which mappings are in use, and which are garbage to reap next epoch? */
+    unsigned long *inuse;
+    unsigned long *garbage;
+};
+
 #define MAPHASH_ENTRIES 8
 #define MAPHASH_HASHFN(pfn) ((pfn) & (MAPHASH_ENTRIES-1))
 #define MAPHASHENT_NOTINUSE ((u32)~0U)
@@ -73,10 +83,6 @@ struct mapcache_vcpu {
 };
 
 struct mapcache_domain {
-    /* The number of array entries, and a cursor into the array. */
-    unsigned int entries;
-    unsigned int cursor;
-
     /* Protects map_domain_page(). */
     spinlock_t lock;
 
@@ -84,9 +90,8 @@ struct mapcache_domain {
     unsigned int epoch;
     u32 tlbflush_timestamp;
 
-    /* Which mappings are in use, and which are garbage to reap next epoch? */
-    unsigned long *inuse;
-    unsigned long *garbage;
+    /* Accounting of the domain-wide mapcache. */
+    struct mapcache cache;
 };
 
 void mapcache_domain_init(struct domain *d);
-- 
2.55.0

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:41AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> Currently, only PV domains have a mapcache, which is domain-wide one.
> In preparation for enabling per-vCPU mapcaches for HVM domains, split
> the accounting out into a struct mapcache, embedded in struct
> mapcache_domain.
> 
> Introduce has_mapcache(), for whether a vCPU has a mapcache, and
> vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
> and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
> caller has to take into account.  For now has_mapcache() is
> is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
> accounting.  map_domain_page() and unmap_domain_page() stop repeating
> the is_pv_vcpu() check mapcache_current_vcpu() has made.
> 
> Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
> into three helpers: mapcache_lock() takes the lock and catches up with
> a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
> after a wrap, and mapcache_unlock() drops the lock.  A per-vCPU
> mapcache can then skip them without map_domain_page() being
> re-indented.
> 
> struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
> and so do the structures containing it.
> 
> No functional change.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version, split out of "x86/hvm: introduce per-vCPU
>   mapcache for vCPU-PT domains" for review.
> ---
>  xen/arch/x86/domain_page.c        | 155 ++++++++++++++++++++----------
>  xen/arch/x86/include/asm/domain.h |  19 ++--
>  2 files changed, 116 insertions(+), 58 deletions(-)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index 9119f735f5..5d90f77bed 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -18,17 +18,23 @@
>  #include <asm/hardirq.h>
>  #include <asm/setup.h>
>  
> +/*
> + * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
> + * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
> + * the directmap, which covers all physical address space for them.

I've been wondering whether there could be a way to write this comment
so that it doesn't need rewording once we switch to per-vCPU
mapcaches, as to avoid adding a comment here that will very likely be
completely re-worded by a later change in the series.  However I
cannot find anything much better rather than stripping most of the
comment content.

> + */
> +static bool has_mapcache(const struct vcpu *v)
> +{
> +    return is_pv_vcpu(v);
> +}
> +
>  static inline struct vcpu *mapcache_current_vcpu(void)
>  {
>      struct vcpu *v = this_cpu(pgtable_vcpu);
>      struct vcpu *curr = current;
>  
> -    /*
> -     * During early boot pgtable_vcpu is not set, callers must handle NULL.
> -     * Non-PV domains don't have a mapcache, the directmap covers all physical
> -     * address space.
> -     */
> -    if ( !v || !is_pv_vcpu(v) )
> +    /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
> +    if ( !v || !has_mapcache(v) )
>          return NULL;
>  
>      /*
> @@ -54,6 +60,55 @@ static inline struct vcpu *mapcache_current_vcpu(void)
>      return ACCESS_ONCE(this_cpu(pgtable_vcpu));
>  }
>  
> +/*
> + * The accounting for v's mapcache: the domain's, with *dcache set so the
> + * caller takes the domain-wide lock and TLB-epoch machinery into account.
> + */
> +static struct mapcache *vcpu_mapcache(struct vcpu *v,
> +                                      struct mapcache_domain **dcache)
> +{
> +    struct domain *d = v->domain;
> +
> +    *dcache = &d->arch.pv.mapcache;
> +    return &(*dcache)->cache;
> +}
> +
> +/*
> + * The domain-wide mapcache's lock and TLB-epoch machinery, taken around
> + * allocating a slot: the domain's vCPUs share the mapcache's slots, and each
> + * catches up with a wrap another one caused, flushing its TLB if it has not
> + * been flushed since.

Thinking forward about how this function will end up looking when we
have per-vCPU map caches, do you maybe want to place this comment
inside the function body itself?

I expect per-vCPU splitting will add a short-circuit at the top of the
function that will skip taking the lock or doing the flushing, and
hence the comment won't need shuffling around?

> + */
> +static void mapcache_lock(struct mapcache_domain *dcache,

Maybe mapcache_enter() would be a better suited name once there's no
longer a need to take a lock for the mapcache?  Just wondering, no
strong opinion really (the more that I don't even know how the code
will end up looking yet).

> +                          struct mapcache_vcpu *vcache)
> +{
> +    spin_lock(&dcache->lock);
> +
> +    /* Has some other CPU caused a wrap? We must flush if so. */
> +    if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
> +    {
> +        vcache->shadow_epoch = dcache->epoch;
> +        if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
> +        {
> +            perfc_incr(domain_page_tlb_flush);
> +            flush_tlb_local();
> +        }
> +    }
> +}
> +
> +/* A wrap has just flushed this CPU's TLB: start a new epoch. */
> +static void mapcache_new_epoch(struct mapcache_domain *dcache,
> +                               struct mapcache_vcpu *vcache)
> +{

Since this is now a function on its own, I would possibly assert that
dcache->lock is taken on entry.

Let me know what you think of the suggestions above, overall I think
the logic is fine.

Thanks, Roger.

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[George Dunlap]

On Thu, Oct 8, 2026 at 9:22 AM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:41AM +0100, George Dunlap wrote:
> > From: Roger Pau Monné <roger.pau@citrix.com>
> >
> > Currently, only PV domains have a mapcache, which is domain-wide one.
> > In preparation for enabling per-vCPU mapcaches for HVM domains, split
> > the accounting out into a struct mapcache, embedded in struct
> > mapcache_domain.
> >
> > Introduce has_mapcache(), for whether a vCPU has a mapcache, and
> > vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
> > and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
> > caller has to take into account.  For now has_mapcache() is
> > is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
> > accounting.  map_domain_page() and unmap_domain_page() stop repeating
> > the is_pv_vcpu() check mapcache_current_vcpu() has made.
> >
> > Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
> > into three helpers: mapcache_lock() takes the lock and catches up with
> > a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
> > after a wrap, and mapcache_unlock() drops the lock.  A per-vCPU
> > mapcache can then skip them without map_domain_page() being
> > re-indented.
> >
> > struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
> > and so do the structures containing it.
> >
> > No functional change.
> >
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> > Signed-off-by: George Dunlap <gwd@xenproject.org>
> > ---
> > Changes in v3:
> > - New in this version, split out of "x86/hvm: introduce per-vCPU
> >   mapcache for vCPU-PT domains" for review.
> > ---
> >  xen/arch/x86/domain_page.c        | 155 ++++++++++++++++++++----------
> >  xen/arch/x86/include/asm/domain.h |  19 ++--
> >  2 files changed, 116 insertions(+), 58 deletions(-)
> >
> > diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> > index 9119f735f5..5d90f77bed 100644
> > --- a/xen/arch/x86/domain_page.c
> > +++ b/xen/arch/x86/domain_page.c
> > @@ -18,17 +18,23 @@
> >  #include <asm/hardirq.h>
> >  #include <asm/setup.h>
> >
> > +/*
> > + * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
> > + * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
> > + * the directmap, which covers all physical address space for them.
>
> I've been wondering whether there could be a way to write this comment
> so that it doesn't need rewording once we switch to per-vCPU
> mapcaches, as to avoid adding a comment here that will very likely be
> completely re-worded by a later change in the series.  However I
> cannot find anything much better rather than stripping most of the
> comment content.

I think describing it the way it is, then changing it, makes the
series or the state thereof more clear.  But I'm not precious about
it; we could make it something like this:

/*
 * Whether v may use a mapcache rather than the directmap (a PV domain's
 * mapcache may still be disabled, see mapcache_domain_init()).
 */

That said, as a meta-comment: We have a limited amount of time on my
contract, and also a limited amount of time before we want to check
this in to the tree.  Is spending our time changing non-functional
intermittent git tree state really what you want to spend that time
on?

>
> > + */
> > +static bool has_mapcache(const struct vcpu *v)
> > +{
> > +    return is_pv_vcpu(v);
> > +}
> > +
> >  static inline struct vcpu *mapcache_current_vcpu(void)
> >  {
> >      struct vcpu *v = this_cpu(pgtable_vcpu);
> >      struct vcpu *curr = current;
> >
> > -    /*
> > -     * During early boot pgtable_vcpu is not set, callers must handle NULL.
> > -     * Non-PV domains don't have a mapcache, the directmap covers all physical
> > -     * address space.
> > -     */
> > -    if ( !v || !is_pv_vcpu(v) )
> > +    /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
> > +    if ( !v || !has_mapcache(v) )
> >          return NULL;
> >
> >      /*
> > @@ -54,6 +60,55 @@ static inline struct vcpu *mapcache_current_vcpu(void)
> >      return ACCESS_ONCE(this_cpu(pgtable_vcpu));
> >  }
> >
> > +/*
> > + * The accounting for v's mapcache: the domain's, with *dcache set so the
> > + * caller takes the domain-wide lock and TLB-epoch machinery into account.
> > + */
> > +static struct mapcache *vcpu_mapcache(struct vcpu *v,
> > +                                      struct mapcache_domain **dcache)
> > +{
> > +    struct domain *d = v->domain;
> > +
> > +    *dcache = &d->arch.pv.mapcache;
> > +    return &(*dcache)->cache;
> > +}
> > +
> > +/*
> > + * The domain-wide mapcache's lock and TLB-epoch machinery, taken around
> > + * allocating a slot: the domain's vCPUs share the mapcache's slots, and each
> > + * catches up with a wrap another one caused, flushing its TLB if it has not
> > + * been flushed since.
>
> Thinking forward about how this function will end up looking when we
> have per-vCPU map caches, do you maybe want to place this comment
> inside the function body itself?
>
> I expect per-vCPU splitting will add a short-circuit at the top of the
> function that will skip taking the lock or doing the flushing, and
> hence the comment won't need shuffling around?

Yup, we can do this.

> > + */
> > +static void mapcache_lock(struct mapcache_domain *dcache,
>
> Maybe mapcache_enter() would be a better suited name once there's no
> longer a need to take a lock for the mapcache?  Just wondering, no
> strong opinion really (the more that I don't even know how the code
> will end up looking yet).

Yeah, can do that.

> > +/* A wrap has just flushed this CPU's TLB: start a new epoch. */
> > +static void mapcache_new_epoch(struct mapcache_domain *dcache,
> > +                               struct mapcache_vcpu *vcache)
> > +{
>
> Since this is now a function on its own, I would possibly assert that
> dcache->lock is taken on entry.
>
> Let me know what you think of the suggestions above, overall I think
> the logic is fine.

Great, thanks.
 -George

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[Roger Pau Monné]

On Mon, Oct 12, 2026 at 03:19:20PM +0100, George Dunlap wrote:
> On Thu, Oct 8, 2026 at 9:22 AM Roger Pau Monné <roger@xenproject.org> wrote:
> >
> > On Wed, Oct 07, 2026 at 11:40:41AM +0100, George Dunlap wrote:
> > > From: Roger Pau Monné <roger.pau@citrix.com>
> > >
> > > Currently, only PV domains have a mapcache, which is domain-wide one.
> > > In preparation for enabling per-vCPU mapcaches for HVM domains, split
> > > the accounting out into a struct mapcache, embedded in struct
> > > mapcache_domain.
> > >
> > > Introduce has_mapcache(), for whether a vCPU has a mapcache, and
> > > vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
> > > and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
> > > caller has to take into account.  For now has_mapcache() is
> > > is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
> > > accounting.  map_domain_page() and unmap_domain_page() stop repeating
> > > the is_pv_vcpu() check mapcache_current_vcpu() has made.
> > >
> > > Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
> > > into three helpers: mapcache_lock() takes the lock and catches up with
> > > a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
> > > after a wrap, and mapcache_unlock() drops the lock.  A per-vCPU
> > > mapcache can then skip them without map_domain_page() being
> > > re-indented.
> > >
> > > struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
> > > and so do the structures containing it.
> > >
> > > No functional change.
> > >
> > > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > > Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> > > Signed-off-by: George Dunlap <gwd@xenproject.org>
> > > ---
> > > Changes in v3:
> > > - New in this version, split out of "x86/hvm: introduce per-vCPU
> > >   mapcache for vCPU-PT domains" for review.
> > > ---
> > >  xen/arch/x86/domain_page.c        | 155 ++++++++++++++++++++----------
> > >  xen/arch/x86/include/asm/domain.h |  19 ++--
> > >  2 files changed, 116 insertions(+), 58 deletions(-)
> > >
> > > diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> > > index 9119f735f5..5d90f77bed 100644
> > > --- a/xen/arch/x86/domain_page.c
> > > +++ b/xen/arch/x86/domain_page.c
> > > @@ -18,17 +18,23 @@
> > >  #include <asm/hardirq.h>
> > >  #include <asm/setup.h>
> > >
> > > +/*
> > > + * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
> > > + * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
> > > + * the directmap, which covers all physical address space for them.
> >
> > I've been wondering whether there could be a way to write this comment
> > so that it doesn't need rewording once we switch to per-vCPU
> > mapcaches, as to avoid adding a comment here that will very likely be
> > completely re-worded by a later change in the series.  However I
> > cannot find anything much better rather than stripping most of the
> > comment content.
> 
> I think describing it the way it is, then changing it, makes the
> series or the state thereof more clear.  But I'm not precious about
> it; we could make it something like this:
> 
> /*
>  * Whether v may use a mapcache rather than the directmap (a PV domain's
>  * mapcache may still be disabled, see mapcache_domain_init()).
>  */
> 
> That said, as a meta-comment: We have a limited amount of time on my
> contract, and also a limited amount of time before we want to check
> this in to the tree.  Is spending our time changing non-functional
> intermittent git tree state really what you want to spend that time
> on?

It's sometimes hard to know whether a comment is in an intermediate form,
or in the final one, hence it's difficult to trim review to not be
picky about intermittent state once the final form might be
acceptable.  That's why I've suggested to try to add comments in its
final form (or in a form that just need expansion by later changes).

I would maybe use:

/* Whether the vCPU supports mapcache usage, might not be in use. */

Personally: referencing functions by name in code comments is also
something that I usually try to avoid doing if possible.  As the code
evolves it's not unlikely for functions to change names, and we have
(historically) not been very good at keeping the referencing comments
up to date.

Thanks, Roger.

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[Jan Beulich]

On 07.10.2026 12:40, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> Currently, only PV domains have a mapcache, which is domain-wide one.
> In preparation for enabling per-vCPU mapcaches for HVM domains, split
> the accounting out into a struct mapcache, embedded in struct
> mapcache_domain.
> 
> Introduce has_mapcache(), for whether a vCPU has a mapcache, and
> vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
> and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
> caller has to take into account.  For now has_mapcache() is
> is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
> accounting.  map_domain_page() and unmap_domain_page() stop repeating
> the is_pv_vcpu() check mapcache_current_vcpu() has made.
> 
> Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
> into three helpers: mapcache_lock() takes the lock and catches up with
> a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
> after a wrap, and mapcache_unlock() drops the lock.  A per-vCPU
> mapcache can then skip them without map_domain_page() being
> re-indented.
> 
> struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
> and so do the structures containing it.
> 
> No functional change.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version, split out of "x86/hvm: introduce per-vCPU
>   mapcache for vCPU-PT domains" for review.
> ---
>  xen/arch/x86/domain_page.c        | 155 ++++++++++++++++++++----------
>  xen/arch/x86/include/asm/domain.h |  19 ++--
>  2 files changed, 116 insertions(+), 58 deletions(-)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index 9119f735f5..5d90f77bed 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -18,17 +18,23 @@
>  #include <asm/hardirq.h>
>  #include <asm/setup.h>
>  
> +/*
> + * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
> + * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
> + * the directmap, which covers all physical address space for them.
> + */
> +static bool has_mapcache(const struct vcpu *v)
> +{
> +    return is_pv_vcpu(v);
> +}
> +
>  static inline struct vcpu *mapcache_current_vcpu(void)
>  {
>      struct vcpu *v = this_cpu(pgtable_vcpu);
>      struct vcpu *curr = current;
>  
> -    /*
> -     * During early boot pgtable_vcpu is not set, callers must handle NULL.
> -     * Non-PV domains don't have a mapcache, the directmap covers all physical
> -     * address space.
> -     */
> -    if ( !v || !is_pv_vcpu(v) )
> +    /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
> +    if ( !v || !has_mapcache(v) )
>          return NULL;

(For context below: The replacement of is_pv_vcpu() here is easy to follow.)

> @@ -54,6 +60,55 @@ static inline struct vcpu *mapcache_current_vcpu(void)
>      return ACCESS_ONCE(this_cpu(pgtable_vcpu));
>  }
>  
> +/*
> + * The accounting for v's mapcache: the domain's, with *dcache set so the
> + * caller takes the domain-wide lock and TLB-epoch machinery into account.
> + */
> +static struct mapcache *vcpu_mapcache(struct vcpu *v,
> +                                      struct mapcache_domain **dcache)
> +{
> +    struct domain *d = v->domain;
> +
> +    *dcache = &d->arch.pv.mapcache;
> +    return &(*dcache)->cache;
> +}

This function only retrieves data. Can't its first parameter therefore be
pointer-to-const?

> @@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
>  #endif
>  
>      v = mapcache_current_vcpu();
> -    if ( !v || !is_pv_vcpu(v) )
> +    if ( !v )
>          return mfn_to_virt(mfn_x(mfn));

I don't quite understand why this is_pv_vcpu() is okay to drop. What
mapcache_current_vcpu() returns isn't what it ran is_pv_vcpu() on. (Same
in unmap_domain_page() then, obviously.)

> @@ -249,15 +294,23 @@ void mapcache_domain_init(struct domain *d)
>  int mapcache_vcpu_init(struct vcpu *v)
>  {
>      struct domain *d = v->domain;
> -    struct mapcache_domain *dcache = &d->arch.pv.mapcache;
> +    struct mapcache_domain *dcache;
> +    struct mapcache *cache;
>      unsigned long i;
> -    unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
> -    unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
> +    unsigned int ents, nr;
>  
> -    if ( !is_pv_vcpu(v) || !dcache->inuse )
> +    if ( !has_mapcache(v) )
>          return 0;
>  
> -    if ( ents > dcache->entries )
> +    cache = vcpu_mapcache(v, &dcache);
> +    if ( !cache->inuse )
> +        /* Domain-wide mapcache disabled (fully direct-mapped build). */
> +        return 0;

While technically not wrong, this comment appears to be quite a bit
forward-looking.

> +    ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
> +    nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));

Is there a reason these cannot be the initializers of their variables
anymore?

Jan

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[George Dunlap]

On Mon, Oct 12, 2026 at 2:48 PM Jan Beulich <jbeulich@suse.com> wrote:
> > @@ -54,6 +60,55 @@ static inline struct vcpu *mapcache_current_vcpu(void)
> >      return ACCESS_ONCE(this_cpu(pgtable_vcpu));
> >  }
> >
> > +/*
> > + * The accounting for v's mapcache: the domain's, with *dcache set so the
> > + * caller takes the domain-wide lock and TLB-epoch machinery into account.
> > + */
> > +static struct mapcache *vcpu_mapcache(struct vcpu *v,
> > +                                      struct mapcache_domain **dcache)
> > +{
> > +    struct domain *d = v->domain;
> > +
> > +    *dcache = &d->arch.pv.mapcache;
> > +    return &(*dcache)->cache;
> > +}
>
> This function only retrieves data. Can't its first parameter therefore be
> pointer-to-const?

Currently that's true; but in a further patch, it may be
&v->arch.mapcache.cache, which is in the vcpu structure itself,
effectively stripping off the const-ness by pointing to individual
fields in a struct.

>
> > @@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
> >  #endif
> >
> >      v = mapcache_current_vcpu();
> > -    if ( !v || !is_pv_vcpu(v) )
> > +    if ( !v )
> >          return mfn_to_virt(mfn_x(mfn));
>
> I don't quite understand why this is_pv_vcpu() is okay to drop. What
> mapcache_current_vcpu() returns isn't what it ran is_pv_vcpu() on. (Same
> in unmap_domain_page() then, obviously.)

So mapcache_current_vcpu() is a bit weird; the actual invariant is,
"Return the vcpu whose cache structure we should use if present; or
NULL if there's no cache structure to check and no need to sync."  So
if pgtable_vcpu is HVM, return NULL and just use the raw address,
because at this point in the series, both HVM and the idle domain have
full directmaps; if pgtable_vcpu is PV, then maybe sync, and then
return the result of the sync.  That may be a PV domain with a
mapcache, or an idle domain with no mapcache, but in either case you
can then read cache.inuse to see if there's something there.

So yes, it's a different vCPU; but it's still defined in such a way to
be safe to check cache.inuse.  The only possible value *change* after
a sync is the idle domain; I don't think we need to do another check
to make sure it's still a PV domain.

In fact, there's an argument that we should modify
mapcache_current_vcpu() to take pointers to dcache, and vcache, and
return just cache; that gets rid of v as a local variable of
map_domain_page entirely, and makes the logic ("get the cache we
should use") more clean, and gets rid of one mfn_to_virt() instance.
That would require some non-trivial changes, so I'd rather avoid that,
but I can do it if you want.

> > @@ -249,15 +294,23 @@ void mapcache_domain_init(struct domain *d)
> >  int mapcache_vcpu_init(struct vcpu *v)
> >  {
> >      struct domain *d = v->domain;
> > -    struct mapcache_domain *dcache = &d->arch.pv.mapcache;
> > +    struct mapcache_domain *dcache;
> > +    struct mapcache *cache;
> >      unsigned long i;
> > -    unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
> > -    unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
> > +    unsigned int ents, nr;
> >
> > -    if ( !is_pv_vcpu(v) || !dcache->inuse )
> > +    if ( !has_mapcache(v) )
> >          return 0;
> >
> > -    if ( ents > dcache->entries )
> > +    cache = vcpu_mapcache(v, &dcache);
> > +    if ( !cache->inuse )
> > +        /* Domain-wide mapcache disabled (fully direct-mapped build). */
> > +        return 0;
>
> While technically not wrong, this comment appears to be quite a bit
> forward-looking.

Yes; I'll drop the last bit.

>
> > +    ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
> > +    nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
>
> Is there a reason these cannot be the initializers of their variables
> anymore?

No reason.  I'll put them back.

 -George

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[Jan Beulich]

On 12.10.2026 18:07, George Dunlap wrote:
> On Mon, Oct 12, 2026 at 2:48 PM Jan Beulich <jbeulich@suse.com> wrote:
>>> @@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
>>>  #endif
>>>
>>>      v = mapcache_current_vcpu();
>>> -    if ( !v || !is_pv_vcpu(v) )
>>> +    if ( !v )
>>>          return mfn_to_virt(mfn_x(mfn));
>>
>> I don't quite understand why this is_pv_vcpu() is okay to drop. What
>> mapcache_current_vcpu() returns isn't what it ran is_pv_vcpu() on. (Same
>> in unmap_domain_page() then, obviously.)
> 
> So mapcache_current_vcpu() is a bit weird; the actual invariant is,
> "Return the vcpu whose cache structure we should use if present; or
> NULL if there's no cache structure to check and no need to sync."  So
> if pgtable_vcpu is HVM, return NULL and just use the raw address,

I was going to say "But there's no returning of NULL there if
pgtable_vcpu is HVM, at least not at this point of the series." Only
then I realized that I was being confused by there being both v and
curr, and neither of them being pointer-to-const (i.e. both being
equal in terms of which of the two might be returned from the function,
yet as a result of earlier changes neither is being returned anymore).

> because at this point in the series, both HVM and the idle domain have
> full directmaps; if pgtable_vcpu is PV, then maybe sync, and then
> return the result of the sync.  That may be a PV domain with a
> mapcache, or an idle domain with no mapcache, but in either case you
> can then read cache.inuse to see if there's something there.
> 
> So yes, it's a different vCPU; but it's still defined in such a way to
> be safe to check cache.inuse.  The only possible value *change* after
> a sync is the idle domain; I don't think we need to do another check
> to make sure it's still a PV domain.

Yet then - aren't the is_pv_vcpu() you drop here redundant already
before this patch/series? In which case (since it's not entirely
straightforward to reason about and beneficial in its own right),
might it make sense to have this be a separate change, with (in
particular) dedicated justification? (ISTR having thought about
making a patch to this effect a little while ago - perhaps around when
Roger last touched this logic -, but then deciding that something was
wrong about my reasoning. That may have been a mistake, as it looks
now.)

> In fact, there's an argument that we should modify
> mapcache_current_vcpu() to take pointers to dcache, and vcache, and
> return just cache; that gets rid of v as a local variable of
> map_domain_page entirely, and makes the logic ("get the cache we
> should use") more clean, and gets rid of one mfn_to_virt() instance.
> That would require some non-trivial changes, so I'd rather avoid that,
> but I can do it if you want.

Sounds entirely plausible, but can as well be done as cleanup later.

Jan

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[George Dunlap]

On Tue, Oct 13, 2026 at 7:26 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 12.10.2026 18:07, George Dunlap wrote:
> > On Mon, Oct 12, 2026 at 2:48 PM Jan Beulich <jbeulich@suse.com> wrote:
> >>> @@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
> >>>  #endif
> >>>
> >>>      v = mapcache_current_vcpu();
> >>> -    if ( !v || !is_pv_vcpu(v) )
> >>> +    if ( !v )
> >>>          return mfn_to_virt(mfn_x(mfn));
> >>
> >> I don't quite understand why this is_pv_vcpu() is okay to drop. What
> >> mapcache_current_vcpu() returns isn't what it ran is_pv_vcpu() on. (Same
> >> in unmap_domain_page() then, obviously.)
> >
> > So mapcache_current_vcpu() is a bit weird; the actual invariant is,
> > "Return the vcpu whose cache structure we should use if present; or
> > NULL if there's no cache structure to check and no need to sync."  So
> > if pgtable_vcpu is HVM, return NULL and just use the raw address,
>
> I was going to say "But there's no returning of NULL there if
> pgtable_vcpu is HVM, at least not at this point of the series." Only
> then I realized that I was being confused by there being both v and
> curr, and neither of them being pointer-to-const (i.e. both being
> equal in terms of which of the two might be returned from the function,
> yet as a result of earlier changes neither is being returned anymore).
>
> > because at this point in the series, both HVM and the idle domain have
> > full directmaps; if pgtable_vcpu is PV, then maybe sync, and then
> > return the result of the sync.  That may be a PV domain with a
> > mapcache, or an idle domain with no mapcache, but in either case you
> > can then read cache.inuse to see if there's something there.
> >
> > So yes, it's a different vCPU; but it's still defined in such a way to
> > be safe to check cache.inuse.  The only possible value *change* after
> > a sync is the idle domain; I don't think we need to do another check
> > to make sure it's still a PV domain.
>
> Yet then - aren't the is_pv_vcpu() you drop here redundant already
> before this patch/series? In which case (since it's not entirely
> straightforward to reason about and beneficial in its own right),
> might it make sense to have this be a separate change, with (in
> particular) dedicated justification? (ISTR having thought about
> making a patch to this effect a little while ago - perhaps around when
> Roger last touched this logic -, but then deciding that something was
> wrong about my reasoning. That may have been a mistake, as it looks
> now.)
>
> > In fact, there's an argument that we should modify
> > mapcache_current_vcpu() to take pointers to dcache, and vcache, and
> > return just cache; that gets rid of v as a local variable of
> > map_domain_page entirely, and makes the logic ("get the cache we
> > should use") more clean, and gets rid of one mfn_to_virt() instance.
> > That would require some non-trivial changes, so I'd rather avoid that,
> > but I can do it if you want.
>
> Sounds entirely plausible, but can as well be done as cleanup later.

I'm a bit confused about what, if any, change you're requesting here.
Are you requesting a preparatory patch just removing the is_pv_vcpu(),
with an argument as to why it's redundant?  Or are you OK with having
this cleanup work be put until later?  Or do you want me to put the
is_pv_vcpu() back until we feel like cleaning it up properly?

Thanks,
 -George

Re: [PATCH v3 08/18] x86/domain_page: prepare the mapcache for per-vCPU accounting

[Jan Beulich]

On 13.10.2026 11:53, George Dunlap wrote:
> On Tue, Oct 13, 2026 at 7:26 AM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 12.10.2026 18:07, George Dunlap wrote:
>>> On Mon, Oct 12, 2026 at 2:48 PM Jan Beulich <jbeulich@suse.com> wrote:
>>>>> @@ -74,12 +130,12 @@ void *map_domain_page(mfn_t mfn)
>>>>>  #endif
>>>>>
>>>>>      v = mapcache_current_vcpu();
>>>>> -    if ( !v || !is_pv_vcpu(v) )
>>>>> +    if ( !v )
>>>>>          return mfn_to_virt(mfn_x(mfn));
>>>>
>>>> I don't quite understand why this is_pv_vcpu() is okay to drop. What
>>>> mapcache_current_vcpu() returns isn't what it ran is_pv_vcpu() on. (Same
>>>> in unmap_domain_page() then, obviously.)
>>>
>>> So mapcache_current_vcpu() is a bit weird; the actual invariant is,
>>> "Return the vcpu whose cache structure we should use if present; or
>>> NULL if there's no cache structure to check and no need to sync."  So
>>> if pgtable_vcpu is HVM, return NULL and just use the raw address,
>>
>> I was going to say "But there's no returning of NULL there if
>> pgtable_vcpu is HVM, at least not at this point of the series." Only
>> then I realized that I was being confused by there being both v and
>> curr, and neither of them being pointer-to-const (i.e. both being
>> equal in terms of which of the two might be returned from the function,
>> yet as a result of earlier changes neither is being returned anymore).
>>
>>> because at this point in the series, both HVM and the idle domain have
>>> full directmaps; if pgtable_vcpu is PV, then maybe sync, and then
>>> return the result of the sync.  That may be a PV domain with a
>>> mapcache, or an idle domain with no mapcache, but in either case you
>>> can then read cache.inuse to see if there's something there.
>>>
>>> So yes, it's a different vCPU; but it's still defined in such a way to
>>> be safe to check cache.inuse.  The only possible value *change* after
>>> a sync is the idle domain; I don't think we need to do another check
>>> to make sure it's still a PV domain.
>>
>> Yet then - aren't the is_pv_vcpu() you drop here redundant already
>> before this patch/series? In which case (since it's not entirely
>> straightforward to reason about and beneficial in its own right),
>> might it make sense to have this be a separate change, with (in
>> particular) dedicated justification? (ISTR having thought about
>> making a patch to this effect a little while ago - perhaps around when
>> Roger last touched this logic -, but then deciding that something was
>> wrong about my reasoning. That may have been a mistake, as it looks
>> now.)
>>
>>> In fact, there's an argument that we should modify
>>> mapcache_current_vcpu() to take pointers to dcache, and vcache, and
>>> return just cache; that gets rid of v as a local variable of
>>> map_domain_page entirely, and makes the logic ("get the cache we
>>> should use") more clean, and gets rid of one mfn_to_virt() instance.
>>> That would require some non-trivial changes, so I'd rather avoid that,
>>> but I can do it if you want.
>>
>> Sounds entirely plausible, but can as well be done as cleanup later.
> 
> I'm a bit confused about what, if any, change you're requesting here.
> Are you requesting a preparatory patch just removing the is_pv_vcpu(),
> with an argument as to why it's redundant?  Or are you OK with having
> this cleanup work be put until later?  Or do you want me to put the
> is_pv_vcpu() back until we feel like cleaning it up properly?

Now the confusion is on my side: Aren't what you suggested as further
cleanup and the dropping of the is_pv_vcpu() independent? If it had
been just your first question, I would have said "yes, if possible".
But the following question starting with "Or" is irritating. Yes,
that _other_ cleanup work can (as I thought you also suggested) be
deferred.

Putting back the is_pv_vcpu() isn't really necessary, as long as
their removal can be properly justified. (Hence why I think that would
be better as a separate change.)

Jan

[PATCH v3 09/18] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

struct mapcache_vcpu, the per-vCPU part of the mapcache (the maphash
and the vCPU's shadow of the domain's TLB epoch), lives in struct
pv_vcpu, as only PV vCPUs have a mapcache.  In preparation for
implementing mapcaches for HVM domains, move it to struct arch_vcpu:
v->arch.pv.mapcache becomes v->arch.mapcache.

struct pv_vcpu shrinks from 512 to 376 bytes, and struct vcpu grows
from 2880 to 3008 bytes (x86_64, debug build), within its page: from
here on HVM vCPUs carry the 136-byte struct mapcache_vcpu too, ahead of
its use by those of vCPU-PT domains.

No functional change.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
  mapcache for vCPU-PT domains" for review.
---
 xen/arch/x86/domain_page.c        | 6 +++---
 xen/arch/x86/include/asm/domain.h | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 5d90f77bed..f91ec8a75a 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -134,7 +134,7 @@ void *map_domain_page(mfn_t mfn)
         return mfn_to_virt(mfn_x(mfn));
 
     cache = vcpu_mapcache(v, &dcache);
-    vcache = &v->arch.pv.mapcache;
+    vcache = &v->arch.mapcache;
     if ( !cache->inuse )
         return mfn_to_virt(mfn_x(mfn));
 
@@ -231,7 +231,7 @@ void unmap_domain_page(const void *ptr)
 
     idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
     mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
-    hashent = &v->arch.pv.mapcache.hash[MAPHASH_HASHFN(mfn)];
+    hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
 
     local_irq_save(flags);
 
@@ -334,7 +334,7 @@ int mapcache_vcpu_init(struct vcpu *v)
     BUILD_BUG_ON(MAPHASHENT_NOTINUSE < MAPCACHE_ENTRIES);
     for ( i = 0; i < MAPHASH_ENTRIES; i++ )
     {
-        struct vcpu_maphash_entry *hashent = &v->arch.pv.mapcache.hash[i];
+        struct vcpu_maphash_entry *hashent = &v->arch.mapcache.hash[i];
 
         hashent->mfn = ~0UL; /* never valid to map */
         hashent->idx = MAPHASHENT_NOTINUSE;
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 6f5873879e..cfa24c4e75 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -552,9 +552,6 @@ struct arch_domain
 
 struct pv_vcpu
 {
-    /* map_domain_page() mapping cache. */
-    struct mapcache_vcpu mapcache;
-
     unsigned int vgc_flags;
 
     struct trap_info *trap_ctxt;
@@ -650,6 +647,9 @@ struct arch_vcpu
 #define async_exception_state(t) async_exception_state[(t)-1]
     uint8_t async_exception_mask;
 
+    /* map_domain_page() mapping cache. */
+    struct mapcache_vcpu mapcache;
+
     /* Virtual Machine Extensions */
     union {
         struct pv_vcpu pv;
-- 
2.55.0

Re: [PATCH v3 09/18] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:42AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> struct mapcache_vcpu, the per-vCPU part of the mapcache (the maphash
> and the vCPU's shadow of the domain's TLB epoch), lives in struct
> pv_vcpu, as only PV vCPUs have a mapcache.  In preparation for
> implementing mapcaches for HVM domains, move it to struct arch_vcpu:
> v->arch.pv.mapcache becomes v->arch.mapcache.
> 
> struct pv_vcpu shrinks from 512 to 376 bytes, and struct vcpu grows
> from 2880 to 3008 bytes (x86_64, debug build), within its page: from
> here on HVM vCPUs carry the 136-byte struct mapcache_vcpu too, ahead of
> its use by those of vCPU-PT domains.
> 
> No functional change.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Reviewed-by: Roger Pau Monné <roger@xenproject.org>

> ---
> Changes in v3:
> - New in this version, split out of "x86/hvm: introduce per-vCPU
>   mapcache for vCPU-PT domains" for review.
> ---
>  xen/arch/x86/domain_page.c        | 6 +++---
>  xen/arch/x86/include/asm/domain.h | 6 +++---
>  2 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index 5d90f77bed..f91ec8a75a 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -134,7 +134,7 @@ void *map_domain_page(mfn_t mfn)
>          return mfn_to_virt(mfn_x(mfn));
>  
>      cache = vcpu_mapcache(v, &dcache);
> -    vcache = &v->arch.pv.mapcache;
> +    vcache = &v->arch.mapcache;
>      if ( !cache->inuse )
>          return mfn_to_virt(mfn_x(mfn));
>  
> @@ -231,7 +231,7 @@ void unmap_domain_page(const void *ptr)
>  
>      idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
>      mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
> -    hashent = &v->arch.pv.mapcache.hash[MAPHASH_HASHFN(mfn)];
> +    hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
>  
>      local_irq_save(flags);
>  
> @@ -334,7 +334,7 @@ int mapcache_vcpu_init(struct vcpu *v)
>      BUILD_BUG_ON(MAPHASHENT_NOTINUSE < MAPCACHE_ENTRIES);
>      for ( i = 0; i < MAPHASH_ENTRIES; i++ )
>      {
> -        struct vcpu_maphash_entry *hashent = &v->arch.pv.mapcache.hash[i];
> +        struct vcpu_maphash_entry *hashent = &v->arch.mapcache.hash[i];
>  
>          hashent->mfn = ~0UL; /* never valid to map */
>          hashent->idx = MAPHASHENT_NOTINUSE;
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index 6f5873879e..cfa24c4e75 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -552,9 +552,6 @@ struct arch_domain
>  
>  struct pv_vcpu
>  {
> -    /* map_domain_page() mapping cache. */
> -    struct mapcache_vcpu mapcache;
> -
>      unsigned int vgc_flags;
>  
>      struct trap_info *trap_ctxt;
> @@ -650,6 +647,9 @@ struct arch_vcpu
>  #define async_exception_state(t) async_exception_state[(t)-1]
>      uint8_t async_exception_mask;
>  
> +    /* map_domain_page() mapping cache. */
> +    struct mapcache_vcpu mapcache;

FWIW, I would maybe place this at the top of the structure, like it
was placed in pv_vcpu?

In any case the placement of the fields of arch_vcpu is a mess
already, so it doesn't make much of a difference.

Thanks, Roger.

[PATCH v3 10/18] x86/domain_page: don't unmap a mapcache entry that is not present

[George Dunlap]

Mappings must be torn down by the vCPU that created them, and only
once.  Have unmap_domain_page() check that the entry it is handed is
present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
an MFN out of an empty PTE and act on it, as a second unmap of an entry
the first one zapped would.  (A second unmap of an entry the maphash
kept is caught, as before, only by the refcount ASSERT() in debug
builds.)

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
  mapcache for vCPU-PT domains" for review.
---
 xen/arch/x86/domain_page.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f91ec8a75a..708df90a00 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -230,6 +230,16 @@ void unmap_domain_page(const void *ptr)
     ASSERT(cache->inuse);
 
     idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
+    /*
+     * Mappings must be unmapped once, by the vCPU that created them.  With
+     * nothing present there is nothing to tear down: leave the hash and the
+     * bitmaps alone rather than act on a stale or foreign pointer.
+     */
+    if ( !(l1e_get_flags(MAPCACHE_L1ENT(idx)) & _PAGE_PRESENT) )
+    {
+        ASSERT_UNREACHABLE();
+        return;
+    }
     mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
     hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
 
-- 
2.55.0

Re: [PATCH v3 10/18] x86/domain_page: don't unmap a mapcache entry that is not present

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:43AM +0100, George Dunlap wrote:
> Mappings must be torn down by the vCPU that created them, and only
> once.  Have unmap_domain_page() check that the entry it is handed is
> present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
> an MFN out of an empty PTE and act on it, as a second unmap of an entry
> the first one zapped would.  (A second unmap of an entry the maphash
> kept is caught, as before, only by the refcount ASSERT() in debug
> builds.)
> 
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>

Reviewed-by: Roger Pau Monné <roger@xenporject.org>

> ---
> Changes in v3:
> - New in this version, split out of "x86/hvm: introduce per-vCPU
>   mapcache for vCPU-PT domains" for review.
> ---
>  xen/arch/x86/domain_page.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index f91ec8a75a..708df90a00 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -230,6 +230,16 @@ void unmap_domain_page(const void *ptr)
>      ASSERT(cache->inuse);
>  
>      idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
> +    /*
> +     * Mappings must be unmapped once, by the vCPU that created them.  With
> +     * nothing present there is nothing to tear down: leave the hash and the
> +     * bitmaps alone rather than act on a stale or foreign pointer.

The comment makes more claims that what can be checked for: there's no
check that the unmap is being done by the same vCPU that created it,
neither is easy to add one.  Any mismatch like that would become more
obvious once we switch to a per-vCPU mapcache.  I would possibly
reduce the comment to:

/* Ensure the entry to unmap is populated. */

The commit message already have the lengthily details.

Thanks, Roger.

Re: [PATCH v3 10/18] x86/domain_page: don't unmap a mapcache entry that is not present

[George Dunlap]

On Thu, Oct 8, 2026 at 12:04 PM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Wed, Oct 07, 2026 at 11:40:43AM +0100, George Dunlap wrote:
> > Mappings must be torn down by the vCPU that created them, and only
> > once.  Have unmap_domain_page() check that the entry it is handed is
> > present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
> > an MFN out of an empty PTE and act on it, as a second unmap of an entry
> > the first one zapped would.  (A second unmap of an entry the maphash
> > kept is caught, as before, only by the refcount ASSERT() in debug
> > builds.)
> >
> > Assisted-by: Claude Code:claude-opus-5-5
> > Signed-off-by: George Dunlap <gwd@xenproject.org>
>
> Reviewed-by: Roger Pau Monné <roger@xenporject.org>
>
> > ---
> > Changes in v3:
> > - New in this version, split out of "x86/hvm: introduce per-vCPU
> >   mapcache for vCPU-PT domains" for review.
> > ---
> >  xen/arch/x86/domain_page.c | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> >
> > diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> > index f91ec8a75a..708df90a00 100644
> > --- a/xen/arch/x86/domain_page.c
> > +++ b/xen/arch/x86/domain_page.c
> > @@ -230,6 +230,16 @@ void unmap_domain_page(const void *ptr)
> >      ASSERT(cache->inuse);
> >
> >      idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
> > +    /*
> > +     * Mappings must be unmapped once, by the vCPU that created them.  With
> > +     * nothing present there is nothing to tear down: leave the hash and the
> > +     * bitmaps alone rather than act on a stale or foreign pointer.
>
> The comment makes more claims that what can be checked for: there's no
> check that the unmap is being done by the same vCPU that created it,
> neither is easy to add one.  Any mismatch like that would become more
> obvious once we switch to a per-vCPU mapcache.  I would possibly
> reduce the comment to:
>
> /* Ensure the entry to unmap is populated. */
>
> The commit message already have the lengthily details.

The problem is that now the comment just repeats the condition of the
if() statement, which an experienced developer can already read
without effort; it doesn't say *why* we're doing the check.  What
about something like the following?

    /*
     * Sanity check: an entry that isn't present means a double or stray
     * unmap.  Play it safe: don't act on the MFN in an empty PTE.
     */

Thanks,
 -George

Re: [PATCH v3 10/18] x86/domain_page: don't unmap a mapcache entry that is not present

[Roger Pau Monné]

On Tue, Oct 13, 2026 at 11:33:55AM +0100, George Dunlap wrote:
> On Thu, Oct 8, 2026 at 12:04 PM Roger Pau Monné <roger@xenproject.org> wrote:
> >
> > On Wed, Oct 07, 2026 at 11:40:43AM +0100, George Dunlap wrote:
> > > Mappings must be torn down by the vCPU that created them, and only
> > > once.  Have unmap_domain_page() check that the entry it is handed is
> > > present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
> > > an MFN out of an empty PTE and act on it, as a second unmap of an entry
> > > the first one zapped would.  (A second unmap of an entry the maphash
> > > kept is caught, as before, only by the refcount ASSERT() in debug
> > > builds.)
> > >
> > > Assisted-by: Claude Code:claude-opus-5-5
> > > Signed-off-by: George Dunlap <gwd@xenproject.org>
> >
> > Reviewed-by: Roger Pau Monné <roger@xenporject.org>
> >
> > > ---
> > > Changes in v3:
> > > - New in this version, split out of "x86/hvm: introduce per-vCPU
> > >   mapcache for vCPU-PT domains" for review.
> > > ---
> > >  xen/arch/x86/domain_page.c | 10 ++++++++++
> > >  1 file changed, 10 insertions(+)
> > >
> > > diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> > > index f91ec8a75a..708df90a00 100644
> > > --- a/xen/arch/x86/domain_page.c
> > > +++ b/xen/arch/x86/domain_page.c
> > > @@ -230,6 +230,16 @@ void unmap_domain_page(const void *ptr)
> > >      ASSERT(cache->inuse);
> > >
> > >      idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
> > > +    /*
> > > +     * Mappings must be unmapped once, by the vCPU that created them.  With
> > > +     * nothing present there is nothing to tear down: leave the hash and the
> > > +     * bitmaps alone rather than act on a stale or foreign pointer.
> >
> > The comment makes more claims that what can be checked for: there's no
> > check that the unmap is being done by the same vCPU that created it,
> > neither is easy to add one.  Any mismatch like that would become more
> > obvious once we switch to a per-vCPU mapcache.  I would possibly
> > reduce the comment to:
> >
> > /* Ensure the entry to unmap is populated. */
> >
> > The commit message already have the lengthily details.
> 
> The problem is that now the comment just repeats the condition of the
> if() statement, which an experienced developer can already read
> without effort; it doesn't say *why* we're doing the check.  What
> about something like the following?
> 
>     /*
>      * Sanity check: an entry that isn't present means a double or stray
>      * unmap.  Play it safe: don't act on the MFN in an empty PTE.
>      */

Oh, I'm actually happy if you want to drop the comment altogether.  I
think the check + ASSERT_UNREACHABLE() is already clear to understand.
And we expect anyone touching this code to know that a double or stray
free of an entry that's not present is not a good thing.

Thanks, Roger.

[PATCH v3 11/18] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains

[George Dunlap]

From: Roger Pau Monné <roger.pau@citrix.com>

Add a struct mapcache to struct mapcache_vcpu for the per-vCPU
accounting. Since MAPCACHE_VCPU_ENTRIES is 16, its inuse and garbage
bitmaps are a word each: declare them inline in struct mapcache_vcpu,
rather than mapping them in the per-domain area as the domain-wide
bitmaps are.

A per-vCPU mapcache is only ever used by its own vCPU, and so by one
pCPU at a time. We can therefore skip both the lock and the TLB-epoch
tracking the domain-wide mapcache needs; a wrap still flushes the
local TLB.

struct mapcache_vcpu grows from 136 to 176 bytes, and struct vcpu from
3008 to 3072 bytes (x86_64, debug build), within its page.

Unlike a domain-wide mapcache, a per-vCPU one cannot, by design,
borrow slots from other vCPUs: MAPCACHE_VCPU_ENTRIES (16) is the most
single-page mappings such a vCPU can hold at once, interrupt handlers'
included, and one more hits the BUG_ON() in map_domain_page(). The PoD
zero-page sweep held up to 16 on top of its callers' until "x86/PoD: map
one page at a time in p2m_pod_zero_check()".

mapcache_current_vcpu()'s lazy-context-switch handling
(sync_local_execstate() plus a pgtable_vcpu re-read) now applies to
any vCPU with a mapcache: when the idle vCPU runs lazily on an HVM
vCPU's monitor table, the full switch happens before the idle context
uses a mapcache.

In release builds map_domain_page() still tries the directmap first,
for any vCPU; debug builds exercise the new mapcache.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- HVM only, and per-vCPU from the start: the HVM enablement of "x86:
  Initialize the mapcache for PV, HVM, and idle domains" (from the
  directmap removal series) is folded in, without the idle domain and
  without a domain-wide HVM mapcache.
- Retitle (was: "x86/mm: introduce a per-vCPU mapcache when using ASI").
- Say which second unmaps the not-present check in unmap_domain_page()
  catches: those of an entry the first unmap zapped.
- State the budget of single-page mappings a vCPU can hold at once.
- The mapcache accounting refactor, the move of struct mapcache_vcpu
  to struct arch_vcpu and the not-present check in unmap_domain_page()
  are split out, for review, into the three patches before this one.
---
 xen/arch/x86/domain_page.c        | 70 ++++++++++++++++++++++---------
 xen/arch/x86/include/asm/domain.h |  6 +++
 2 files changed, 57 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 708df90a00..0500bb9dba 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -20,12 +20,13 @@
 
 /*
  * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
- * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
- * the directmap, which covers all physical address space for them.
+ * disabled, see mapcache_domain_init()) and the vCPUs of an HVM domain using
+ * per-vCPU page-tables.  Other HVM vCPUs reach everything through the
+ * directmap, which covers all physical address space for them.
  */
 static bool has_mapcache(const struct vcpu *v)
 {
-    return is_pv_vcpu(v);
+    return is_pv_vcpu(v) || v->domain->arch.vcpu_pt;
 }
 
 static inline struct vcpu *mapcache_current_vcpu(void)
@@ -38,10 +39,10 @@ static inline struct vcpu *mapcache_current_vcpu(void)
         return NULL;
 
     /*
-     * If we are in a lazy context-switch state from a PV vCPU do a full switch
-     * to the idle vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would
-     * change the page tables under our feet an invalidate any in-use mapcache
-     * entries.
+     * If we are in a lazy context-switch state from a vCPU with a mapcache do
+     * a full switch to the idle vCPU now, otherwise an incoming
+     * FLUSH_VCPU_STATE IPI would change the page tables under our feet an
+     * invalidate any in-use mapcache entries.
      */
     if ( unlikely(this_cpu(curr_vcpu) != curr) )
     {
@@ -61,14 +62,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
 }
 
 /*
- * The accounting for v's mapcache: the domain's, with *dcache set so the
- * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
+ * otherwise, in which case *dcache is set so the caller takes the domain-wide
+ * lock and TLB-epoch machinery into account.
  */
 static struct mapcache *vcpu_mapcache(struct vcpu *v,
                                       struct mapcache_domain **dcache)
 {
     struct domain *d = v->domain;
 
+    if ( d->arch.vcpu_pt )
+    {
+        *dcache = NULL;
+        return &v->arch.mapcache.cache;
+    }
+
     *dcache = &d->arch.pv.mapcache;
     return &(*dcache)->cache;
 }
@@ -77,11 +85,15 @@ static struct mapcache *vcpu_mapcache(struct vcpu *v,
  * The domain-wide mapcache's lock and TLB-epoch machinery, taken around
  * allocating a slot: the domain's vCPUs share the mapcache's slots, and each
  * catches up with a wrap another one caused, flushing its TLB if it has not
- * been flushed since.
+ * been flushed since.  A per-vCPU mapcache (no dcache) needs neither: only its
+ * own vCPU, and so one pCPU at a time, uses it.
  */
 static void mapcache_lock(struct mapcache_domain *dcache,
                           struct mapcache_vcpu *vcache)
 {
+    if ( !dcache )
+        return;
+
     spin_lock(&dcache->lock);
 
     /* Has some other CPU caused a wrap? We must flush if so. */
@@ -100,13 +112,17 @@ static void mapcache_lock(struct mapcache_domain *dcache,
 static void mapcache_new_epoch(struct mapcache_domain *dcache,
                                struct mapcache_vcpu *vcache)
 {
+    if ( !dcache )
+        return;
+
     vcache->shadow_epoch = ++dcache->epoch;
     dcache->tlbflush_timestamp = tlbflush_current_time();
 }
 
 static void mapcache_unlock(struct mapcache_domain *dcache)
 {
-    spin_unlock(&dcache->lock);
+    if ( dcache )
+        spin_unlock(&dcache->lock);
 }
 
 #define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
@@ -191,7 +207,12 @@ void *map_domain_page(mfn_t mfn)
         }
         BUG_ON(idx >= cache->entries);
 
-        /* /Second/, flush TLBs. */
+        /*
+         * /Second/, flush TLBs.  A per-vCPU mapcache needs a local flush
+         * only: its entries are never _PAGE_GLOBAL, and this vCPU's
+         * page-tables can only have been loaded on another pCPU via
+         * switch_cr3_cr4(), which flushes non-global mappings.
+         */
         perfc_incr(domain_page_tlb_flush);
         flush_tlb_local();
         mapcache_new_epoch(dcache, vcache);
@@ -313,11 +334,19 @@ int mapcache_vcpu_init(struct vcpu *v)
         return 0;
 
     cache = vcpu_mapcache(v, &dcache);
-    if ( !cache->inuse )
+    if ( !dcache )
+    {
+        /* Per-vCPU bitmaps are a word each: they live in the vCPU itself. */
+        cache->inuse = v->arch.mapcache.inuse;
+        cache->garbage = v->arch.mapcache.garbage;
+        ents = MAPCACHE_VCPU_ENTRIES;
+    }
+    else if ( !cache->inuse )
         /* Domain-wide mapcache disabled (fully direct-mapped build). */
         return 0;
+    else
+        ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
 
-    ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
     nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
 
     if ( ents > cache->entries )
@@ -326,13 +355,16 @@ int mapcache_vcpu_init(struct vcpu *v)
         int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
                                           NIL(l1_pgentry_t *), NULL);
 
-        /* Populate bit maps. */
-        if ( !rc )
+        /* Populate the domain-wide bit maps (kept in the per-domain area). */
+        if ( !rc && dcache )
+        {
             rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
                                           nr, NULL, NIL(struct page_info *));
-        if ( !rc )
-            rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
-                                          nr, NULL, NIL(struct page_info *));
+            if ( !rc )
+                rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
+                                              nr, NULL,
+                                              NIL(struct page_info *));
+        }
 
         if ( rc )
             return rc;
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index cfa24c4e75..80415ab2c2 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -80,6 +80,12 @@ struct mapcache_vcpu {
         uint32_t      idx;
         uint32_t      refcnt;
     } hash[MAPHASH_ENTRIES];
+
+    /* Accounting of a per-vCPU mapcache (vCPU-PT domains). */
+    struct mapcache cache;
+    /* Backing for cache.{inuse,garbage}. */
+    DECLARE_BITMAP(inuse, MAPCACHE_VCPU_ENTRIES);
+    DECLARE_BITMAP(garbage, MAPCACHE_VCPU_ENTRIES);
 };
 
 struct mapcache_domain {
-- 
2.55.0

Re: [PATCH v3 11/18] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:44AM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
> 
> Add a struct mapcache to struct mapcache_vcpu for the per-vCPU
> accounting. Since MAPCACHE_VCPU_ENTRIES is 16, its inuse and garbage
> bitmaps are a word each: declare them inline in struct mapcache_vcpu,
> rather than mapping them in the per-domain area as the domain-wide
> bitmaps are.
> 
> A per-vCPU mapcache is only ever used by its own vCPU, and so by one
> pCPU at a time. We can therefore skip both the lock and the TLB-epoch
> tracking the domain-wide mapcache needs; a wrap still flushes the
> local TLB.
> 
> struct mapcache_vcpu grows from 136 to 176 bytes, and struct vcpu from
> 3008 to 3072 bytes (x86_64, debug build), within its page.
> 
> Unlike a domain-wide mapcache, a per-vCPU one cannot, by design,
> borrow slots from other vCPUs: MAPCACHE_VCPU_ENTRIES (16) is the most
> single-page mappings such a vCPU can hold at once, interrupt handlers'
> included, and one more hits the BUG_ON() in map_domain_page(). The PoD
> zero-page sweep held up to 16 on top of its callers' until "x86/PoD: map
> one page at a time in p2m_pod_zero_check()".

Is it really the best approach to keep the same limit?  As you note,
with the mapcache being per-vcpu the limit of 16 active mappings is
going to be strictly enforce, which it wasn't before.

The per-vcpu mapcache area is a whole L1 page-directory (it cannot be
smaller), so that's 512 entries per-vcpu, of which we limit ourselves
to 16.

Can we consider using bigger per-vcpu mapcaches, maybe 512 is overkill
for the bitmap sizes, but using greater than 16 will likely avoid
frequent TLB flushes to re-use the entries.  Maybe 64 entries, so the
bitmap is still a single unsigned long on 64bit arches.  Note that
declare bitmap will already use at least an unsigned long for the
bitmap itself.

> 
> mapcache_current_vcpu()'s lazy-context-switch handling
> (sync_local_execstate() plus a pgtable_vcpu re-read) now applies to
> any vCPU with a mapcache: when the idle vCPU runs lazily on an HVM
> vCPU's monitor table, the full switch happens before the idle context
> uses a mapcache.
> 
> In release builds map_domain_page() still tries the directmap first,
> for any vCPU; debug builds exercise the new mapcache.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - HVM only, and per-vCPU from the start: the HVM enablement of "x86:
>   Initialize the mapcache for PV, HVM, and idle domains" (from the
>   directmap removal series) is folded in, without the idle domain and
>   without a domain-wide HVM mapcache.
> - Retitle (was: "x86/mm: introduce a per-vCPU mapcache when using ASI").
> - Say which second unmaps the not-present check in unmap_domain_page()
>   catches: those of an entry the first unmap zapped.
> - State the budget of single-page mappings a vCPU can hold at once.
> - The mapcache accounting refactor, the move of struct mapcache_vcpu
>   to struct arch_vcpu and the not-present check in unmap_domain_page()
>   are split out, for review, into the three patches before this one.
> ---
>  xen/arch/x86/domain_page.c        | 70 ++++++++++++++++++++++---------
>  xen/arch/x86/include/asm/domain.h |  6 +++
>  2 files changed, 57 insertions(+), 19 deletions(-)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index 708df90a00..0500bb9dba 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -20,12 +20,13 @@
>  
>  /*
>   * Whether v has a mapcache: PV vCPUs (whose domain-wide mapcache may still be
> - * disabled, see mapcache_domain_init()).  HVM vCPUs reach everything through
> - * the directmap, which covers all physical address space for them.
> + * disabled, see mapcache_domain_init()) and the vCPUs of an HVM domain using
> + * per-vCPU page-tables.  Other HVM vCPUs reach everything through the
> + * directmap, which covers all physical address space for them.
>   */
>  static bool has_mapcache(const struct vcpu *v)
>  {
> -    return is_pv_vcpu(v);
> +    return is_pv_vcpu(v) || v->domain->arch.vcpu_pt;
>  }
>  
>  static inline struct vcpu *mapcache_current_vcpu(void)
> @@ -38,10 +39,10 @@ static inline struct vcpu *mapcache_current_vcpu(void)
>          return NULL;
>  
>      /*
> -     * If we are in a lazy context-switch state from a PV vCPU do a full switch
> -     * to the idle vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would
> -     * change the page tables under our feet an invalidate any in-use mapcache
> -     * entries.
> +     * If we are in a lazy context-switch state from a vCPU with a mapcache do
> +     * a full switch to the idle vCPU now, otherwise an incoming
> +     * FLUSH_VCPU_STATE IPI would change the page tables under our feet an
> +     * invalidate any in-use mapcache entries.
>       */
>      if ( unlikely(this_cpu(curr_vcpu) != curr) )
>      {
> @@ -61,14 +62,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
>  }
>  
>  /*
> - * The accounting for v's mapcache: the domain's, with *dcache set so the
> - * caller takes the domain-wide lock and TLB-epoch machinery into account.
> + * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
> + * otherwise, in which case *dcache is set so the caller takes the domain-wide
> + * lock and TLB-epoch machinery into account.
>   */
>  static struct mapcache *vcpu_mapcache(struct vcpu *v,
>                                        struct mapcache_domain **dcache)
>  {
>      struct domain *d = v->domain;
>  
> +    if ( d->arch.vcpu_pt )
> +    {
> +        *dcache = NULL;
> +        return &v->arch.mapcache.cache;
> +    }
> +
>      *dcache = &d->arch.pv.mapcache;
>      return &(*dcache)->cache;
>  }
> @@ -77,11 +85,15 @@ static struct mapcache *vcpu_mapcache(struct vcpu *v,
>   * The domain-wide mapcache's lock and TLB-epoch machinery, taken around
>   * allocating a slot: the domain's vCPUs share the mapcache's slots, and each
>   * catches up with a wrap another one caused, flushing its TLB if it has not
> - * been flushed since.
> + * been flushed since.  A per-vCPU mapcache (no dcache) needs neither: only its
> + * own vCPU, and so one pCPU at a time, uses it.
>   */
>  static void mapcache_lock(struct mapcache_domain *dcache,
>                            struct mapcache_vcpu *vcache)
>  {
> +    if ( !dcache )
> +        return;
> +
>      spin_lock(&dcache->lock);
>  
>      /* Has some other CPU caused a wrap? We must flush if so. */
> @@ -100,13 +112,17 @@ static void mapcache_lock(struct mapcache_domain *dcache,
>  static void mapcache_new_epoch(struct mapcache_domain *dcache,
>                                 struct mapcache_vcpu *vcache)
>  {
> +    if ( !dcache )
> +        return;
> +
>      vcache->shadow_epoch = ++dcache->epoch;
>      dcache->tlbflush_timestamp = tlbflush_current_time();
>  }
>  
>  static void mapcache_unlock(struct mapcache_domain *dcache)
>  {
> -    spin_unlock(&dcache->lock);
> +    if ( dcache )
> +        spin_unlock(&dcache->lock);
>  }
>  
>  #define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
> @@ -191,7 +207,12 @@ void *map_domain_page(mfn_t mfn)
>          }
>          BUG_ON(idx >= cache->entries);
>  
> -        /* /Second/, flush TLBs. */
> +        /*
> +         * /Second/, flush TLBs.  A per-vCPU mapcache needs a local flush
> +         * only: its entries are never _PAGE_GLOBAL, and this vCPU's
> +         * page-tables can only have been loaded on another pCPU via
> +         * switch_cr3_cr4(), which flushes non-global mappings.
> +         */

Shouldn't the above comment be inside of mapcache_new_epoch()?  It's a
bit disconnected from the code here, as apparently there's no
different handling here between per-domain and per-vcpu mapcaches.

>          perfc_incr(domain_page_tlb_flush);
>          flush_tlb_local();
>          mapcache_new_epoch(dcache, vcache);
> @@ -313,11 +334,19 @@ int mapcache_vcpu_init(struct vcpu *v)
>          return 0;
>  
>      cache = vcpu_mapcache(v, &dcache);
> -    if ( !cache->inuse )
> +    if ( !dcache )
> +    {
> +        /* Per-vCPU bitmaps are a word each: they live in the vCPU itself. */
> +        cache->inuse = v->arch.mapcache.inuse;
> +        cache->garbage = v->arch.mapcache.garbage;

Well, the size of the bitmap is set to 16 bits, but due to the usage of
DECLARE_BITMAP() the underlying type is unsigned long, so it's
effectively using 64 bits (a quad).

Thanks, Roger.

[PATCH v3 12/18] x86/mm: let Xen's page-table walker operate on a given root

[George Dunlap]

map_pages_to_xen() and modify_xen_mappings() maintain Xen's own mappings,
walking from idle_pg_table, whose Xen slots every other root page-table
copies.  A later patch introduces a second directmap hierarchy: page
tables for the directmap virtual address range that some guest contexts
use in place of idle_pg_table's.  It needs the same maintenance at the
same virtual addresses.

Give virt_to_xen_l{3,2,1}e() the L4 to walk from, and move the bodies of
map_pages_to_xen() and modify_xen_mappings() into map_pages_in() and
modify_mappings_in(), which take the root as well.  The public functions
pass idle_pg_table.  The EFI runtime-services page-tables mirror
idle_pg_table's L4 entries only, so a newly allocated L3 is propagated
to them only when the root is idle_pg_table.

TLB flushing is unaffected: it is by virtual address, and the same
addresses are mapped in either hierarchy.

No functional change.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/mm.c | 55 ++++++++++++++++++++++++++++++++++-------------
 1 file changed, 40 insertions(+), 15 deletions(-)

diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 7258935528..94b96c3d9b 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5201,13 +5201,14 @@ static DEFINE_SPINLOCK(map_pgdir_lock);
 
 /*
  * For virt_to_xen_lXe() functions, they take a linear address and return a
- * pointer to Xen's LX entry. Caller needs to unmap the pointer.
+ * pointer to Xen's LX entry in the page-table hierarchy rooted at @root
+ * (idle_pg_table for Xen's own mappings).  Caller needs to unmap the pointer.
  */
-static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
+static l3_pgentry_t *virt_to_xen_l3e(l4_pgentry_t *root, unsigned long v)
 {
     l4_pgentry_t *pl4e;
 
-    pl4e = &idle_pg_table[l4_table_offset(v)];
+    pl4e = &root[l4_table_offset(v)];
     if ( !(l4e_get_flags(*pl4e) & _PAGE_PRESENT) )
     {
         bool locking = system_state > SYS_STATE_boot;
@@ -5223,7 +5224,8 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
             l4_pgentry_t l4e = l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
 
             l4e_write(pl4e, l4e);
-            efi_update_l4_pgtable(l4_table_offset(v), l4e);
+            if ( root == idle_pg_table )
+                efi_update_l4_pgtable(l4_table_offset(v), l4e);
             l3mfn = INVALID_MFN;
         }
         if ( locking )
@@ -5234,11 +5236,11 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
     return map_l3t_from_l4e(*pl4e) + l3_table_offset(v);
 }
 
-static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
+static l2_pgentry_t *virt_to_xen_l2e(l4_pgentry_t *root, unsigned long v)
 {
     l3_pgentry_t *pl3e, l3e;
 
-    pl3e = virt_to_xen_l3e(v);
+    pl3e = virt_to_xen_l3e(root, v);
     if ( !pl3e )
         return NULL;
 
@@ -5272,11 +5274,11 @@ static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
     return map_l2t_from_l3e(l3e) + l2_table_offset(v);
 }
 
-static l1_pgentry_t *virt_to_xen_l1e(unsigned long v)
+static l1_pgentry_t *virt_to_xen_l1e(l4_pgentry_t *root, unsigned long v)
 {
     l2_pgentry_t *pl2e, l2e;
 
-    pl2e = virt_to_xen_l2e(v);
+    pl2e = virt_to_xen_l2e(root, v);
     if ( !pl2e )
         return NULL;
 
@@ -5352,7 +5354,7 @@ mfn_t xen_map_to_mfn(unsigned long va)
     bool locking = system_state > SYS_STATE_boot;
     unsigned int l2_offset = l2_table_offset(va);
     unsigned int l1_offset = l1_table_offset(va);
-    const l3_pgentry_t *pl3e = virt_to_xen_l3e(va);
+    const l3_pgentry_t *pl3e = virt_to_xen_l3e(idle_pg_table, va);
     const l2_pgentry_t *pl2e = NULL;
     const l1_pgentry_t *pl1e = NULL;
     struct page_info *l3page;
@@ -5392,7 +5394,12 @@ mfn_t xen_map_to_mfn(unsigned long va)
     return ret;
 }
 
-int map_pages_to_xen(
+/*
+ * map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
+ * another hierarchy sharing its virtual addresses.
+ */
+static int map_pages_in(
+    l4_pgentry_t *root,
     unsigned long virt,
     mfn_t mfn,
     unsigned long nr_mfns,
@@ -5443,7 +5450,7 @@ int map_pages_to_xen(
         UNMAP_DOMAIN_PAGE(pl3e);
         UNMAP_DOMAIN_PAGE(pl2e);
 
-        pl3e = virt_to_xen_l3e(virt);
+        pl3e = virt_to_xen_l3e(root, virt);
         if ( !pl3e )
             goto out;
 
@@ -5567,7 +5574,7 @@ int map_pages_to_xen(
             free_xen_pagetable(l2mfn);
         }
 
-        pl2e = virt_to_xen_l2e(virt);
+        pl2e = virt_to_xen_l2e(root, virt);
         if ( !pl2e )
             goto out;
 
@@ -5612,7 +5619,7 @@ int map_pages_to_xen(
             /* Normal page mapping. */
             if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
             {
-                pl1e = virt_to_xen_l1e(virt);
+                pl1e = virt_to_xen_l1e(root, virt);
                 if ( pl1e == NULL )
                     goto out;
             }
@@ -5797,6 +5804,15 @@ int map_pages_to_xen(
     return rc;
 }
 
+int map_pages_to_xen(
+    unsigned long virt,
+    mfn_t mfn,
+    unsigned long nr_mfns,
+    pte_attr_t flags)
+{
+    return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+}
+
 int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
 {
     return map_pages_to_xen(virt, INVALID_MFN, nr_mfns, MAP_SMALL_PAGES);
@@ -5813,8 +5829,12 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
  * flags are ignored.
  *
  * It is an error to call with present flags over an unpopulated range.
+ *
+ * modify_mappings_in() does so in the hierarchy rooted at @root, as
+ * map_pages_in() does.
  */
-int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+                              unsigned long e, pte_attr_t nf)
 {
     bool locking = system_state > SYS_STATE_boot;
     l3_pgentry_t *pl3e = NULL;
@@ -5841,7 +5861,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
         UNMAP_DOMAIN_PAGE(pl2e);
         UNMAP_DOMAIN_PAGE(pl3e);
 
-        pl3e = virt_to_xen_l3e(v);
+        pl3e = virt_to_xen_l3e(root, v);
         if ( !pl3e )
             goto out;
 
@@ -6092,6 +6112,11 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
 
 #undef flush_area
 
+int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+{
+    return modify_mappings_in(idle_pg_table, s, e, nf);
+}
+
 int destroy_xen_mappings(unsigned long s, unsigned long e)
 {
     return modify_xen_mappings(s, e, _PAGE_NONE);
-- 
2.55.0

Re: [PATCH v3 12/18] x86/mm: let Xen's page-table walker operate on a given root

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:45AM +0100, George Dunlap wrote:
> map_pages_to_xen() and modify_xen_mappings() maintain Xen's own mappings,
> walking from idle_pg_table, whose Xen slots every other root page-table
> copies.  A later patch introduces a second directmap hierarchy: page
> tables for the directmap virtual address range that some guest contexts
> use in place of idle_pg_table's.  It needs the same maintenance at the
> same virtual addresses.

I assume that "second directmap hierarchy" refers to the sparse
directmap approach?

> Give virt_to_xen_l{3,2,1}e() the L4 to walk from, and move the bodies of
> map_pages_to_xen() and modify_xen_mappings() into map_pages_in() and
> modify_mappings_in(), which take the root as well.  The public functions
> pass idle_pg_table.  The EFI runtime-services page-tables mirror
> idle_pg_table's L4 entries only, so a newly allocated L3 is propagated
> to them only when the root is idle_pg_table.
> 
> TLB flushing is unaffected: it is by virtual address, and the same
> addresses are mapped in either hierarchy.

The virtual address space is just part of it, so far
map_pages_to_xen() has been doing global TLB flushes, because all CPUs
used the same page-tables for the Xen slots that map_pages_to_xen()
modifies.  With the change to pass different L4, there's also the
assumption that the Xen slots on those L4 will be different, and hence
not all CPUs might need flushing?

Maybe I'm thinking ahead too much - but if this is going to be used to
manage the sparse directmap we might want to avoid IPIs to CPUs that
are running with the full directmap when the sparse one is changed,
as those are useless.

Thanks, Roger.

[PATCH v3 13/18] x86/mm: build and maintain a sparse view of the directmap

[George Dunlap]

The directmap maps all of RAM, and every root page-table carries it, so
while any guest context runs, all guest memory is reachable from Xen's
address space.  For ASI, guest contexts are to run on a view of the
directmap which maps only what Xen must reach through it at all times,
leaving guest memory to transient mappings (map_domain_page()).

Rather than remove the directmap for everyone, keep the full directmap
as it is, for the idle domain and for guest contexts not using ASI, and
build a second, sparse view: page-tables for the same virtual addresses
(the part of the directmap PV guests can see, where everything Xen
reaches through the directmap lives), mapping a subset of what the full
directmap maps, each entry a copy of the full directmap's.  Boot, dom0
construction and the idle domain keep the full directmap, so nothing in
them needs converting.

The sparse view maps the xenheap, and memory never handed to the heap
allocator (Xen's image, boot-time allocations, firmware tables).  It is
built once the boot allocator has handed its memory to the heap, from
the full directmap minus the ranges the heap was given, plus the xenheap
allocations made meanwhile (NUMA node heap metadata).  The page-tables
then come from the heap, and no other CPU is up.  The sparse view has an
L3 for each of its L4 slots from the start, so its L4 entries never
change and root page-tables can take copies of them.  It is then kept in
step:

 - xenheap allocations are added, and taken out again before the pages
   are freed.  If that fails the pages stay mapped, and are leaked with
   a warning rather than reused (the same reasoning as in "xen/page_alloc:
   Add a path for xenheap when there is no direct map" of the directmap
   removal series);
 - memory handed to the heap later (boot modules, .init, memory
   hot-add) is taken out, and is not handed over if that fails;
 - a PV dom0's initrd, assigned to dom0 in place, is taken out;
 - page-tables from the boot allocator, which the sparse view maps with
   every other boot-time allocation, are taken out when idle_pg_table's
   hierarchy frees them, and are not freed if that fails, so that the
   sparse view never keeps a page the heap may hand out;
 - changes to the full directmap through map_pages_to_xen() and
   modify_xen_mappings() are mirrored.  After boot, map_pages_to_xen()
   maps into the directmap only memory Xen keeps (stack guard and shadow
   stack pages, firmware tables) or hot-added memory about to be handed
   to the heap, which the second rule then takes out again.

Common code gains four hooks for this, arch_heap_pages_init(),
arch_xenheap_needs_scrub() and arch_xenheap_pages_{alloc,free}(), no-ops
unless CONFIG_SPARSE_DIRECTMAP is selected.  A failed
arch_xenheap_pages_alloc() may have mapped part of the pages:
alloc_xenheap_pages() then takes them out through
arch_xenheap_pages_free(), as a free would, before freeing them, and
leaks them if that fails too.  xen_mapping_lookup(), built only with
CONFIG_SPARSE_DIRECTMAP, reads an entry of either hierarchy under the
L3 table lock.

alloc_xenheap_pages() accepts pages still awaiting scrubbing
(MEMF_no_scrub).  That is harmless while the directmap maps all memory
anyway, but the sparse view would make such a page's contents, a dying
domain's data for instance, reachable from every context on it until
overwritten.  With a sparse view configured, arch_xenheap_needs_scrub()
makes xenheap allocations scrub those pages first, as domain heap
allocations do; clean pages need nothing extra.

arch_mfns_in_directmap() says whether an MFN range is reachable through
the directmap; it now answers for every context, so false once a sparse
view is configured.  Its one caller, init_node_heap(), then keeps a NUMA
node's heap metadata in the xenheap instead of carving it out of the
node's memory through the directmap: carved out of a range handed to the
heap, it would be left out of the sparse view, which every heap
allocation needs it in.

The sparse view is built only when some guest type is to use it, which
nothing can request yet: this patch changes nothing on its own.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/Kconfig                        |   1 +
 xen/arch/x86/Makefile                       |   1 +
 xen/arch/x86/include/asm/mm.h               |  17 +-
 xen/arch/x86/include/asm/sparse-directmap.h |  63 +++
 xen/arch/x86/mm.c                           | 128 +++++-
 xen/arch/x86/pv/dom0_build.c                |   4 +
 xen/arch/x86/setup.c                        |   4 +
 xen/arch/x86/sparse-directmap.c             | 424 ++++++++++++++++++++
 xen/common/Kconfig                          |  18 +
 xen/common/page_alloc.c                     |  41 +-
 xen/include/xen/mm.h                        |  35 ++
 11 files changed, 722 insertions(+), 14 deletions(-)
 create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
 create mode 100644 xen/arch/x86/sparse-directmap.c

diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index e5535ac484..cd72102ccc 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -31,6 +31,7 @@ config X86
 	select HAS_SCHED_GRANULARITY
 	select HAS_SHARED_INFO
 	imply HAS_SOFT_RESET
+	select HAS_SPARSE_DIRECTMAP
 	select HAS_UBSAN
 	select HAS_VMAP
 	select HAS_VPCI if HVM
diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index 8a205c3d27..22b3c8bf6b 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -63,6 +63,7 @@ obj-y += shutdown.o
 obj-y += smp.o
 obj-y += smpboot.o
 obj-y += spec_ctrl.o
+obj-$(CONFIG_SPARSE_DIRECTMAP) += sparse-directmap.o
 obj-y += srat.o
 obj-y += string.o
 obj-$(CONFIG_SYSCTL) += sysctl.o
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 87131dd4bf..0d1e319c09 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -7,6 +7,7 @@
 #include <xen/rwlock.h>
 #include <asm/io.h>
 #include <asm/page.h>
+#include <asm/sparse-directmap.h>
 #include <asm/uaccess.h>
 
 /*
@@ -595,6 +596,15 @@ mfn_t alloc_xen_pagetable(void);
 void free_xen_pagetable(mfn_t mfn);
 void *alloc_mapped_pagetable(mfn_t *pmfn);
 
+int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
+                 unsigned long nr_mfns, pte_attr_t flags);
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
+                       pte_attr_t nf);
+#ifdef CONFIG_SPARSE_DIRECTMAP
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+                         pte_attr_t *flags, unsigned long *nr);
+#endif
+
 int __sync_local_execstate(void);
 
 /* Arch-specific portion of memory_op hypercall. */
@@ -655,12 +665,17 @@ void write_32bit_pse_identmap(uint32_t *l2);
 
 /*
  * x86 maps part of physical memory via the directmap region.
- * Return whether the range of MFN falls in the directmap region.
+ * Return whether the range of MFN falls in the directmap region, as seen from
+ * every context: with a sparse directmap view configured (see
+ * asm/sparse-directmap.h), no range is, except through a xenheap allocation.
  */
 static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
 {
     unsigned long eva = min(DIRECTMAP_VIRT_END, HYPERVISOR_VIRT_END);
 
+    if ( sparse_dmap_configured() )
+        return false;
+
     return (mfn + nr) <= (virt_to_mfn(eva - 1) + 1);
 }
 
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
new file mode 100644
index 0000000000..48c1f3c951
--- /dev/null
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef X86_SPARSE_DIRECTMAP_H
+#define X86_SPARSE_DIRECTMAP_H
+
+#include <xen/mm-frame.h>
+#include <xen/mm-types.h>
+#include <asm/page.h>
+
+#ifdef CONFIG_SPARSE_DIRECTMAP
+
+/*
+ * The part of the directmap the sparse view covers: the part PV guests can
+ * see, which is where everything Xen reaches through the directmap lives.
+ * SPARSE_DMAP_END_MFN is the first MFN past it.
+ */
+#define SPARSE_DMAP_START   DIRECTMAP_VIRT_START
+#define SPARSE_DMAP_END     HYPERVISOR_VIRT_END
+#define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
+
+extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
+
+/* Root of the sparse view, once built (NULL before, or if not configured). */
+extern l4_pgentry_t *sparse_dmap_root;
+
+static inline bool sparse_dmap_configured(void)
+{
+    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
+}
+
+void sparse_dmap_init(void);
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+                           pte_attr_t flags);
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+                              pte_attr_t nf);
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr);
+int sparse_dmap_drop_pagetable(mfn_t mfn);
+
+#else /* !CONFIG_SPARSE_DIRECTMAP */
+
+static inline bool sparse_dmap_configured(void) { return false; }
+static inline void sparse_dmap_init(void) {}
+static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
+                                         unsigned long nr, pte_attr_t flags)
+{
+    return 0;
+}
+static inline int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+                                            pte_attr_t nf)
+{
+    return 0;
+}
+static inline int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+    return 0;
+}
+static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+    return 0;
+}
+
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+#endif /* X86_SPARSE_DIRECTMAP_H */
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 94b96c3d9b..4a1bd9d146 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -127,6 +127,7 @@
 #include <asm/setup.h>
 #include <asm/shadow.h>
 #include <asm/shared.h>
+#include <asm/sparse-directmap.h>
 #include <asm/trampoline.h>
 #include <asm/traps.h>
 #include <asm/x86_emulate.h>
@@ -5394,11 +5395,104 @@ mfn_t xen_map_to_mfn(unsigned long va)
     return ret;
 }
 
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/*
+ * Look @va up in the hierarchy rooted at @root, without allocating anything.
+ * Returns the MFN mapped at @va, or INVALID_MFN, with the mapping's flags in
+ * L1 form in *flags, and in *nr the number of pages from @va to the end of
+ * the entry that maps @va (or that finds it unmapped).
+ */
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+                         pte_attr_t *flags, unsigned long *nr)
+{
+    bool locking = system_state > SYS_STATE_boot;
+    l4_pgentry_t l4e = root[l4_table_offset(va)];
+    const l3_pgentry_t *pl3e = NULL;
+    const l2_pgentry_t *pl2e = NULL;
+    const l1_pgentry_t *pl1e = NULL;
+    struct page_info *l3page;
+    unsigned int shift = L4_PAGETABLE_SHIFT;
+    mfn_t ret = INVALID_MFN;
+
+    L3T_INIT(l3page);
+    *flags = 0;
+
+    if ( !(l4e_get_flags(l4e) & _PAGE_PRESENT) )
+        goto out;
+
+    pl3e = map_l3t_from_l4e(l4e) + l3_table_offset(va);
+    l3page = l4e_get_page(l4e);
+    L3T_LOCK(l3page);
+
+    shift = L3_PAGETABLE_SHIFT;
+    if ( !(l3e_get_flags(*pl3e) & _PAGE_PRESENT) )
+        goto out;
+    if ( l3e_get_flags(*pl3e) & _PAGE_PSE )
+    {
+        /* The superpage PAT bit sits among the address bits. */
+        *flags = lNf_to_l1f(l3e_get_flags(*pl3e)) |
+                 (l3e_get_intpte(*pl3e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+        ret = _mfn((l3e_get_pfn(*pl3e) & ~((1UL << (2 * PAGETABLE_ORDER)) - 1)) +
+                   (PFN_DOWN(va) & ((1UL << (2 * PAGETABLE_ORDER)) - 1)));
+        goto out;
+    }
+
+    pl2e = map_l2t_from_l3e(*pl3e) + l2_table_offset(va);
+    shift = L2_PAGETABLE_SHIFT;
+    if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
+        goto out;
+    if ( l2e_get_flags(*pl2e) & _PAGE_PSE )
+    {
+        *flags = lNf_to_l1f(l2e_get_flags(*pl2e)) |
+                 (l2e_get_intpte(*pl2e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+        ret = _mfn((l2e_get_pfn(*pl2e) & ~((1UL << PAGETABLE_ORDER) - 1)) +
+                   l1_table_offset(va));
+        goto out;
+    }
+
+    pl1e = map_l1t_from_l2e(*pl2e) + l1_table_offset(va);
+    shift = PAGE_SHIFT;
+    if ( l1e_get_flags(*pl1e) & _PAGE_PRESENT )
+    {
+        *flags = l1e_get_flags(*pl1e);
+        ret = l1e_get_mfn(*pl1e);
+    }
+
+ out:
+    L3T_UNLOCK(l3page);
+    unmap_domain_page(pl1e);
+    unmap_domain_page(pl2e);
+    unmap_domain_page(pl3e);
+    *nr = ((((va >> shift) + 1) << shift) - va) >> PAGE_SHIFT;
+    return ret;
+}
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+/*
+ * Free a page-table page that the hierarchy rooted at @root has stopped
+ * using.  idle_pg_table's hierarchy has pages from the boot allocator,
+ * which the sparse directmap view maps along with every other boot-time
+ * allocation: such a page leaves the sparse view before it reaches the
+ * heap, and is kept if it cannot.  That takes the sparse view's L3 lock
+ * with an L3 lock of idle_pg_table's hierarchy held, the only order in
+ * which the two nest.  The sparse view's own page-tables come from the
+ * heap, which it does not map, and are freed directly, as the caller may
+ * hold the sparse view's L3 lock; so are tables freed straight after being
+ * allocated, having lost a race to be installed.
+ */
+static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
+{
+    if ( root == idle_pg_table && sparse_dmap_drop_pagetable(mfn) )
+        return;
+
+    free_xen_pagetable(mfn);
+}
+
 /*
  * map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
  * another hierarchy sharing its virtual addresses.
  */
-static int map_pages_in(
+int map_pages_in(
     l4_pgentry_t *root,
     unsigned long virt,
     mfn_t mfn,
@@ -5503,10 +5597,10 @@ static int map_pages_in(
                         ol2e = l2t[i];
                         if ( (l2e_get_flags(ol2e) & _PAGE_PRESENT) &&
                              !(l2e_get_flags(ol2e) & _PAGE_PSE) )
-                            free_xen_pagetable(l2e_get_mfn(ol2e));
+                            free_pagetable_in(root, l2e_get_mfn(ol2e));
                     }
                     unmap_domain_page(l2t);
-                    free_xen_pagetable(l3e_get_mfn(ol3e));
+                    free_pagetable_in(root, l3e_get_mfn(ol3e));
                 }
             }
 
@@ -5604,7 +5698,7 @@ static int map_pages_in(
                         flush_flags(l1e_get_flags(l1t[i]));
                     flush_area(virt, flush_flags);
                     unmap_domain_page(l1t);
-                    free_xen_pagetable(l2e_get_mfn(ol2e));
+                    free_pagetable_in(root, l2e_get_mfn(ol2e));
                 }
             }
 
@@ -5736,7 +5830,7 @@ static int map_pages_in(
                     flush_area(virt - PAGE_SIZE,
                                FLUSH_TLB_GLOBAL |
                                FLUSH_ORDER(PAGETABLE_ORDER));
-                    free_xen_pagetable(l2e_get_mfn(ol2e));
+                    free_pagetable_in(root, l2e_get_mfn(ol2e));
                 }
                 else if ( locking )
                     spin_unlock(&map_pgdir_lock);
@@ -5783,7 +5877,7 @@ static int map_pages_in(
                 flush_area(virt - PAGE_SIZE,
                            FLUSH_TLB_GLOBAL |
                            FLUSH_ORDER(2*PAGETABLE_ORDER));
-                free_xen_pagetable(l3e_get_mfn(ol3e));
+                free_pagetable_in(root, l3e_get_mfn(ol3e));
             }
             else if ( locking )
                 spin_unlock(&map_pgdir_lock);
@@ -5810,7 +5904,12 @@ int map_pages_to_xen(
     unsigned long nr_mfns,
     pte_attr_t flags)
 {
-    return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+    int rc = map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+
+    if ( !rc )
+        rc = sparse_dmap_mirror_map(virt, mfn, nr_mfns, flags);
+
+    return rc;
 }
 
 int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
@@ -5833,8 +5932,8 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
  * modify_mappings_in() does so in the hierarchy rooted at @root, as
  * map_pages_in() does.
  */
-static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
-                              unsigned long e, pte_attr_t nf)
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+                       unsigned long e, pte_attr_t nf)
 {
     bool locking = system_state > SYS_STATE_boot;
     l3_pgentry_t *pl3e = NULL;
@@ -6044,7 +6143,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
                 if ( locking )
                     spin_unlock(&map_pgdir_lock);
                 flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
-                free_xen_pagetable(l1mfn);
+                free_pagetable_in(root, l1mfn);
             }
             else if ( locking )
                 spin_unlock(&map_pgdir_lock);
@@ -6088,7 +6187,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
                 if ( locking )
                     spin_unlock(&map_pgdir_lock);
                 flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
-                free_xen_pagetable(l2mfn);
+                free_pagetable_in(root, l2mfn);
             }
             else if ( locking )
                 spin_unlock(&map_pgdir_lock);
@@ -6114,7 +6213,12 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
 
 int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
 {
-    return modify_mappings_in(idle_pg_table, s, e, nf);
+    int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+
+    if ( !rc )
+        rc = sparse_dmap_mirror_modify(s, e, nf);
+
+    return rc;
 }
 
 int destroy_xen_mappings(unsigned long s, unsigned long e)
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index bf980eba8c..f0a1197e62 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -22,6 +22,7 @@
 #include <asm/pv/mm.h>
 #include <asm/setup.h>
 #include <asm/shadow.h>
+#include <asm/sparse-directmap.h>
 
 /* Allow ring-3 access in long mode as guest cannot use ring 1 ... */
 #define BASE_PROT (_PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_USER)
@@ -651,6 +652,9 @@ static int __init dom0_construct(const struct boot_domain *bd)
             while ( count-- )
                 if ( assign_pages(mfn_to_page(_mfn(mfn++)), 1, d, 0) )
                     BUG();
+            /* The initrd is dom0's now: out of the sparse directmap. */
+            if ( sparse_dmap_remove(_mfn(initrd_mfn), PFN_UP(initrd_len)) )
+                panic("Cannot unmap dom0's initrd\n");
             /*
              * We have mapped the initrd directly into dom0, and assigned the
              * pages. Tell the boot_module handling that we've freed it, so the
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index a636e3205c..e52388ad28 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -54,6 +54,7 @@
 #include <asm/spec_ctrl.h>
 #include <asm/stubs.h>
 #include <asm/tboot.h>
+#include <asm/sparse-directmap.h>
 #include <asm/trampoline.h>
 #include <asm/traps.h>
 
@@ -1938,6 +1939,9 @@ void asmlinkage __init noreturn __start_xen(void)
 
     system_state = SYS_STATE_boot;
 
+    /* The heap now has all memory: page-tables come from it from here on. */
+    sparse_dmap_init();
+
     bsp_stack = cpu_alloc_stack(0); /* Needs to know IDT vs FRED */
     if ( !bsp_stack )
         panic("No memory for BSP stack\n");
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
new file mode 100644
index 0000000000..1eb092c3ea
--- /dev/null
+++ b/xen/arch/x86/sparse-directmap.c
@@ -0,0 +1,424 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * A sparse view of the directmap.
+ *
+ * Xen maps all of RAM at DIRECTMAP_VIRT_START, and every root page-table
+ * carries those mappings.  The sparse view is a second set of page-tables
+ * for the same virtual addresses (the part of the directmap PV guests can
+ * see, which is where everything Xen reaches through the directmap lives),
+ * mapping only what Xen needs to reach through the directmap at all times.
+ *
+ * Memory in the full directmap can be classified into three groups:
+ *
+ * - Memory never handed to the heap.  This includes Xen's image,
+ *   boot-time allocations, and firmware tables.  Mapped in the sparse
+ *   view.
+ * - Memory handed to the heap, and not allocated to the xenheap.  Not
+ *   mapped in the sparse view: guest memory will come from here.
+ * - Memory handed to the heap, then allocated to the xenheap.  Mapped
+ *   in the sparse view.
+ *
+ * Guest contexts selected on the command line run on root page-tables
+ * carrying the sparse view's L3s in the directmap slots.
+ *
+ * Conceptually, the sparse view starts as a copy of the full directmap
+ * (idle_pg_table's hierarchy), and then:
+ * - Removes memory when it is handed to the heap.
+ * - Adds pages back when they are allocated as xenheap pages, and removes
+ *   them again when they are freed.
+ * - Mirrors changes to the full directmap (map_pages_to_xen(),
+ *   modify_xen_mappings()).
+ * - Removes memory handed to a domain without passing through the heap
+ *   (a PV dom0's initrd, used in place), and page-tables the boot
+ *   allocator provided when idle_pg_table's hierarchy frees them.
+ *
+ * Every virtual address mapping in the sparse view should be either
+ * identical to the corresponding full view mapping, or empty.
+ *
+ * The sparse view is built once the boot allocator has handed its
+ * memory to the heap (sparse_dmap_init()).  Until then, the hooks record
+ * the ranges handed to the heap and the xenheap allocations made from
+ * them, which sparse_dmap_init() leaves out and puts back.
+ */
+
+#include <xen/domain_page.h>
+#include <xen/init.h>
+#include <xen/lib.h>
+#include <xen/mm.h>
+#include <xen/numa.h>
+#include <xen/sort.h>
+
+#include <asm/page.h>
+#include <asm/sparse-directmap.h>
+
+bool __ro_after_init opt_sparse_dmap_pv;
+bool __ro_after_init opt_sparse_dmap_hvm;
+
+/* Root of the sparse view: only ever walked and copied from, never loaded. */
+static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
+l4_pgentry_t *__ro_after_init sparse_dmap_root;
+
+/*
+ * Before the sparse view is built: the MFN ranges handed to the heap, which
+ * it leaves out, and the xenheap allocations made meanwhile (heap metadata of
+ * NUMA nodes), which it puts back.
+ */
+struct mfn_range {
+    unsigned long s, e;
+};
+static struct mfn_range early_heap[256];
+static unsigned int nr_early_heap;
+static struct mfn_range early_xenheap[2 * MAX_NUMNODES];
+static unsigned int nr_early_xenheap;
+static bool early_overflow;
+
+/*
+ * Clip [*mfn, *mfn + *nr) to what the sparse view covers; false if nothing is
+ * left.
+ */
+static bool clip_to_sparse_view(unsigned long *mfn, unsigned long *nr)
+{
+    unsigned long end = SPARSE_DMAP_END_MFN;
+
+    if ( *mfn >= end )
+        return false;
+    if ( *nr > end - *mfn )
+        *nr = end - *mfn;
+
+    return *nr;
+}
+
+static void early_range_add(struct mfn_range *r, unsigned int *nr,
+                            unsigned int max, unsigned long s, unsigned long e)
+{
+    if ( *nr && r[*nr - 1].e == s )
+        r[*nr - 1].e = e;
+    else if ( *nr < max )
+        r[(*nr)++] = (struct mfn_range){ s, e };
+    else
+        early_overflow = true;
+}
+
+static void early_range_del(struct mfn_range *r, unsigned int *nr,
+                            unsigned long s, unsigned long e)
+{
+    unsigned int i;
+
+    for ( i = 0; i < *nr; i++ )
+        if ( r[i].s == s && r[i].e == e )
+        {
+            r[i] = r[--*nr];
+            return;
+        }
+}
+
+/*
+ * Copy the full directmap's mappings of MFNs [mfn, mfn + nr) into the sparse
+ * view.
+ */
+static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
+                              unsigned long nr)
+{
+    unsigned long va = (unsigned long)mfn_to_virt(mfn);
+    unsigned long end = va + (nr << PAGE_SHIFT);
+
+    while ( va < end )
+    {
+        pte_attr_t flags;
+        unsigned long n;
+        mfn_t m = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+        int rc;
+
+        n = min(n, (end - va) >> PAGE_SHIFT);
+        if ( !mfn_eq(m, INVALID_MFN) &&
+             (rc = map_pages_in(root, va, m, n, flags)) )
+            return rc;
+        va += n << PAGE_SHIFT;
+    }
+
+    return 0;
+}
+
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+{
+    unsigned long va = (unsigned long)mfn_to_virt(mfn);
+
+    return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
+                              _PAGE_NONE);
+}
+
+/* Memory is about to be handed to the heap allocator. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+    unsigned long s = mfn_x(mfn);
+
+    if ( !sparse_dmap_configured() || !clip_to_sparse_view(&s, &nr) )
+        return true;
+
+    if ( !sparse_dmap_root )
+    {
+        early_range_add(early_heap, &nr_early_heap, ARRAY_SIZE(early_heap),
+                        s, s + nr);
+        return true;
+    }
+
+    if ( remove_from_sparse_view(s, nr) )
+    {
+        printk(XENLOG_ERR
+               "sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
+               s, s + nr - 1);
+        return false;
+    }
+
+    return true;
+}
+
+/*
+ * Xenheap pages enter the sparse view, which guest contexts run on: they
+ * must not bring along what a previous owner left in them, such as a dying
+ * domain's data still awaiting the scrubber.  Pages needing scrubbing are
+ * scrubbed as they are allocated then; clean ones need nothing.
+ */
+bool arch_xenheap_needs_scrub(void)
+{
+    return sparse_dmap_configured();
+}
+
+/* Pages have been allocated from the heap as xenheap pages. */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+    unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+    if ( !sparse_dmap_configured() )
+        return 0;
+
+    /* Xenheap pages must be reachable through the sparse view. */
+    BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+    if ( !sparse_dmap_root )
+    {
+        early_range_add(early_xenheap, &nr_early_xenheap,
+                        ARRAY_SIZE(early_xenheap), s, s + nr);
+        return 0;
+    }
+
+    return add_to_sparse_view(sparse_dmap_root, s, nr);
+}
+
+/*
+ * Xenheap pages are about to be freed.  A failure means the pages may still
+ * be reachable through the sparse view, and must not be reused.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+    unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+    if ( !sparse_dmap_configured() )
+        return 0;
+
+    BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+    if ( !sparse_dmap_root )
+    {
+        early_range_del(early_xenheap, &nr_early_xenheap, s, s + nr);
+        return 0;
+    }
+
+    return remove_from_sparse_view(s, nr);
+}
+
+/* Memory leaves Xen's hands without passing through the heap. */
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+    return arch_heap_pages_init(mfn, nr) ? 0 : -ENOMEM;
+}
+
+/*
+ * A page-table page of idle_pg_table's hierarchy is about to be freed.  One
+ * the boot allocator provided is in the sparse view, as every boot-time
+ * allocation is: take it out before the heap may hand it on.  One allocated
+ * since came from the heap and is not in the sparse view, which a lookup
+ * finds without changing, or flushing, anything.  Non-zero: the page may
+ * still be mapped, and must not be freed.
+ */
+int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+    unsigned long s = mfn_x(mfn), nr = 1, n;
+    pte_attr_t flags;
+    int rc;
+
+    if ( !sparse_dmap_root || !clip_to_sparse_view(&s, &nr) ||
+         mfn_eq(xen_mapping_lookup(sparse_dmap_root,
+                                   (unsigned long)mfn_to_virt(s), &flags, &n),
+                INVALID_MFN) )
+        return 0;
+
+    rc = remove_from_sparse_view(s, 1);
+    if ( rc )
+        printk(XENLOG_WARNING
+               "Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
+               mfn_x(mfn));
+
+    return rc;
+}
+
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+                           pte_attr_t flags)
+{
+    if ( !sparse_dmap_root || virt < SPARSE_DMAP_START ||
+         virt >= SPARSE_DMAP_END )
+        return 0;
+
+    nr = min(nr, (SPARSE_DMAP_END - virt) >> PAGE_SHIFT);
+
+    return map_pages_in(sparse_dmap_root, virt, mfn, nr, flags);
+}
+
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+                              pte_attr_t nf)
+{
+    unsigned long va;
+
+    if ( !sparse_dmap_root || e <= SPARSE_DMAP_START ||
+         s >= SPARSE_DMAP_END )
+        return 0;
+
+    s = max(s, SPARSE_DMAP_START);
+    e = min(e, SPARSE_DMAP_END);
+
+    if ( !(nf & _PAGE_PRESENT) )
+        return modify_mappings_in(sparse_dmap_root, s, e, nf);
+
+    /*
+     * A permission change applies where the sparse view maps anything: it
+     * never creates mappings, and the sparse view need not map all of [s, e).
+     */
+    for ( va = s; va < e; )
+    {
+        pte_attr_t flags;
+        unsigned long n;
+        mfn_t m = xen_mapping_lookup(sparse_dmap_root, va, &flags, &n);
+        int rc;
+
+        n = min(n, (e - va) >> PAGE_SHIFT);
+        if ( !mfn_eq(m, INVALID_MFN) &&
+             (rc = modify_mappings_in(sparse_dmap_root, va,
+                                      va + (n << PAGE_SHIFT), nf)) )
+            return rc;
+        va += n << PAGE_SHIFT;
+    }
+
+    return 0;
+}
+
+static int __init cf_check cmp_mfn_range(const void *a, const void *b)
+{
+    const struct mfn_range *l = a, *r = b;
+
+    return l->s < r->s ? -1 : l->s > r->s;
+}
+
+static void __init cf_check swap_mfn_range(void *a, void *b)
+{
+    SWAP(*(struct mfn_range *)a, *(struct mfn_range *)b);
+}
+
+/*
+ * Build the sparse view: the full directmap, minus the ranges handed to the
+ * heap so far, plus the xenheap allocations made from them.  Called once the
+ * heap has taken over from the boot allocator, so that the page-tables come
+ * from the heap, and before any other CPU is up.
+ */
+void __init sparse_dmap_init(void)
+{
+    unsigned long va, mapped = 0;
+    unsigned int i, j;
+
+    if ( !sparse_dmap_configured() )
+        return;
+
+    if ( early_overflow )
+    {
+        printk(XENLOG_WARNING
+               "Sparse directmap: too many early heap ranges, disabled\n");
+        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = false;
+        return;
+    }
+
+    /* An L3 for every slot, so that the L4 entries never change. */
+    for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END;
+          va += 1UL << L4_PAGETABLE_SHIFT )
+    {
+        mfn_t l3mfn;
+        void *l3t = alloc_mapped_pagetable(&l3mfn);
+
+        if ( !l3t )
+            panic("Sparse directmap: out of memory\n");
+        unmap_domain_page(l3t);
+        sparse_l4[l4_table_offset(va)] =
+            l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
+    }
+
+    sort(early_heap, nr_early_heap, sizeof(*early_heap), cmp_mfn_range,
+         swap_mfn_range);
+
+    /*
+     * The full directmap, leaf by leaf, minus the ranges added to the
+     * heap.  Leaves come in address order, which is MFN order in the
+     * directmap, so the cursor into the sorted (and disjoint) heap
+     * ranges only moves forward.
+     */
+    for ( va = SPARSE_DMAP_START, j = 0; va < SPARSE_DMAP_END; )
+    {
+        pte_attr_t flags;
+        unsigned long n, s, e;
+        mfn_t m = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+
+        n = min(n, (SPARSE_DMAP_END - va) >> PAGE_SHIFT);
+        if ( mfn_eq(m, INVALID_MFN) )
+        {
+            va += n << PAGE_SHIFT;
+            continue;
+        }
+
+        for ( s = mfn_x(m), e = s + n; s < e; )
+        {
+            unsigned long upto = e;
+
+            while ( j < nr_early_heap && early_heap[j].e <= s )
+                j++;
+            if ( j < nr_early_heap && early_heap[j].s <= s )
+            {
+                s = min(e, early_heap[j].e);
+                continue;
+            }
+            if ( j < nr_early_heap )
+                upto = min(e, early_heap[j].s);
+
+            if ( map_pages_in(sparse_l4, va + ((s - mfn_x(m)) << PAGE_SHIFT),
+                              _mfn(s), upto - s, flags) )
+                panic("Sparse directmap: out of memory\n");
+            mapped += upto - s;
+            s = upto;
+        }
+
+        va += n << PAGE_SHIFT;
+    }
+
+    /*
+     * Now add back in ranges from the heap which were allocated to
+     * the xenheap.
+     */
+    for ( i = 0; i < nr_early_xenheap; i++ )
+    {
+        unsigned long nr = early_xenheap[i].e - early_xenheap[i].s;
+
+        if ( add_to_sparse_view(sparse_l4, early_xenheap[i].s, nr) )
+            panic("Sparse directmap: out of memory\n");
+        mapped += nr;
+    }
+
+    sparse_dmap_root = sparse_l4;
+
+    printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
+}
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 5b289e444f..8f2f605e45 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -167,6 +167,9 @@ config HAS_STATIC_MEMORY
 config HAS_SOFT_RESET
 	bool
 
+config HAS_SPARSE_DIRECTMAP
+	bool
+
 config HAS_STACK_PROTECTOR
 	bool
 
@@ -299,6 +302,21 @@ config SPECULATIVE_HARDEN_LOCK
 	  This option is disabled by default at run time, and needs to be
 	  enabled on the command line.
 
+
+config SPARSE_DIRECTMAP
+	bool "Sparse directmap view"
+	depends on HAS_SPARSE_DIRECTMAP
+	help
+	  Build support for a sparse view of the directmap, alongside the
+	  full one: it maps only the memory Xen needs to reach at all times
+	  (the Xen heap, Xen's own image, firmware tables), while guest
+	  memory is reached through transient mappings.  Guest contexts
+	  selected with the `asi=` command line option run on the sparse
+	  directmap view, narrowing the memory reachable from Xen's address
+	  space while they run, and hence the memory disclosure surface of
+	  speculative side channels.
+
+	  If unsure, say N.
 endmenu
 
 menu "Other hardening"
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index fe150eccd7..69e8e982b6 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1971,6 +1971,9 @@ static void init_heap_pages(
     unsigned long i;
     bool need_scrub = scrub_debug;
 
+    if ( !arch_heap_pages_init(page_to_mfn(pg), nr_pages) )
+        return;
+
     /*
      * Keep MFN 0 away from the buddy allocator to avoid crossing zone
      * boundary when merging two buddies.
@@ -2540,10 +2543,33 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)
     if ( !(memflags >> _MEMF_bits) )
         memflags |= MEMF_bits(xenheap_bits);
 
-    pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub);
+    /*
+     * Pages still awaiting scrubbing may do as xenheap pages: the directmap
+     * maps them anyway.  Not so if an arch's sparse directmap view makes them
+     * reachable where the rest of the heap's memory is not.
+     */
+    if ( !arch_xenheap_needs_scrub() )
+        memflags |= MEMF_no_scrub;
+
+    pg = alloc_domheap_pages(NULL, order, memflags);
     if ( unlikely(pg == NULL) )
         return NULL;
 
+    if ( arch_xenheap_pages_alloc(page_to_mfn(pg), order) )
+    {
+        /*
+         * The hook may have mapped part of the pages before failing: take
+         * them out again as a free would, and leak them if that fails too.
+         */
+        if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+            printk(XENLOG_WARNING
+                   "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+                   mfn_x(page_to_mfn(pg)), 1U << order);
+        else
+            free_domheap_pages(pg, order);
+        return NULL;
+    }
+
     for ( i = 0; i < (1u << order); i++ )
         pg[i].count_info |= PGC_xen_heap;
 
@@ -2562,6 +2588,19 @@ void free_xenheap_pages(void *v, unsigned int order)
 
     pg = virt_to_page(v);
 
+    /*
+     * If the pages cannot be taken out of an arch's sparse directmap view,
+     * they may remain reachable through it: leak them rather than hand them
+     * out.
+     */
+    if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+    {
+        printk(XENLOG_WARNING
+               "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+               mfn_x(page_to_mfn(pg)), 1U << order);
+        return;
+    }
+
     for ( i = 0; i < (1u << order); i++ )
         pg[i].count_info &= ~PGC_xen_heap;
 
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 153c57f5dd..2bcc45cf58 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -94,6 +94,41 @@ bool scrub_free_pages(void);
 #define alloc_xenheap_page() (alloc_xenheap_pages(0,0))
 #define free_xenheap_page(v) (free_xenheap_pages(v,0))
 
+/*
+ * Hooks for an architecture keeping a second, sparse view of its directmap,
+ * which maps the xenheap but not the rest of the heap's memory.
+ */
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/* Memory is about to be handed to the heap.  False: don't, keep it out. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr);
+/* Xenheap pages are to be allocated.  True: scrub those needing it first. */
+bool arch_xenheap_needs_scrub(void);
+/*
+ * Xenheap pages were allocated.  Non-zero: failed, possibly part-way; the
+ * pages go through arch_xenheap_pages_free() before being freed again.
+ */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
+/* Xenheap pages are to be freed.  Non-zero: failed, don't reuse them. */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+#else
+static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+    return true;
+}
+static inline bool arch_xenheap_needs_scrub(void)
+{
+    return false;
+}
+static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+    return 0;
+}
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+    return 0;
+}
+#endif
+
 /* Free an allocation, and zero the pointer to it. */
 #define FREE_XENHEAP_PAGES(p, o) do { \
     void *_ptr_ = (p);                \
-- 
2.55.0

Re: [PATCH v3 13/18] x86/mm: build and maintain a sparse view of the directmap

[George Dunlap]

On Wed, Oct 7, 2026 at 12:00 PM George Dunlap <gwd@xenproject.org> wrote:
> The directmap maps all of RAM, and every root page-table carries it, so
> while any guest context runs, all guest memory is reachable from Xen's
> address space.  For ASI, guest contexts are to run on a view of the
> directmap which maps only what Xen must reach through it at all times,
> leaving guest memory to transient mappings (map_domain_page()).

Note that a deeper review has found a bunch of random corner-cases,
mostly to do with allocation failures when modifying the sparse view's
pagetables.  I'm going through them; my current plan is in most cases
to add fixes after this series, to make them easier to review, unless
they're obviously core to the functionality of this patch as it is.

 -George

Re: [PATCH v3 13/18] x86/mm: build and maintain a sparse view of the directmap

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:46AM +0100, George Dunlap wrote:
> The directmap maps all of RAM, and every root page-table carries it, so
> while any guest context runs, all guest memory is reachable from Xen's
> address space.  For ASI, guest contexts are to run on a view of the
> directmap which maps only what Xen must reach through it at all times,
> leaving guest memory to transient mappings (map_domain_page()).
> 
> Rather than remove the directmap for everyone, keep the full directmap
> as it is, for the idle domain and for guest contexts not using ASI, and
> build a second, sparse view: page-tables for the same virtual addresses
> (the part of the directmap PV guests can see, where everything Xen
> reaches through the directmap lives), mapping a subset of what the full
> directmap maps, each entry a copy of the full directmap's.  Boot, dom0
> construction and the idle domain keep the full directmap, so nothing in
> them needs converting.
> 
> The sparse view maps the xenheap, and memory never handed to the heap
> allocator (Xen's image, boot-time allocations, firmware tables).  It is
> built once the boot allocator has handed its memory to the heap, from
> the full directmap minus the ranges the heap was given, plus the xenheap
> allocations made meanwhile (NUMA node heap metadata).  The page-tables
> then come from the heap, and no other CPU is up.  The sparse view has an
> L3 for each of its L4 slots from the start, so its L4 entries never
> change and root page-tables can take copies of them.  It is then kept in
> step:
> 
>  - xenheap allocations are added, and taken out again before the pages
>    are freed.  If that fails the pages stay mapped, and are leaked with
>    a warning rather than reused (the same reasoning as in "xen/page_alloc:
>    Add a path for xenheap when there is no direct map" of the directmap
>    removal series);

I would probably remove this last sentence, it doesn't add any
meaningful information.

>  - memory handed to the heap later (boot modules, .init, memory
>    hot-add) is taken out, and is not handed over if that fails;
>  - a PV dom0's initrd, assigned to dom0 in place, is taken out;

I assume dom0 kernel memory is freed, and it doesn't need the same
kind of special handling the initrd needs due to it being assigned in
place.

>  - page-tables from the boot allocator, which the sparse view maps with
>    every other boot-time allocation, are taken out when idle_pg_table's
>    hierarchy frees them, and are not freed if that fails, so that the
>    sparse view never keeps a page the heap may hand out;
>  - changes to the full directmap through map_pages_to_xen() and
>    modify_xen_mappings() are mirrored.  After boot, map_pages_to_xen()
>    maps into the directmap only memory Xen keeps (stack guard and shadow
>    stack pages, firmware tables) or hot-added memory about to be handed
>    to the heap, which the second rule then takes out again.
> 
> Common code gains four hooks for this, arch_heap_pages_init(),
> arch_xenheap_needs_scrub() and arch_xenheap_pages_{alloc,free}(), no-ops
> unless CONFIG_SPARSE_DIRECTMAP is selected.  A failed
> arch_xenheap_pages_alloc() may have mapped part of the pages:
> alloc_xenheap_pages() then takes them out through
> arch_xenheap_pages_free(), as a free would, before freeing them, and
> leaks them if that fails too.  xen_mapping_lookup(), built only with
> CONFIG_SPARSE_DIRECTMAP, reads an entry of either hierarchy under the
> L3 table lock.
> 
> alloc_xenheap_pages() accepts pages still awaiting scrubbing
> (MEMF_no_scrub).  That is harmless while the directmap maps all memory
> anyway, but the sparse view would make such a page's contents, a dying
> domain's data for instance, reachable from every context on it until
> overwritten.  With a sparse view configured, arch_xenheap_needs_scrub()
> makes xenheap allocations scrub those pages first, as domain heap
> allocations do; clean pages need nothing extra.

There's another possible issue here I think: even when requesting
scrubbed pages the allocator can decide to scrub them in-place,
mapping them using map_domain_page().  Could this also cause concern,
as Xen is mapping possibly sensitive data in a context that's not
supposed to access it, even if just for the scrubbing.

> arch_mfns_in_directmap() says whether an MFN range is reachable through
> the directmap; it now answers for every context, so false once a sparse
> view is configured.  Its one caller, init_node_heap(), then keeps a NUMA
> node's heap metadata in the xenheap instead of carving it out of the
> node's memory through the directmap: carved out of a range handed to the
> heap, it would be left out of the sparse view, which every heap
> allocation needs it in.
> 
> The sparse view is built only when some guest type is to use it, which
> nothing can request yet: this patch changes nothing on its own.

Is it worth mentioning that modify_xen_mappings_lite() doesn't need
adjustment because it only deals with virtual addresses in the
XEN_VIRT_{START,END} range, and those are cloned in the sparse
directmap view?

> 
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version.
> ---
>  xen/arch/x86/Kconfig                        |   1 +
>  xen/arch/x86/Makefile                       |   1 +
>  xen/arch/x86/include/asm/mm.h               |  17 +-
>  xen/arch/x86/include/asm/sparse-directmap.h |  63 +++
>  xen/arch/x86/mm.c                           | 128 +++++-
>  xen/arch/x86/pv/dom0_build.c                |   4 +
>  xen/arch/x86/setup.c                        |   4 +
>  xen/arch/x86/sparse-directmap.c             | 424 ++++++++++++++++++++
>  xen/common/Kconfig                          |  18 +
>  xen/common/page_alloc.c                     |  41 +-
>  xen/include/xen/mm.h                        |  35 ++
>  11 files changed, 722 insertions(+), 14 deletions(-)
>  create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
>  create mode 100644 xen/arch/x86/sparse-directmap.c

I have to admit reviewing this 1K line diff is challenging.  I've
tried my best, but it's likely I've missed stuff.

> diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
> index e5535ac484..cd72102ccc 100644
> --- a/xen/arch/x86/Kconfig
> +++ b/xen/arch/x86/Kconfig
> @@ -31,6 +31,7 @@ config X86
>  	select HAS_SCHED_GRANULARITY
>  	select HAS_SHARED_INFO
>  	imply HAS_SOFT_RESET
> +	select HAS_SPARSE_DIRECTMAP
>  	select HAS_UBSAN
>  	select HAS_VMAP
>  	select HAS_VPCI if HVM
> diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
> index 8a205c3d27..22b3c8bf6b 100644
> --- a/xen/arch/x86/Makefile
> +++ b/xen/arch/x86/Makefile
> @@ -63,6 +63,7 @@ obj-y += shutdown.o
>  obj-y += smp.o
>  obj-y += smpboot.o
>  obj-y += spec_ctrl.o
> +obj-$(CONFIG_SPARSE_DIRECTMAP) += sparse-directmap.o
>  obj-y += srat.o
>  obj-y += string.o
>  obj-$(CONFIG_SYSCTL) += sysctl.o
> diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
> index 87131dd4bf..0d1e319c09 100644
> --- a/xen/arch/x86/include/asm/mm.h
> +++ b/xen/arch/x86/include/asm/mm.h
> @@ -7,6 +7,7 @@
>  #include <xen/rwlock.h>
>  #include <asm/io.h>
>  #include <asm/page.h>
> +#include <asm/sparse-directmap.h>
>  #include <asm/uaccess.h>
>  
>  /*
> @@ -595,6 +596,15 @@ mfn_t alloc_xen_pagetable(void);
>  void free_xen_pagetable(mfn_t mfn);
>  void *alloc_mapped_pagetable(mfn_t *pmfn);
>  
> +int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
> +                 unsigned long nr_mfns, pte_attr_t flags);
> +int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
> +                       pte_attr_t nf);
> +#ifdef CONFIG_SPARSE_DIRECTMAP
> +mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
> +                         pte_attr_t *flags, unsigned long *nr);
> +#endif
> +
>  int __sync_local_execstate(void);
>  
>  /* Arch-specific portion of memory_op hypercall. */
> @@ -655,12 +665,17 @@ void write_32bit_pse_identmap(uint32_t *l2);
>  
>  /*
>   * x86 maps part of physical memory via the directmap region.
> - * Return whether the range of MFN falls in the directmap region.
> + * Return whether the range of MFN falls in the directmap region, as seen from
> + * every context: with a sparse directmap view configured (see
> + * asm/sparse-directmap.h), no range is, except through a xenheap allocation.
>   */
>  static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
>  {
>      unsigned long eva = min(DIRECTMAP_VIRT_END, HYPERVISOR_VIRT_END);
>  
> +    if ( sparse_dmap_configured() )
> +        return false;
> +
>      return (mfn + nr) <= (virt_to_mfn(eva - 1) + 1);
>  }
>  
> diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
> new file mode 100644
> index 0000000000..48c1f3c951
> --- /dev/null
> +++ b/xen/arch/x86/include/asm/sparse-directmap.h
> @@ -0,0 +1,63 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +#ifndef X86_SPARSE_DIRECTMAP_H
> +#define X86_SPARSE_DIRECTMAP_H
> +
> +#include <xen/mm-frame.h>
> +#include <xen/mm-types.h>
> +#include <asm/page.h>
> +
> +#ifdef CONFIG_SPARSE_DIRECTMAP
> +
> +/*
> + * The part of the directmap the sparse view covers: the part PV guests can
> + * see, which is where everything Xen reaches through the directmap lives.
> + * SPARSE_DMAP_END_MFN is the first MFN past it.
> + */
> +#define SPARSE_DMAP_START   DIRECTMAP_VIRT_START
> +#define SPARSE_DMAP_END     HYPERVISOR_VIRT_END
> +#define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
> +
> +extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
> +
> +/* Root of the sparse view, once built (NULL before, or if not configured). */
> +extern l4_pgentry_t *sparse_dmap_root;
> +
> +static inline bool sparse_dmap_configured(void)
> +{
> +    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
> +}
> +
> +void sparse_dmap_init(void);
> +int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
> +                           pte_attr_t flags);
> +int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
> +                              pte_attr_t nf);
> +int sparse_dmap_remove(mfn_t mfn, unsigned long nr);
> +int sparse_dmap_drop_pagetable(mfn_t mfn);
> +
> +#else /* !CONFIG_SPARSE_DIRECTMAP */
> +
> +static inline bool sparse_dmap_configured(void) { return false; }
> +static inline void sparse_dmap_init(void) {}
> +static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
> +                                         unsigned long nr, pte_attr_t flags)
> +{
> +    return 0;
> +}
> +static inline int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
> +                                            pte_attr_t nf)
> +{
> +    return 0;
> +}
> +static inline int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
> +{
> +    return 0;
> +}
> +static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
> +{
> +    return 0;
> +}
> +
> +#endif /* CONFIG_SPARSE_DIRECTMAP */
> +
> +#endif /* X86_SPARSE_DIRECTMAP_H */
> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
> index 94b96c3d9b..4a1bd9d146 100644
> --- a/xen/arch/x86/mm.c
> +++ b/xen/arch/x86/mm.c
> @@ -127,6 +127,7 @@
>  #include <asm/setup.h>
>  #include <asm/shadow.h>
>  #include <asm/shared.h>
> +#include <asm/sparse-directmap.h>
>  #include <asm/trampoline.h>
>  #include <asm/traps.h>
>  #include <asm/x86_emulate.h>
> @@ -5394,11 +5395,104 @@ mfn_t xen_map_to_mfn(unsigned long va)
>      return ret;
>  }
>  
> +#ifdef CONFIG_SPARSE_DIRECTMAP
> +/*
> + * Look @va up in the hierarchy rooted at @root, without allocating anything.
> + * Returns the MFN mapped at @va, or INVALID_MFN, with the mapping's flags in
> + * L1 form in *flags, and in *nr the number of pages from @va to the end of
> + * the entry that maps @va (or that finds it unmapped).
> + */
> +mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
> +                         pte_attr_t *flags, unsigned long *nr)
> +{
> +    bool locking = system_state > SYS_STATE_boot;
> +    l4_pgentry_t l4e = root[l4_table_offset(va)];
> +    const l3_pgentry_t *pl3e = NULL;
> +    const l2_pgentry_t *pl2e = NULL;
> +    const l1_pgentry_t *pl1e = NULL;
> +    struct page_info *l3page;
> +    unsigned int shift = L4_PAGETABLE_SHIFT;
> +    mfn_t ret = INVALID_MFN;
> +
> +    L3T_INIT(l3page);
> +    *flags = 0;
> +
> +    if ( !(l4e_get_flags(l4e) & _PAGE_PRESENT) )
> +        goto out;
> +
> +    pl3e = map_l3t_from_l4e(l4e) + l3_table_offset(va);
> +    l3page = l4e_get_page(l4e);
> +    L3T_LOCK(l3page);
> +
> +    shift = L3_PAGETABLE_SHIFT;
> +    if ( !(l3e_get_flags(*pl3e) & _PAGE_PRESENT) )
> +        goto out;
> +    if ( l3e_get_flags(*pl3e) & _PAGE_PSE )
> +    {
> +        /* The superpage PAT bit sits among the address bits. */
> +        *flags = lNf_to_l1f(l3e_get_flags(*pl3e)) |
> +                 (l3e_get_intpte(*pl3e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
> +        ret = _mfn((l3e_get_pfn(*pl3e) & ~((1UL << (2 * PAGETABLE_ORDER)) - 1)) +
> +                   (PFN_DOWN(va) & ((1UL << (2 * PAGETABLE_ORDER)) - 1)));
> +        goto out;
> +    }
> +
> +    pl2e = map_l2t_from_l3e(*pl3e) + l2_table_offset(va);
> +    shift = L2_PAGETABLE_SHIFT;
> +    if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
> +        goto out;
> +    if ( l2e_get_flags(*pl2e) & _PAGE_PSE )
> +    {
> +        *flags = lNf_to_l1f(l2e_get_flags(*pl2e)) |
> +                 (l2e_get_intpte(*pl2e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
> +        ret = _mfn((l2e_get_pfn(*pl2e) & ~((1UL << PAGETABLE_ORDER) - 1)) +
> +                   l1_table_offset(va));
> +        goto out;
> +    }
> +
> +    pl1e = map_l1t_from_l2e(*pl2e) + l1_table_offset(va);
> +    shift = PAGE_SHIFT;
> +    if ( l1e_get_flags(*pl1e) & _PAGE_PRESENT )
> +    {
> +        *flags = l1e_get_flags(*pl1e);
> +        ret = l1e_get_mfn(*pl1e);
> +    }
> +
> + out:
> +    L3T_UNLOCK(l3page);
> +    unmap_domain_page(pl1e);
> +    unmap_domain_page(pl2e);
> +    unmap_domain_page(pl3e);
> +    *nr = ((((va >> shift) + 1) << shift) - va) >> PAGE_SHIFT;
> +    return ret;
> +}
> +#endif /* CONFIG_SPARSE_DIRECTMAP */
> +
> +/*
> + * Free a page-table page that the hierarchy rooted at @root has stopped
> + * using.  idle_pg_table's hierarchy has pages from the boot allocator,
> + * which the sparse directmap view maps along with every other boot-time
> + * allocation: such a page leaves the sparse view before it reaches the
> + * heap, and is kept if it cannot.  That takes the sparse view's L3 lock
> + * with an L3 lock of idle_pg_table's hierarchy held, the only order in
> + * which the two nest.  The sparse view's own page-tables come from the
> + * heap, which it does not map, and are freed directly, as the caller may
> + * hold the sparse view's L3 lock; so are tables freed straight after being
> + * allocated, having lost a race to be installed.
> + */
> +static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
> +{
> +    if ( root == idle_pg_table && sparse_dmap_drop_pagetable(mfn) )
> +        return;
> +
> +    free_xen_pagetable(mfn);
> +}
> +
>  /*
>   * map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
>   * another hierarchy sharing its virtual addresses.
>   */
> -static int map_pages_in(
> +int map_pages_in(
>      l4_pgentry_t *root,
>      unsigned long virt,
>      mfn_t mfn,
> @@ -5503,10 +5597,10 @@ static int map_pages_in(
>                          ol2e = l2t[i];
>                          if ( (l2e_get_flags(ol2e) & _PAGE_PRESENT) &&
>                               !(l2e_get_flags(ol2e) & _PAGE_PSE) )
> -                            free_xen_pagetable(l2e_get_mfn(ol2e));
> +                            free_pagetable_in(root, l2e_get_mfn(ol2e));
>                      }
>                      unmap_domain_page(l2t);
> -                    free_xen_pagetable(l3e_get_mfn(ol3e));
> +                    free_pagetable_in(root, l3e_get_mfn(ol3e));
>                  }
>              }
>  
> @@ -5604,7 +5698,7 @@ static int map_pages_in(
>                          flush_flags(l1e_get_flags(l1t[i]));
>                      flush_area(virt, flush_flags);
>                      unmap_domain_page(l1t);
> -                    free_xen_pagetable(l2e_get_mfn(ol2e));
> +                    free_pagetable_in(root, l2e_get_mfn(ol2e));
>                  }
>              }
>  
> @@ -5736,7 +5830,7 @@ static int map_pages_in(
>                      flush_area(virt - PAGE_SIZE,
>                                 FLUSH_TLB_GLOBAL |
>                                 FLUSH_ORDER(PAGETABLE_ORDER));
> -                    free_xen_pagetable(l2e_get_mfn(ol2e));
> +                    free_pagetable_in(root, l2e_get_mfn(ol2e));
>                  }
>                  else if ( locking )
>                      spin_unlock(&map_pgdir_lock);
> @@ -5783,7 +5877,7 @@ static int map_pages_in(
>                  flush_area(virt - PAGE_SIZE,
>                             FLUSH_TLB_GLOBAL |
>                             FLUSH_ORDER(2*PAGETABLE_ORDER));
> -                free_xen_pagetable(l3e_get_mfn(ol3e));
> +                free_pagetable_in(root, l3e_get_mfn(ol3e));
>              }
>              else if ( locking )
>                  spin_unlock(&map_pgdir_lock);
> @@ -5810,7 +5904,12 @@ int map_pages_to_xen(
>      unsigned long nr_mfns,
>      pte_attr_t flags)
>  {
> -    return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
> +    int rc = map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
> +
> +    if ( !rc )
> +        rc = sparse_dmap_mirror_map(virt, mfn, nr_mfns, flags);
> +
> +    return rc;
>  }
>  
>  int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
> @@ -5833,8 +5932,8 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
>   * modify_mappings_in() does so in the hierarchy rooted at @root, as
>   * map_pages_in() does.
>   */
> -static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
> -                              unsigned long e, pte_attr_t nf)
> +int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
> +                       unsigned long e, pte_attr_t nf)
>  {
>      bool locking = system_state > SYS_STATE_boot;
>      l3_pgentry_t *pl3e = NULL;
> @@ -6044,7 +6143,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
>                  if ( locking )
>                      spin_unlock(&map_pgdir_lock);
>                  flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
> -                free_xen_pagetable(l1mfn);
> +                free_pagetable_in(root, l1mfn);
>              }
>              else if ( locking )
>                  spin_unlock(&map_pgdir_lock);
> @@ -6088,7 +6187,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
>                  if ( locking )
>                      spin_unlock(&map_pgdir_lock);
>                  flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
> -                free_xen_pagetable(l2mfn);
> +                free_pagetable_in(root, l2mfn);
>              }
>              else if ( locking )
>                  spin_unlock(&map_pgdir_lock);
> @@ -6114,7 +6213,12 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
>  
>  int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
>  {
> -    return modify_mappings_in(idle_pg_table, s, e, nf);
> +    int rc = modify_mappings_in(idle_pg_table, s, e, nf);
> +
> +    if ( !rc )
> +        rc = sparse_dmap_mirror_modify(s, e, nf);
> +
> +    return rc;
>  }
>  
>  int destroy_xen_mappings(unsigned long s, unsigned long e)
> diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
> index bf980eba8c..f0a1197e62 100644
> --- a/xen/arch/x86/pv/dom0_build.c
> +++ b/xen/arch/x86/pv/dom0_build.c
> @@ -22,6 +22,7 @@
>  #include <asm/pv/mm.h>
>  #include <asm/setup.h>
>  #include <asm/shadow.h>
> +#include <asm/sparse-directmap.h>
>  
>  /* Allow ring-3 access in long mode as guest cannot use ring 1 ... */
>  #define BASE_PROT (_PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_USER)
> @@ -651,6 +652,9 @@ static int __init dom0_construct(const struct boot_domain *bd)
>              while ( count-- )
>                  if ( assign_pages(mfn_to_page(_mfn(mfn++)), 1, d, 0) )
>                      BUG();
> +            /* The initrd is dom0's now: out of the sparse directmap. */
> +            if ( sparse_dmap_remove(_mfn(initrd_mfn), PFN_UP(initrd_len)) )
> +                panic("Cannot unmap dom0's initrd\n");
>              /*
>               * We have mapped the initrd directly into dom0, and assigned the
>               * pages. Tell the boot_module handling that we've freed it, so the
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index a636e3205c..e52388ad28 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -54,6 +54,7 @@
>  #include <asm/spec_ctrl.h>
>  #include <asm/stubs.h>
>  #include <asm/tboot.h>
> +#include <asm/sparse-directmap.h>
>  #include <asm/trampoline.h>
>  #include <asm/traps.h>
>  
> @@ -1938,6 +1939,9 @@ void asmlinkage __init noreturn __start_xen(void)
>  
>      system_state = SYS_STATE_boot;
>  
> +    /* The heap now has all memory: page-tables come from it from here on. */
> +    sparse_dmap_init();
> +
>      bsp_stack = cpu_alloc_stack(0); /* Needs to know IDT vs FRED */
>      if ( !bsp_stack )
>          panic("No memory for BSP stack\n");
> diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
> new file mode 100644
> index 0000000000..1eb092c3ea
> --- /dev/null
> +++ b/xen/arch/x86/sparse-directmap.c
> @@ -0,0 +1,424 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * A sparse view of the directmap.
> + *
> + * Xen maps all of RAM at DIRECTMAP_VIRT_START, and every root page-table
> + * carries those mappings.  The sparse view is a second set of page-tables
> + * for the same virtual addresses (the part of the directmap PV guests can
> + * see, which is where everything Xen reaches through the directmap lives),
> + * mapping only what Xen needs to reach through the directmap at all times.
> + *
> + * Memory in the full directmap can be classified into three groups:
> + *
> + * - Memory never handed to the heap.  This includes Xen's image,
> + *   boot-time allocations, and firmware tables.  Mapped in the sparse
> + *   view.
> + * - Memory handed to the heap, and not allocated to the xenheap.  Not
> + *   mapped in the sparse view: guest memory will come from here.
> + * - Memory handed to the heap, then allocated to the xenheap.  Mapped
> + *   in the sparse view.
> + *
> + * Guest contexts selected on the command line run on root page-tables
> + * carrying the sparse view's L3s in the directmap slots.
> + *
> + * Conceptually, the sparse view starts as a copy of the full directmap
> + * (idle_pg_table's hierarchy), and then:
> + * - Removes memory when it is handed to the heap.
> + * - Adds pages back when they are allocated as xenheap pages, and removes
> + *   them again when they are freed.
> + * - Mirrors changes to the full directmap (map_pages_to_xen(),
> + *   modify_xen_mappings()).
> + * - Removes memory handed to a domain without passing through the heap
> + *   (a PV dom0's initrd, used in place), and page-tables the boot
> + *   allocator provided when idle_pg_table's hierarchy frees them.
> + *
> + * Every virtual address mapping in the sparse view should be either
> + * identical to the corresponding full view mapping, or empty.
> + *
> + * The sparse view is built once the boot allocator has handed its
> + * memory to the heap (sparse_dmap_init()).  Until then, the hooks record
> + * the ranges handed to the heap and the xenheap allocations made from
> + * them, which sparse_dmap_init() leaves out and puts back.
> + */
> +
> +#include <xen/domain_page.h>
> +#include <xen/init.h>
> +#include <xen/lib.h>
> +#include <xen/mm.h>
> +#include <xen/numa.h>
> +#include <xen/sort.h>
> +
> +#include <asm/page.h>
> +#include <asm/sparse-directmap.h>
> +
> +bool __ro_after_init opt_sparse_dmap_pv;
> +bool __ro_after_init opt_sparse_dmap_hvm;

Do we need an extra opt_sparse_dmap_dom0?  So that dom0 can opt-out
independently of everything else?

> +
> +/* Root of the sparse view: only ever walked and copied from, never loaded. */
> +static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
> +l4_pgentry_t *__ro_after_init sparse_dmap_root;
> +
> +/*
> + * Before the sparse view is built: the MFN ranges handed to the heap, which
> + * it leaves out, and the xenheap allocations made meanwhile (heap metadata of
> + * NUMA nodes), which it puts back.

Explicitly mentioning NUMA node metadata is likely to go out of date,
as changing that logic will most likely forget to update the comment
here.

> + */
> +struct mfn_range {
> +    unsigned long s, e;
> +};
> +static struct mfn_range early_heap[256];
> +static unsigned int nr_early_heap;
> +static struct mfn_range early_xenheap[2 * MAX_NUMNODES];
> +static unsigned int nr_early_xenheap;
> +static bool early_overflow;
> +
> +/*
> + * Clip [*mfn, *mfn + *nr) to what the sparse view covers; false if nothing is
> + * left.
> + */
> +static bool clip_to_sparse_view(unsigned long *mfn, unsigned long *nr)
> +{
> +    unsigned long end = SPARSE_DMAP_END_MFN;
> +
> +    if ( *mfn >= end )
> +        return false;
> +    if ( *nr > end - *mfn )
> +        *nr = end - *mfn;
> +
> +    return *nr;
> +}
> +
> +static void early_range_add(struct mfn_range *r, unsigned int *nr,
> +                            unsigned int max, unsigned long s, unsigned long e)
> +{
> +    if ( *nr && r[*nr - 1].e == s )
> +        r[*nr - 1].e = e;
> +    else if ( *nr < max )
> +        r[(*nr)++] = (struct mfn_range){ s, e };
> +    else
> +        early_overflow = true;
> +}
> +
> +static void early_range_del(struct mfn_range *r, unsigned int *nr,
> +                            unsigned long s, unsigned long e)
> +{
> +    unsigned int i;
> +
> +    for ( i = 0; i < *nr; i++ )
> +        if ( r[i].s == s && r[i].e == e )
> +        {
> +            r[i] = r[--*nr];
> +            return;
> +        }
> +}
> +
> +/*
> + * Copy the full directmap's mappings of MFNs [mfn, mfn + nr) into the sparse
> + * view.
> + */
> +static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
> +                              unsigned long nr)
> +{
> +    unsigned long va = (unsigned long)mfn_to_virt(mfn);
> +    unsigned long end = va + (nr << PAGE_SHIFT);
> +
> +    while ( va < end )
> +    {
> +        pte_attr_t flags;
> +        unsigned long n;
> +        mfn_t m = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
> +        int rc;

Since you are fetching the mfn from the idle (non-sparse) page-tables,
don't you want to check it matches the mfn passed as the function
parameter?

> +        n = min(n, (end - va) >> PAGE_SHIFT);
> +        if ( !mfn_eq(m, INVALID_MFN) &&
> +             (rc = map_pages_in(root, va, m, n, flags)) )
> +            return rc;
> +        va += n << PAGE_SHIFT;
> +    }
> +
> +    return 0;
> +}
> +
> +static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
> +{
> +    unsigned long va = (unsigned long)mfn_to_virt(mfn);
> +
> +    return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
> +                              _PAGE_NONE);
> +}
> +
> +/* Memory is about to be handed to the heap allocator. */
> +bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
> +{
> +    unsigned long s = mfn_x(mfn);
> +
> +    if ( !sparse_dmap_configured() || !clip_to_sparse_view(&s, &nr) )
> +        return true;
> +
> +    if ( !sparse_dmap_root )
> +    {
> +        early_range_add(early_heap, &nr_early_heap, ARRAY_SIZE(early_heap),
> +                        s, s + nr);
> +        return true;
> +    }
> +
> +    if ( remove_from_sparse_view(s, nr) )
> +    {
> +        printk(XENLOG_ERR
> +               "sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
> +               s, s + nr - 1);
> +        return false;
> +    }
> +
> +    return true;
> +}
> +
> +/*
> + * Xenheap pages enter the sparse view, which guest contexts run on: they
> + * must not bring along what a previous owner left in them, such as a dying
> + * domain's data still awaiting the scrubber.  Pages needing scrubbing are
> + * scrubbed as they are allocated then; clean ones need nothing.
> + */
> +bool arch_xenheap_needs_scrub(void)
> +{
> +    return sparse_dmap_configured();
> +}
> +
> +/* Pages have been allocated from the heap as xenheap pages. */
> +int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
> +{
> +    unsigned long s = mfn_x(mfn), nr = 1UL << order;
> +
> +    if ( !sparse_dmap_configured() )
> +        return 0;
> +
> +    /* Xenheap pages must be reachable through the sparse view. */
> +    BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
> +
> +    if ( !sparse_dmap_root )
> +    {
> +        early_range_add(early_xenheap, &nr_early_xenheap,
> +                        ARRAY_SIZE(early_xenheap), s, s + nr);
> +        return 0;
> +    }
> +
> +    return add_to_sparse_view(sparse_dmap_root, s, nr);
> +}
> +
> +/*
> + * Xenheap pages are about to be freed.  A failure means the pages may still
> + * be reachable through the sparse view, and must not be reused.
> + */
> +int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
> +{
> +    unsigned long s = mfn_x(mfn), nr = 1UL << order;
> +
> +    if ( !sparse_dmap_configured() )
> +        return 0;
> +
> +    BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
> +
> +    if ( !sparse_dmap_root )
> +    {
> +        early_range_del(early_xenheap, &nr_early_xenheap, s, s + nr);
> +        return 0;
> +    }
> +
> +    return remove_from_sparse_view(s, nr);
> +}
> +
> +/* Memory leaves Xen's hands without passing through the heap. */
> +int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
> +{
> +    return arch_heap_pages_init(mfn, nr) ? 0 : -ENOMEM;
> +}
> +
> +/*
> + * A page-table page of idle_pg_table's hierarchy is about to be freed.  One
> + * the boot allocator provided is in the sparse view, as every boot-time
> + * allocation is: take it out before the heap may hand it on.  One allocated
> + * since came from the heap and is not in the sparse view, which a lookup
> + * finds without changing, or flushing, anything.  Non-zero: the page may
> + * still be mapped, and must not be freed.
> + */
> +int sparse_dmap_drop_pagetable(mfn_t mfn)
> +{
> +    unsigned long s = mfn_x(mfn), nr = 1, n;
> +    pte_attr_t flags;
> +    int rc;
> +
> +    if ( !sparse_dmap_root || !clip_to_sparse_view(&s, &nr) ||
> +         mfn_eq(xen_mapping_lookup(sparse_dmap_root,
> +                                   (unsigned long)mfn_to_virt(s), &flags, &n),
> +                INVALID_MFN) )
> +        return 0;
> +
> +    rc = remove_from_sparse_view(s, 1);
> +    if ( rc )
> +        printk(XENLOG_WARNING
> +               "Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
> +               mfn_x(mfn));
> +
> +    return rc;
> +}
> +
> +int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
> +                           pte_attr_t flags)
> +{
> +    if ( !sparse_dmap_root || virt < SPARSE_DMAP_START ||
> +         virt >= SPARSE_DMAP_END )
> +        return 0;
> +
> +    nr = min(nr, (SPARSE_DMAP_END - virt) >> PAGE_SHIFT);
> +
> +    return map_pages_in(sparse_dmap_root, virt, mfn, nr, flags);
> +}
> +
> +int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
> +                              pte_attr_t nf)
> +{
> +    unsigned long va;
> +
> +    if ( !sparse_dmap_root || e <= SPARSE_DMAP_START ||
> +         s >= SPARSE_DMAP_END )
> +        return 0;
> +
> +    s = max(s, SPARSE_DMAP_START);
> +    e = min(e, SPARSE_DMAP_END);
> +
> +    if ( !(nf & _PAGE_PRESENT) )
> +        return modify_mappings_in(sparse_dmap_root, s, e, nf);
> +
> +    /*
> +     * A permission change applies where the sparse view maps anything: it
> +     * never creates mappings, and the sparse view need not map all of [s, e).
> +     */
> +    for ( va = s; va < e; )
> +    {
> +        pte_attr_t flags;
> +        unsigned long n;
> +        mfn_t m = xen_mapping_lookup(sparse_dmap_root, va, &flags, &n);
> +        int rc;
> +
> +        n = min(n, (e - va) >> PAGE_SHIFT);
> +        if ( !mfn_eq(m, INVALID_MFN) &&
> +             (rc = modify_mappings_in(sparse_dmap_root, va,
> +                                      va + (n << PAGE_SHIFT), nf)) )
> +            return rc;
> +        va += n << PAGE_SHIFT;
> +    }
> +
> +    return 0;
> +}
> +
> +static int __init cf_check cmp_mfn_range(const void *a, const void *b)
> +{
> +    const struct mfn_range *l = a, *r = b;
> +
> +    return l->s < r->s ? -1 : l->s > r->s;

The last arm of this ternary operator is doing an implicit conversion
from boolean to integer, I think that's not fine MISRA-wise?

> +}
> +
> +static void __init cf_check swap_mfn_range(void *a, void *b)
> +{
> +    SWAP(*(struct mfn_range *)a, *(struct mfn_range *)b);
> +}
> +
> +/*
> + * Build the sparse view: the full directmap, minus the ranges handed to the
> + * heap so far, plus the xenheap allocations made from them.  Called once the
> + * heap has taken over from the boot allocator, so that the page-tables come
> + * from the heap, and before any other CPU is up.
> + */
> +void __init sparse_dmap_init(void)
> +{
> +    unsigned long va, mapped = 0;
> +    unsigned int i, j;
> +
> +    if ( !sparse_dmap_configured() )
> +        return;
> +
> +    if ( early_overflow )
> +    {
> +        printk(XENLOG_WARNING
> +               "Sparse directmap: too many early heap ranges, disabled\n");
> +        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = false;
> +        return;
> +    }
> +
> +    /* An L3 for every slot, so that the L4 entries never change. */
> +    for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END;
> +          va += 1UL << L4_PAGETABLE_SHIFT )
> +    {
> +        mfn_t l3mfn;
> +        void *l3t = alloc_mapped_pagetable(&l3mfn);
> +
> +        if ( !l3t )
> +            panic("Sparse directmap: out of memory\n");
> +        unmap_domain_page(l3t);
> +        sparse_l4[l4_table_offset(va)] =
> +            l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
> +    }
> +
> +    sort(early_heap, nr_early_heap, sizeof(*early_heap), cmp_mfn_range,
> +         swap_mfn_range);
> +
> +    /*
> +     * The full directmap, leaf by leaf, minus the ranges added to the
> +     * heap.  Leaves come in address order, which is MFN order in the
> +     * directmap, so the cursor into the sorted (and disjoint) heap
> +     * ranges only moves forward.
> +     */
> +    for ( va = SPARSE_DMAP_START, j = 0; va < SPARSE_DMAP_END; )
> +    {
> +        pte_attr_t flags;
> +        unsigned long n, s, e;
> +        mfn_t m = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
> +
> +        n = min(n, (SPARSE_DMAP_END - va) >> PAGE_SHIFT);
> +        if ( mfn_eq(m, INVALID_MFN) )
> +        {
> +            va += n << PAGE_SHIFT;
> +            continue;
> +        }
> +
> +        for ( s = mfn_x(m), e = s + n; s < e; )
> +        {
> +            unsigned long upto = e;
> +
> +            while ( j < nr_early_heap && early_heap[j].e <= s )
> +                j++;
> +            if ( j < nr_early_heap && early_heap[j].s <= s )
> +            {
> +                s = min(e, early_heap[j].e);
> +                continue;
> +            }
> +            if ( j < nr_early_heap )
> +                upto = min(e, early_heap[j].s);
> +
> +            if ( map_pages_in(sparse_l4, va + ((s - mfn_x(m)) << PAGE_SHIFT),
> +                              _mfn(s), upto - s, flags) )

Why use map_pages_in() directly here, instead of add_to_sparse_view()?

> +                panic("Sparse directmap: out of memory\n");
> +            mapped += upto - s;
> +            s = upto;

The above logic is complicated to follow, maybe some simple comments
will help.

> +        }
> +
> +        va += n << PAGE_SHIFT;
> +    }
> +
> +    /*
> +     * Now add back in ranges from the heap which were allocated to
> +     * the xenheap.
> +     */
> +    for ( i = 0; i < nr_early_xenheap; i++ )
> +    {
> +        unsigned long nr = early_xenheap[i].e - early_xenheap[i].s;
> +
> +        if ( add_to_sparse_view(sparse_l4, early_xenheap[i].s, nr) )
> +            panic("Sparse directmap: out of memory\n");
> +        mapped += nr;
> +    }
> +
> +    sparse_dmap_root = sparse_l4;

This logic involves a fair amount of calls to map_pages_in(), I think
there will be no flushes in this case, because we are strictly
populating the page-tables, and hence the old PTE will always be
non-present.  Worth confirming, otherwise we might need some TLB flush
avoidance, as doing TLB flushes makes no sense here, the page-tables
are not in use.

Thanks, Roger.

[PATCH v3 14/18] x86/domain_page: key the directmap fast paths on the loaded view

[George Dunlap]

map_domain_page() returns directmap addresses whenever it can: for any
MFN the directmap covers in release builds, and for vCPUs without a
mapcache.  Once guest contexts may run on the sparse view of the
directmap, modify this to take the fast path returning the directmap
address only when the full directmap is loaded.

The release-build shortcut ahead of the vCPU lookup looks at the vCPU
whose page-tables are loaded, without completing a lazy context
switch: if that vCPU is not on the sparse view, its page-tables map
the page through the directmap, and completing the switch would only
move the CPU to the idle vCPU's page-tables, which map it too.

mapcache_current_vcpu() may complete the lazy context switch
(sync_local_execstate()) and switch us onto the idle vCPU (which has a
full view), so check again if we can use the directmap afterwards.

A vCPU on the sparse view always has a mapcache: HVM ones through
per-vCPU page-tables, PV ones through the domain-wide mapcache, which
mapcache_domain_init() now sets up for a domain using the sparse view in
release builds too, rather than only when the directmap does not cover
all memory.

map_domain_page_global() mappings must be valid in every context: use
vmap() for all of them once a sparse view is configured, as
arch_mfns_in_directmap() then says.

Add d->arch.sparse_dmap and sparse_dmap_active for this; nothing sets
them yet.

No functional change.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/domain_page.c                  | 44 +++++++++++++++++++--
 xen/arch/x86/include/asm/domain.h           |  6 +++
 xen/arch/x86/include/asm/sparse-directmap.h |  8 ++++
 xen/arch/x86/sparse-directmap.c             |  1 +
 4 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 0500bb9dba..48f85b670d 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -61,6 +61,16 @@ static inline struct vcpu *mapcache_current_vcpu(void)
     return ACCESS_ONCE(this_cpu(pgtable_vcpu));
 }
 
+/*
+ * Whether the directmap in @v's page-tables is the sparse view, which maps
+ * none of the heap's memory beyond the xenheap: the mapcache must serve
+ * every mapping then.
+ */
+static bool on_sparse_dmap(const struct vcpu *v)
+{
+    return sparse_dmap_active && v->domain->arch.sparse_dmap;
+}
+
 /*
  * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
  * otherwise, in which case *dcache is set so the caller takes the domain-wide
@@ -142,17 +152,39 @@ void *map_domain_page(mfn_t mfn)
 
 #ifdef NDEBUG
     if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
-        return mfn_to_virt(mfn_x(mfn));
+    {
+        const struct vcpu *pt_vcpu = this_cpu(pgtable_vcpu);
+
+        /*
+         * The page-tables loaded map @mfn through the directmap, unless they
+         * carry the sparse view.  Don't complete a lazy context switch to
+         * find out: it would only move this CPU to the idle vCPU's
+         * page-tables, which map @mfn too.
+         */
+        if ( !pt_vcpu || !on_sparse_dmap(pt_vcpu) )
+            return mfn_to_virt(mfn_x(mfn));
+    }
 #endif
 
+    /* No vCPU (early boot), or no mapcache: the full directmap is loaded. */
     v = mapcache_current_vcpu();
     if ( !v )
         return mfn_to_virt(mfn_x(mfn));
 
+#ifdef NDEBUG
+    /* The lookup may have completed a lazy switch off the sparse view. */
+    if ( !on_sparse_dmap(v) &&
+         mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+        return mfn_to_virt(mfn_x(mfn));
+#endif
+
     cache = vcpu_mapcache(v, &dcache);
     vcache = &v->arch.mapcache;
     if ( !cache->inuse )
+    {
+        ASSERT(!on_sparse_dmap(v));
         return mfn_to_virt(mfn_x(mfn));
+    }
 
     perfc_incr(map_domain_page_count);
 
@@ -307,7 +339,9 @@ void mapcache_domain_init(struct domain *d)
     ASSERT(is_pv_domain(d));
 
 #ifdef NDEBUG
-    if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+    /* Not needed when the directmap loaded for the domain covers everything. */
+    if ( !d->arch.sparse_dmap && !mem_hotplug &&
+         max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
         return;
 #endif
 
@@ -392,8 +426,12 @@ void *map_domain_page_global(mfn_t mfn)
              system_state < SYS_STATE_active) ||
             local_irq_is_enabled()));
 
+    /*
+     * Global mappings must hold in every context: a sparse view of the
+     * directmap rules the directmap out.
+     */
 #ifdef NDEBUG
-    if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+    if ( arch_mfns_in_directmap(mfn_x(mfn), 1) )
         return mfn_to_virt(mfn_x(mfn));
 #endif
 
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 80415ab2c2..76a22b4040 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -499,6 +499,12 @@ struct arch_domain
      */
     bool vcpu_pt;
 
+    /*
+     * Root page-tables carry the sparse view of the directmap, once it is
+     * active (see asm/sparse-directmap.h).
+     */
+    bool sparse_dmap;
+
     /* Emulated devices enabled bitmap. */
     uint32_t emulation_flags;
 } __cacheline_aligned;
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 48c1f3c951..512f8064d6 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -22,6 +22,12 @@ extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
 /* Root of the sparse view, once built (NULL before, or if not configured). */
 extern l4_pgentry_t *sparse_dmap_root;
 
+/*
+ * Set once guest contexts run on the sparse view: from then on, the root
+ * page-tables of domains with d->arch.sparse_dmap carry it.
+ */
+extern bool sparse_dmap_active;
+
 static inline bool sparse_dmap_configured(void)
 {
     return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
@@ -37,6 +43,8 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
 
 #else /* !CONFIG_SPARSE_DIRECTMAP */
 
+#define sparse_dmap_active false
+
 static inline bool sparse_dmap_configured(void) { return false; }
 static inline void sparse_dmap_init(void) {}
 static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index 1eb092c3ea..a41731ccfa 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -57,6 +57,7 @@ bool __ro_after_init opt_sparse_dmap_hvm;
 /* Root of the sparse view: only ever walked and copied from, never loaded. */
 static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
 l4_pgentry_t *__ro_after_init sparse_dmap_root;
+bool __read_mostly sparse_dmap_active;
 
 /*
  * Before the sparse view is built: the MFN ranges handed to the heap, which
-- 
2.55.0

Re: [PATCH v3 14/18] x86/domain_page: key the directmap fast paths on the loaded view

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:47AM +0100, George Dunlap wrote:
> map_domain_page() returns directmap addresses whenever it can: for any
> MFN the directmap covers in release builds, and for vCPUs without a
> mapcache.  Once guest contexts may run on the sparse view of the
> directmap, modify this to take the fast path returning the directmap
> address only when the full directmap is loaded.
> 
> The release-build shortcut ahead of the vCPU lookup looks at the vCPU
> whose page-tables are loaded, without completing a lazy context
> switch: if that vCPU is not on the sparse view, its page-tables map
> the page through the directmap, and completing the switch would only
> move the CPU to the idle vCPU's page-tables, which map it too.
> 
> mapcache_current_vcpu() may complete the lazy context switch
> (sync_local_execstate()) and switch us onto the idle vCPU (which has a
> full view), so check again if we can use the directmap afterwards.
> 
> A vCPU on the sparse view always has a mapcache: HVM ones through
> per-vCPU page-tables, PV ones through the domain-wide mapcache, which
> mapcache_domain_init() now sets up for a domain using the sparse view in
> release builds too, rather than only when the directmap does not cover
> all memory.
> 
> map_domain_page_global() mappings must be valid in every context: use
> vmap() for all of them once a sparse view is configured, as
> arch_mfns_in_directmap() then says.
> 
> Add d->arch.sparse_dmap and sparse_dmap_active for this; nothing sets
> them yet.
> 
> No functional change.
> 
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version.
> ---
>  xen/arch/x86/domain_page.c                  | 44 +++++++++++++++++++--
>  xen/arch/x86/include/asm/domain.h           |  6 +++
>  xen/arch/x86/include/asm/sparse-directmap.h |  8 ++++
>  xen/arch/x86/sparse-directmap.c             |  1 +
>  4 files changed, 56 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
> index 0500bb9dba..48f85b670d 100644
> --- a/xen/arch/x86/domain_page.c
> +++ b/xen/arch/x86/domain_page.c
> @@ -61,6 +61,16 @@ static inline struct vcpu *mapcache_current_vcpu(void)
>      return ACCESS_ONCE(this_cpu(pgtable_vcpu));
>  }
>  
> +/*
> + * Whether the directmap in @v's page-tables is the sparse view, which maps
> + * none of the heap's memory beyond the xenheap: the mapcache must serve
> + * every mapping then.
> + */
> +static bool on_sparse_dmap(const struct vcpu *v)
> +{
> +    return sparse_dmap_active && v->domain->arch.sparse_dmap;
> +}
> +
>  /*
>   * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
>   * otherwise, in which case *dcache is set so the caller takes the domain-wide
> @@ -142,17 +152,39 @@ void *map_domain_page(mfn_t mfn)
>  
>  #ifdef NDEBUG
>      if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
> -        return mfn_to_virt(mfn_x(mfn));
> +    {
> +        const struct vcpu *pt_vcpu = this_cpu(pgtable_vcpu);
> +
> +        /*
> +         * The page-tables loaded map @mfn through the directmap, unless they
> +         * carry the sparse view.  Don't complete a lazy context switch to
> +         * find out: it would only move this CPU to the idle vCPU's
> +         * page-tables, which map @mfn too.
> +         */
> +        if ( !pt_vcpu || !on_sparse_dmap(pt_vcpu) )
> +            return mfn_to_virt(mfn_x(mfn));
> +    }
>  #endif
>  
> +    /* No vCPU (early boot), or no mapcache: the full directmap is loaded. */
>      v = mapcache_current_vcpu();
>      if ( !v )
>          return mfn_to_virt(mfn_x(mfn));
>  
> +#ifdef NDEBUG
> +    /* The lookup may have completed a lazy switch off the sparse view. */
> +    if ( !on_sparse_dmap(v) &&
> +         mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
> +        return mfn_to_virt(mfn_x(mfn));
> +#endif
> +
>      cache = vcpu_mapcache(v, &dcache);
>      vcache = &v->arch.mapcache;
>      if ( !cache->inuse )
> +    {
> +        ASSERT(!on_sparse_dmap(v));
>          return mfn_to_virt(mfn_x(mfn));
> +    }
>  
>      perfc_incr(map_domain_page_count);
>  
> @@ -307,7 +339,9 @@ void mapcache_domain_init(struct domain *d)
>      ASSERT(is_pv_domain(d));
>  
>  #ifdef NDEBUG
> -    if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
> +    /* Not needed when the directmap loaded for the domain covers everything. */
> +    if ( !d->arch.sparse_dmap && !mem_hotplug &&
> +         max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
>          return;
>  #endif
>  
> @@ -392,8 +426,12 @@ void *map_domain_page_global(mfn_t mfn)
>               system_state < SYS_STATE_active) ||
>              local_irq_is_enabled()));
>  
> +    /*
> +     * Global mappings must hold in every context: a sparse view of the
> +     * directmap rules the directmap out.
> +     */
>  #ifdef NDEBUG
> -    if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
> +    if ( arch_mfns_in_directmap(mfn_x(mfn), 1) )
>          return mfn_to_virt(mfn_x(mfn));
>  #endif
>  
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index 80415ab2c2..76a22b4040 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -499,6 +499,12 @@ struct arch_domain
>       */
>      bool vcpu_pt;
>  
> +    /*
> +     * Root page-tables carry the sparse view of the directmap, once it is
> +     * active (see asm/sparse-directmap.h).
> +     */
> +    bool sparse_dmap;

What do you mean with "once it is active"?  Either a domain is using
the sparse direct map or not, but it shouldn't have an external
dependency on anything.  Or maybe I'm reading the comment the wrong
way.

> +
>      /* Emulated devices enabled bitmap. */
>      uint32_t emulation_flags;
>  } __cacheline_aligned;
> diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
> index 48c1f3c951..512f8064d6 100644
> --- a/xen/arch/x86/include/asm/sparse-directmap.h
> +++ b/xen/arch/x86/include/asm/sparse-directmap.h
> @@ -22,6 +22,12 @@ extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
>  /* Root of the sparse view, once built (NULL before, or if not configured). */
>  extern l4_pgentry_t *sparse_dmap_root;
>  
> +/*
> + * Set once guest contexts run on the sparse view: from then on, the root
> + * page-tables of domains with d->arch.sparse_dmap carry it.
> + */
> +extern bool sparse_dmap_active;

Shouldn't sparse_dmap_active be opt_sparse_dmap_pv ||
opt_sparse_dmap_hvm?  I'm afraid I don't understand the point of
tracking whether there are domain using the sparse direct map or not,
that's already tracked by d->arch.sparse_dmap?

Thanks, Roger.

[PATCH v3 15/18] x86/mm: flush owned pages before reuse with the sparse directmap view

[George Dunlap]

Since d7bf1b8ea275 ("x86/mm: limit deferred TLB flushing to PV domain
support") a page freed by its owner is flushed from TLBs before it is
handed out again only in builds with PV support, on the premise that PV
domains, with their (limited) control over the host MMU and over when
flushes happen, are the only ones to leave stale entries behind.

Xen's transient mappings of domain pages are another source.  The
mapcache zaps a slot when it is unmapped but defers the flush: the
slot's translation can stay in the TLB of the CPU which used it until
that CPU next flushes, by which time the page may have been freed and
handed to another domain.  Before "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains", only PV vCPUs use the mapcache, so the
PV condition covers it.  Now that the vCPUs of an HVM domain with
per-vCPU page-tables have one as well.

While the full directmap is loaded in every context, stale transient
mappings only map what's already mapped in the full directmap.  On the
sparse view of the directmap, a stale translation of a page handed to
another domain would keep it reachable from a context which has
nothing to do with it.

So flush owned pages before reuse wherever a sparse view is
configured, as before d7bf1b8ea275.  In builds without PV support,
mark_page_free() now also asks a new hook,
arch_freed_pages_need_tlbflush(), declared with the xenheap hooks the
sparse view already uses.  x86 answers with sparse_dmap_configured();
without CONFIG_SPARSE_DIRECTMAP, and on other architectures, the
answer is no.  Builds with PV support flush as before, and HVM-only
builds keep d7bf1b8ea275's saving unless a sparse view is configured,
which nothing does until a later patch.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/sparse-directmap.c | 14 ++++++++++++++
 xen/common/page_alloc.c         | 14 +++++++++-----
 xen/include/xen/mm.h            |  6 ++++++
 3 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index a41731ccfa..ef13f9e95d 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -185,6 +185,20 @@ bool arch_xenheap_needs_scrub(void)
     return sparse_dmap_configured();
 }
 
+/*
+ * Pages freed by their owner are flushed from TLBs before reuse once a
+ * sparse view is configured.  Contexts on the sparse view reach domain-heap
+ * memory only through mappings made for the purpose, and the mapcache zaps a
+ * transient mapping when it is dropped but defers the flush: a translation
+ * left behind would keep the page reachable after it has been handed to
+ * another owner.  With PV support, owned pages are flushed before reuse
+ * anyway.
+ */
+bool arch_freed_pages_need_tlbflush(void)
+{
+    return sparse_dmap_configured();
+}
+
 /* Pages have been allocated from the heap as xenheap pages. */
 int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
 {
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 69e8e982b6..477301ef7c 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1539,12 +1539,16 @@ static bool mark_page_free(struct page_info *pg, mfn_t mfn)
     }
 
     /*
-     * If a page has no owner and there's no PV domain support it does not need
-     * a safety TLB flush.  PV domains are the only domain types that can keep
-     * stale entries on the TLB, as they have (limited) control over the host
-     * MMU and when flushes are performed.
+     * If a page has no owner, or there's no PV domain support and the
+     * architecture doesn't ask for one, it does not need a safety TLB flush.
+     * PV domains can keep stale entries on the TLB, as they have (limited)
+     * control over the host MMU and when flushes are performed.  On x86,
+     * while a sparse directmap view is configured, so can Xen's transient
+     * mappings of the page, whose translations may outlive the mappings.
      */
-    pg->u.free.need_tlbflush = IS_ENABLED(CONFIG_PV) && page_get_owner(pg);
+    pg->u.free.need_tlbflush = page_get_owner(pg) &&
+                               (IS_ENABLED(CONFIG_PV) ||
+                                arch_freed_pages_need_tlbflush());
     if ( pg->u.free.need_tlbflush )
         page_set_tlbflush_timestamp(pg);
 
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 2bcc45cf58..1628f7a897 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -110,6 +110,8 @@ bool arch_xenheap_needs_scrub(void);
 int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
 /* Xenheap pages are to be freed.  Non-zero: failed, don't reuse them. */
 int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+/* Pages freed by their owner need a TLB flush before they are reused. */
+bool arch_freed_pages_need_tlbflush(void);
 #else
 static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
 {
@@ -127,6 +129,10 @@ static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
 {
     return 0;
 }
+static inline bool arch_freed_pages_need_tlbflush(void)
+{
+    return false;
+}
 #endif
 
 /* Free an allocation, and zero the pointer to it. */
-- 
2.55.0

Re: [PATCH v3 15/18] x86/mm: flush owned pages before reuse with the sparse directmap view

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:48AM +0100, George Dunlap wrote:
> Since d7bf1b8ea275 ("x86/mm: limit deferred TLB flushing to PV domain
> support") a page freed by its owner is flushed from TLBs before it is
> handed out again only in builds with PV support, on the premise that PV
> domains, with their (limited) control over the host MMU and over when
> flushes happen, are the only ones to leave stale entries behind.
> 
> Xen's transient mappings of domain pages are another source.  The
> mapcache zaps a slot when it is unmapped but defers the flush: the
> slot's translation can stay in the TLB of the CPU which used it until
> that CPU next flushes, by which time the page may have been freed and
> handed to another domain.  Before "x86/hvm: introduce per-vCPU
> mapcache for vCPU-PT domains", only PV vCPUs use the mapcache, so the
> PV condition covers it.  Now that the vCPUs of an HVM domain with
> per-vCPU page-tables have one as well.
> 
> While the full directmap is loaded in every context, stale transient
> mappings only map what's already mapped in the full directmap.  On the
> sparse view of the directmap, a stale translation of a page handed to
> another domain would keep it reachable from a context which has
> nothing to do with it.

But if it's handed to another domain, then it also hasn't gone through
a free cycle, which is what pg->u.free.need_tlbflush covers?

> So flush owned pages before reuse wherever a sparse view is
> configured, as before d7bf1b8ea275.  In builds without PV support,
> mark_page_free() now also asks a new hook,
> arch_freed_pages_need_tlbflush(), declared with the xenheap hooks the
> sparse view already uses.  x86 answers with sparse_dmap_configured();
> without CONFIG_SPARSE_DIRECTMAP, and on other architectures, the
> answer is no.  Builds with PV support flush as before, and HVM-only
> builds keep d7bf1b8ea275's saving unless a sparse view is configured,
> which nothing does until a later patch.

I think this is not fully correct, in fact I think we need to flush
the map cache on every return to guest, otherwise you risk leaking
contents between processes running on the same vCPU (and by extension
the same physical CPU).

We either need a flush flush before returning to guest, or we need to
invalidate every page that unmapped from the mapcache.

Thanks, Roger.

[PATCH v3 16/18] x86/domain_page: keep the maphash's slots, not their mappings

[George Dunlap]

unmap_domain_page() keeps a recently unmapped slot in the vCPU's
maphash (MAPHASH_ENTRIES, 8 per vCPU) for a quick remap of the same
MFN, and leaves its PTE in place until the entry is displaced by a later
unmap or reclaimed by a full cache.  Until then the page stays mapped,
although nothing refers to the mapping and the mapcache holds no
reference to the page.  A page freed meanwhile and handed to another
domain stays mapped in the previous user's mapcache: in the vCPU's own
area for a domain with per-vCPU page-tables, in the domain-wide area of
a PV domain, where all of its vCPUs can reach it.

While the full directmap is loaded, that reaches nothing it does not
map anyway.  On the sparse view of the directmap, Xen reaches
domain-heap memory only through mappings made for the purpose, and a
mapping kept in the maphash would leave another domain's page
reachable from a context which has nothing to do with it.

Keep the slot, not the mapping: unmapping the last reference zaps the
PTE and keeps the hash entry, and a later hit maps the page again in the
same slot.  The slot has mapped nothing else in between, so that needs
no flush.  Reusing the slot for another page still goes through the
garbage map, or through the replacement of a hash entry when the cache
is full, both of which flush first.  A translation of the zapped PTE
left in a TLB is covered, for owned pages, by the flush the page
allocator arranges before a freed page is reused wherever a sparse view
is configured ("x86/mm: flush owned pages before reuse with the sparse
directmap view").

The cost is one PTE write per unmap into the maphash and per remap from
it.  A second unmap of an entry in the maphash now finds its PTE not
present and is caught by the check unmap_domain_page() has for entries
the first unmap zapped, rather than by the reference count's ASSERT()
in debug builds only.

The change does not depend on asi= or on a sparse view of the directmap
being configured.  PV vCPUs use the domain-wide mapcache on the default
command line too: in debug builds for every page, in release builds for
pages beyond the reach of the full directmap in their page-tables (5TiB,
3.5TiB with CONFIG_BIGMEM).  For them the change adds the PTE writes
above.  It also leaves up to MAPHASH_ENTRIES fewer pages per vCPU mapped
in the domain-wide area.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/domain_page.c        | 34 +++++++++++++++++++++++--------
 xen/arch/x86/include/asm/domain.h |  5 ++++-
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 48f85b670d..eee66b848c 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -195,6 +195,14 @@ void *map_domain_page(mfn_t mfn)
     {
         idx = hashent->idx;
         ASSERT(idx < cache->entries);
+        /*
+         * An entry the maphash kept holds on to its slot, not to its mapping
+         * (see unmap_domain_page()): map the page again.  No flush is needed,
+         * as the slot has mapped nothing else since.
+         */
+        if ( !hashent->refcnt )
+            l1e_write(&MAPCACHE_L1ENT(idx),
+                      l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
         hashent->refcnt++;
         ASSERT(hashent->refcnt);
         ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
@@ -227,8 +235,8 @@ void *map_domain_page(mfn_t mfn)
                 if ( hashent->idx != MAPHASHENT_NOTINUSE && !hashent->refcnt )
                 {
                     idx = hashent->idx;
-                    ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(idx)) == hashent->mfn);
-                    l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+                    ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(idx)) &
+                             _PAGE_PRESENT));
                     hashent->idx = MAPHASHENT_NOTINUSE;
                     hashent->mfn = ~0UL;
                     break;
@@ -298,25 +306,35 @@ void unmap_domain_page(const void *ptr)
 
     local_irq_save(flags);
 
+    /*
+     * The maphash keeps recently freed slots for a quick remap of the same
+     * page, but not their mappings: the page may change hands meanwhile, and
+     * must not stay reachable from this context then.  A translation left in
+     * a TLB is covered, for owned pages, by the flush the page allocator
+     * arranges before a freed page is handed out again wherever a sparse
+     * directmap view is configured; without one, the directmap maps the page
+     * anyway.
+     */
     if ( hashent->idx == idx )
     {
         ASSERT(hashent->mfn == mfn);
         ASSERT(hashent->refcnt);
         hashent->refcnt--;
+        if ( !hashent->refcnt )
+            l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
     }
     else if ( !hashent->refcnt )
     {
+        /* The entry replaced has no mapping left: mark its slot as garbage. */
         if ( hashent->idx != MAPHASHENT_NOTINUSE )
         {
-            /* /First/, zap the PTE. */
-            ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(hashent->idx)) ==
-                   hashent->mfn);
-            l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
-            /* /Second/, mark as garbage. */
+            ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(hashent->idx)) &
+                     _PAGE_PRESENT));
             set_bit(hashent->idx, cache->garbage);
         }
 
-        /* Add newly-freed mapping to the maphash. */
+        /* Add the newly-freed slot to the maphash, without its mapping. */
+        l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
         hashent->mfn = mfn;
         hashent->idx = idx;
     }
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 76a22b4040..51741f9d42 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -74,7 +74,10 @@ struct mapcache_vcpu {
     /* Shadow of mapcache_domain.epoch. */
     unsigned int shadow_epoch;
 
-    /* Lock-free per-VCPU hash of recently-used mappings. */
+    /*
+     * Lock-free per-VCPU hash of recently-used mappings.  An entry without
+     * references keeps its slot but not its mapping (see unmap_domain_page()).
+     */
     struct vcpu_maphash_entry {
         unsigned long mfn;
         uint32_t      idx;
-- 
2.55.0

Re: [PATCH v3 16/18] x86/domain_page: keep the maphash's slots, not their mappings

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:49AM +0100, George Dunlap wrote:
> unmap_domain_page() keeps a recently unmapped slot in the vCPU's
> maphash (MAPHASH_ENTRIES, 8 per vCPU) for a quick remap of the same
> MFN, and leaves its PTE in place until the entry is displaced by a later
> unmap or reclaimed by a full cache.  Until then the page stays mapped,
> although nothing refers to the mapping and the mapcache holds no
> reference to the page.  A page freed meanwhile and handed to another
> domain stays mapped in the previous user's mapcache: in the vCPU's own
> area for a domain with per-vCPU page-tables, in the domain-wide area of
> a PV domain, where all of its vCPUs can reach it.
> 
> While the full directmap is loaded, that reaches nothing it does not
> map anyway.  On the sparse view of the directmap, Xen reaches
> domain-heap memory only through mappings made for the purpose, and a
> mapping kept in the maphash would leave another domain's page
> reachable from a context which has nothing to do with it.
> 
> Keep the slot, not the mapping: unmapping the last reference zaps the
> PTE and keeps the hash entry, and a later hit maps the page again in the
> same slot.  The slot has mapped nothing else in between, so that needs
> no flush.  Reusing the slot for another page still goes through the
> garbage map, or through the replacement of a hash entry when the cache
> is full, both of which flush first.  A translation of the zapped PTE
> left in a TLB is covered, for owned pages, by the flush the page
> allocator arranges before a freed page is reused wherever a sparse view
> is configured ("x86/mm: flush owned pages before reuse with the sparse
> directmap view").

As commented on the previous patch, I'm not convinced that's enough.
I do think we need to do a flush here, as it's not so much as the page
being used by another domain (or by Xen itself), internally the guest
might also have different contexts and we should prevent leaks between
them.

> 
> The cost is one PTE write per unmap into the maphash and per remap from
> it.  A second unmap of an entry in the maphash now finds its PTE not
> present and is caught by the check unmap_domain_page() has for entries
> the first unmap zapped, rather than by the reference count's ASSERT()
> in debug builds only.
> 
> The change does not depend on asi= or on a sparse view of the directmap
> being configured.  PV vCPUs use the domain-wide mapcache on the default
> command line too: in debug builds for every page, in release builds for
> pages beyond the reach of the full directmap in their page-tables (5TiB,
> 3.5TiB with CONFIG_BIGMEM).

There's likely no need to mention again the exact conditions that make
the mapcache active for PV domains, that's IMO too verbose and doesn't
add anything of value to the reasoning here.

Thanks, Roger.

[PATCH v3 17/18] x86/mm: run guest contexts on the sparse directmap view

[George Dunlap]

In init_xen_l4_slots(), if d->arch.sparse_dmap is set, point the
directmap L3 at the sparse view.

d->arch.sparse_dmap is decided at domain creation: PV domains all have a
mapcache, HVM domains need per-vCPU page-tables for one.  No option asks
for the sparse view yet.

The sparse view becomes active once dom0 is built, before it first runs:
dom0's construction runs on the full directmap, which it needs, and its
root page-tables so far (a PV dom0's first L4s, a PVH dom0's monitor
tables) are switched over then.  A PV dom0 booted with dom0=shadow is
switched as well: shadow mode is enabled only later, from a tasklet, and
its shadow L4s then get the sparse view from init_xen_l4_slots(), as for
any PV domain entering shadow mode.  Anything built later gets the
sparse view from init_xen_l4_slots().  The sparse view's slots are
marked NX at activation, as the full directmap's slots are.

The two directmaps differ where the sparse view leaves memory out, so no
translation of the full directmap may remain usable on page-tables
carrying the sparse view.  Directmap mappings stay global nevertheless:
every CR3 write which can move a CPU onto the sparse view goes through
switch_cr3_cr4(), which flushes the old page-tables' translations,
global entries included, before it returns.  The CR3 writes which keep
entries -- a PV vCPU's switch between its kernel and user page-tables,
XPTI's entry and exit -- stay within one vCPU's page-tables, which carry
the same directmap, or go to XPTI's per-CPU root, which maps only Xen's
own per-CPU pages there.  Removals from the sparse view are flushed on
every CPU, global entries included.  PV guests running with global
pages, a PV dom0 not using the sparse view among them, so keep their
directmap TLB entries across switches between guest kernel and user
mode.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 xen/arch/x86/include/asm/sparse-directmap.h |  11 ++
 xen/arch/x86/mm.c                           |   9 +-
 xen/arch/x86/setup.c                        |   3 +
 xen/arch/x86/sparse-directmap.c             | 129 ++++++++++++++++++++
 xen/arch/x86/spec_ctrl.c                    |   7 ++
 5 files changed, 157 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 512f8064d6..41866b77e9 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -33,7 +33,15 @@ static inline bool sparse_dmap_configured(void)
     return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
 }
 
+/* Whether a new domain of the given type is to use the sparse view. */
+static inline bool sparse_dmap_wanted(bool pv)
+{
+    return sparse_dmap_root && (pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
+}
+
 void sparse_dmap_init(void);
+void sparse_dmap_activate(void);
+void sparse_dmap_install(l4_pgentry_t *l4t, bool pv);
 int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
                            pte_attr_t flags);
 int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
@@ -46,7 +54,10 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
 #define sparse_dmap_active false
 
 static inline bool sparse_dmap_configured(void) { return false; }
+static inline bool sparse_dmap_wanted(bool pv) { return false; }
 static inline void sparse_dmap_init(void) {}
+static inline void sparse_dmap_activate(void) {}
+static inline void sparse_dmap_install(l4_pgentry_t *l4t, bool pv) {}
 static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
                                          unsigned long nr, pte_attr_t flags)
 {
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 4a1bd9d146..4e2f9e1f12 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1662,7 +1662,8 @@ struct page_info *perdomain_l3(const struct vcpu *v)
  * share; it is mandatory for a vCPU-PT domain, whose per-domain area is
  * per-vCPU.  All other parameters are optional and will either fill or zero
  * the appropriate slots.  Pagetables not shared with guests will gain the
- * extended directmap.
+ * extended directmap, unless the domain runs on the sparse view of the
+ * directmap, which replaces the directmap slots (see sparse_dmap_install()).
  */
 void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
                        const struct domain *d, const struct vcpu *v,
@@ -1670,7 +1671,7 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
 {
     /*
      * PV vcpus need a shortened directmap.  HVM and Idle vcpus get the full
-     * directmap.
+     * directmap, unless the domain runs on the sparse view (below).
      */
     bool short_directmap = !paging_mode_external(d);
 
@@ -1743,6 +1744,10 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
                (ROOT_PAGETABLE_FIRST_XEN_SLOT + slots -
                 l4_table_offset(XEN_VIRT_START)) * sizeof(*l4t));
     }
+
+    /* The domain's contexts may run on the sparse view of the directmap. */
+    if ( sparse_dmap_active && d->arch.sparse_dmap )
+        sparse_dmap_install(l4t, short_directmap);
 }
 
 bool fill_ro_mpt(mfn_t mfn)
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e52388ad28..21edab9cea 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -2216,6 +2216,9 @@ void asmlinkage __init noreturn __start_xen(void)
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
 
+    /* Before dom0 first runs, and after it was built on the full directmap. */
+    sparse_dmap_activate();
+
     heap_init_late();
 
     init_constructors();
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index ef13f9e95d..54a6e36132 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -46,8 +46,10 @@
 #include <xen/lib.h>
 #include <xen/mm.h>
 #include <xen/numa.h>
+#include <xen/sched.h>
 #include <xen/sort.h>
 
+#include <asm/flushtlb.h>
 #include <asm/page.h>
 #include <asm/sparse-directmap.h>
 
@@ -437,3 +439,130 @@ void __init sparse_dmap_init(void)
 
     printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
 }
+
+/*
+ * Point the directmap slots of the root page-table @l4t at the sparse view.
+ * Every slot it covers gets its L3, whatever the full directmap has there,
+ * so that memory added later is covered too.  A PV root has guest slots
+ * above; any other root loses the rest of the directmap, which the sparse
+ * view does not cover.
+ *
+ * Directmap mappings may be global, in either hierarchy.  Translations of
+ * the full directmap must not remain usable on a root carrying the sparse
+ * view: every CR3 write which can move a CPU from the one to the other goes
+ * through switch_cr3_cr4(), which flushes what the old root left in the TLB,
+ * global entries included, before it returns.  The other CR3 writes, which
+ * may keep global or PCID-tagged entries (_toggle_guest_pt(), XPTI's entry
+ * and exit paths), switch between one vCPU's roots, which carry the same
+ * directmap, or to XPTI's per-CPU root, which maps only this CPU's stack,
+ * IDT and TSS there.  Removals from the sparse view flush all CPUs, global
+ * entries included.
+ */
+void sparse_dmap_install(l4_pgentry_t *l4t, bool pv)
+{
+    unsigned int i;
+
+    ASSERT(sparse_dmap_root);
+
+    for ( i = l4_table_offset(SPARSE_DMAP_START);
+          i < l4_table_offset(SPARSE_DMAP_END); i++ )
+        l4t[i] = sparse_dmap_root[i];
+
+    if ( !pv )
+        for ( ; i < l4_table_offset(DIRECTMAP_VIRT_END); i++ )
+            l4t[i] = l4e_empty();
+}
+
+static void __init sparse_dmap_install_mfn(mfn_t mfn, bool pv)
+{
+    l4_pgentry_t *l4t = map_domain_page(mfn);
+
+    sparse_dmap_install(l4t, pv);
+    unmap_domain_page(l4t);
+}
+
+/*
+ * Let guest contexts run on the sparse view.  Root page-tables built from
+ * here on carry it (init_xen_l4_slots()); those of the domains built so far
+ * (dom0, which has not run yet) are switched over here.  Its construction ran
+ * on the full directmap, and needed to.
+ */
+void __init sparse_dmap_activate(void)
+{
+    struct domain *d;
+    struct vcpu *v;
+    unsigned long va, mapped = 0;
+    unsigned int i, switched = 0;
+
+    if ( !sparse_dmap_root )
+        return;
+
+    /*
+     * Nothing in the directmap is executed: as subarch_init_memory() does
+     * for the full directmap, mark the sparse view's slots NX, before any
+     * root page-table takes a copy of them.
+     */
+    if ( cpu_has_nx )
+        for ( i = l4_table_offset(SPARSE_DMAP_START);
+              i < l4_table_offset(SPARSE_DMAP_END); i++ )
+            l4e_add_flags(sparse_dmap_root[i], _PAGE_NX_BIT);
+
+    rcu_read_lock(&domlist_read_lock);
+
+    for_each_domain ( d )
+    {
+        if ( !d->arch.sparse_dmap )
+            continue;
+
+        /*
+         * A PV domain runs on shadow L4s only in shadow mode, which boot
+         * does not enter before this point: dom0=shadow enables it later,
+         * from a tasklet, and the shadow L4s then get the sparse view from
+         * init_xen_l4_slots(), as for any PV domain entering shadow mode.
+         */
+        switched++;
+        for_each_vcpu ( d, v )
+        {
+            if ( is_hvm_vcpu(v) )
+            {
+                if ( !pagetable_is_null(v->arch.hvm.monitor_table) )
+                    sparse_dmap_install_mfn(
+                        pagetable_get_mfn(v->arch.hvm.monitor_table), false);
+                continue;
+            }
+
+            if ( !pagetable_is_null(v->arch.guest_table) )
+                sparse_dmap_install_mfn(
+                    pagetable_get_mfn(v->arch.guest_table), true);
+            if ( !pagetable_is_null(v->arch.guest_table_user) )
+                sparse_dmap_install_mfn(
+                    pagetable_get_mfn(v->arch.guest_table_user), true);
+        }
+    }
+
+    rcu_read_unlock(&domlist_read_lock);
+
+    /*
+     * Belt and braces: nothing runs on these page-tables yet, and PV dom0's
+     * construction, which ran on its own, returned to the idle page-tables
+     * through switch_cr3_cr4(), dropping what it left in the TLB.
+     */
+    flush_all(FLUSH_TLB_GLOBAL);
+
+    sparse_dmap_active = true;
+
+    for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END; )
+    {
+        pte_attr_t flags;
+        unsigned long n;
+
+        if ( !mfn_eq(xen_mapping_lookup(sparse_dmap_root, va, &flags, &n),
+                     INVALID_MFN) )
+            mapped += n;
+        va += n << PAGE_SHIFT;
+    }
+
+    printk(XENLOG_INFO
+           "Sparse directmap: active, %lu pages mapped, %u domain(s) on it\n",
+           mapped, switched);
+}
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index 999e41aefa..46105ef185 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -1940,6 +1940,13 @@ void spec_ctrl_init_domain(struct domain *d)
 
     d->arch.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_vcpu_pt_hwdom
                                                     : opt_vcpu_pt_hvm);
+
+    /*
+     * The sparse directmap view needs a mapcache: PV domains all have one,
+     * HVM domains through per-vCPU page-tables.
+     */
+    d->arch.sparse_dmap = pv ? sparse_dmap_wanted(true)
+                             : d->arch.vcpu_pt && sparse_dmap_wanted(false);
 }
 
 void __init init_speculation_mitigations(void)
-- 
2.55.0

Re: [PATCH v3 17/18] x86/mm: run guest contexts on the sparse directmap view

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:50AM +0100, George Dunlap wrote:
> In init_xen_l4_slots(), if d->arch.sparse_dmap is set, point the
> directmap L3 at the sparse view.
> 
> d->arch.sparse_dmap is decided at domain creation: PV domains all have a
> mapcache, HVM domains need per-vCPU page-tables for one.  No option asks
> for the sparse view yet.

Is it strictly needed to tie the sparse directmap to having a per-vCPU
mapcache on HVM?  AFAICT HVM domains could also use the classic
per-domain mapcache.

> The sparse view becomes active once dom0 is built, before it first runs:
> dom0's construction runs on the full directmap, which it needs, and its
> root page-tables so far (a PV dom0's first L4s, a PVH dom0's monitor
> tables) are switched over then.  A PV dom0 booted with dom0=shadow is
> switched as well: shadow mode is enabled only later, from a tasklet, and
> its shadow L4s then get the sparse view from init_xen_l4_slots(), as for
> any PV domain entering shadow mode.  Anything built later gets the
> sparse view from init_xen_l4_slots().  The sparse view's slots are
> marked NX at activation, as the full directmap's slots are.

Is NX worth mentioning here?  The expectation is that the sparse dmap
slots will use the same attributes as the full dmap, and hence I
would focus on the differences (if any).

> The two directmaps differ where the sparse view leaves memory out, so no
> translation of the full directmap may remain usable on page-tables
> carrying the sparse view.  Directmap mappings stay global nevertheless:
> every CR3 write which can move a CPU onto the sparse view goes through
> switch_cr3_cr4(), which flushes the old page-tables' translations,
> global entries included, before it returns.  The CR3 writes which keep
> entries -- a PV vCPU's switch between its kernel and user page-tables,
> XPTI's entry and exit -- stay within one vCPU's page-tables, which carry
> the same directmap, or go to XPTI's per-CPU root, which maps only Xen's
> own per-CPU pages there.  Removals from the sparse view are flushed on
> every CPU, global entries included.  PV guests running with global
> pages, a PV dom0 not using the sparse view among them, so keep their
> directmap TLB entries across switches between guest kernel and user
> mode.
> 
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version.
> ---
>  xen/arch/x86/include/asm/sparse-directmap.h |  11 ++
>  xen/arch/x86/mm.c                           |   9 +-
>  xen/arch/x86/setup.c                        |   3 +
>  xen/arch/x86/sparse-directmap.c             | 129 ++++++++++++++++++++
>  xen/arch/x86/spec_ctrl.c                    |   7 ++
>  5 files changed, 157 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
> index 512f8064d6..41866b77e9 100644
> --- a/xen/arch/x86/include/asm/sparse-directmap.h
> +++ b/xen/arch/x86/include/asm/sparse-directmap.h
> @@ -33,7 +33,15 @@ static inline bool sparse_dmap_configured(void)
>      return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
>  }
>  
> +/* Whether a new domain of the given type is to use the sparse view. */
> +static inline bool sparse_dmap_wanted(bool pv)
> +{
> +    return sparse_dmap_root && (pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
> +}
> +
>  void sparse_dmap_init(void);
> +void sparse_dmap_activate(void);
> +void sparse_dmap_install(l4_pgentry_t *l4t, bool pv);
>  int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
>                             pte_attr_t flags);
>  int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
> @@ -46,7 +54,10 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
>  #define sparse_dmap_active false
>  
>  static inline bool sparse_dmap_configured(void) { return false; }
> +static inline bool sparse_dmap_wanted(bool pv) { return false; }
>  static inline void sparse_dmap_init(void) {}
> +static inline void sparse_dmap_activate(void) {}
> +static inline void sparse_dmap_install(l4_pgentry_t *l4t, bool pv) {}
>  static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
>                                           unsigned long nr, pte_attr_t flags)
>  {
> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
> index 4a1bd9d146..4e2f9e1f12 100644
> --- a/xen/arch/x86/mm.c
> +++ b/xen/arch/x86/mm.c
> @@ -1662,7 +1662,8 @@ struct page_info *perdomain_l3(const struct vcpu *v)
>   * share; it is mandatory for a vCPU-PT domain, whose per-domain area is
>   * per-vCPU.  All other parameters are optional and will either fill or zero
>   * the appropriate slots.  Pagetables not shared with guests will gain the
> - * extended directmap.
> + * extended directmap, unless the domain runs on the sparse view of the
> + * directmap, which replaces the directmap slots (see sparse_dmap_install()).
>   */
>  void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
>                         const struct domain *d, const struct vcpu *v,
> @@ -1670,7 +1671,7 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
>  {
>      /*
>       * PV vcpus need a shortened directmap.  HVM and Idle vcpus get the full
> -     * directmap.
> +     * directmap, unless the domain runs on the sparse view (below).
>       */
>      bool short_directmap = !paging_mode_external(d);
>  
> @@ -1743,6 +1744,10 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
>                 (ROOT_PAGETABLE_FIRST_XEN_SLOT + slots -
>                  l4_table_offset(XEN_VIRT_START)) * sizeof(*l4t));
>      }
> +
> +    /* The domain's contexts may run on the sparse view of the directmap. */
> +    if ( sparse_dmap_active && d->arch.sparse_dmap )
> +        sparse_dmap_install(l4t, short_directmap);

Instead of doing the copy of the full directmap from the idle page
tables, and then rewriting, won't it be simpler if the copied
directmap was the correct one right away rather than doing a fixup
afterwards?

>  }
>  
>  bool fill_ro_mpt(mfn_t mfn)
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index e52388ad28..21edab9cea 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -2216,6 +2216,9 @@ void asmlinkage __init noreturn __start_xen(void)
>      if ( !dom0 )
>          panic("Could not set up DOM0 guest OS\n");
>  
> +    /* Before dom0 first runs, and after it was built on the full directmap. */
> +    sparse_dmap_activate();
> +
>      heap_init_late();
>  
>      init_constructors();
> diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
> index ef13f9e95d..54a6e36132 100644
> --- a/xen/arch/x86/sparse-directmap.c
> +++ b/xen/arch/x86/sparse-directmap.c
> @@ -46,8 +46,10 @@
>  #include <xen/lib.h>
>  #include <xen/mm.h>
>  #include <xen/numa.h>
> +#include <xen/sched.h>
>  #include <xen/sort.h>
>  
> +#include <asm/flushtlb.h>
>  #include <asm/page.h>
>  #include <asm/sparse-directmap.h>
>  
> @@ -437,3 +439,130 @@ void __init sparse_dmap_init(void)
>  
>      printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
>  }
> +
> +/*
> + * Point the directmap slots of the root page-table @l4t at the sparse view.
> + * Every slot it covers gets its L3, whatever the full directmap has there,
> + * so that memory added later is covered too.  A PV root has guest slots
> + * above; any other root loses the rest of the directmap, which the sparse
> + * view does not cover.
> + *
> + * Directmap mappings may be global, in either hierarchy.  Translations of
> + * the full directmap must not remain usable on a root carrying the sparse
> + * view: every CR3 write which can move a CPU from the one to the other goes
> + * through switch_cr3_cr4(), which flushes what the old root left in the TLB,
> + * global entries included, before it returns.  The other CR3 writes, which
> + * may keep global or PCID-tagged entries (_toggle_guest_pt(), XPTI's entry
> + * and exit paths), switch between one vCPU's roots, which carry the same
> + * directmap, or to XPTI's per-CPU root, which maps only this CPU's stack,
> + * IDT and TSS there.  Removals from the sparse view flush all CPUs, global
> + * entries included.

IMO parts of this comment are likely out of scope here.  It's fine to
discuss about the flushing strategy in the commit message, however
doing so in a comment here, that's disconnected from the definitions
of switch_cr3_cr4() and _toggle_guest_pt() seems out of place.

I would also avoid referencing function names in code comments: it's
fine to do so in the commit message, but it's likely those references
in comments will go stale as functions change names or disappear.
Commit messages are otherwise tied to a specific point in time, so the
context and the referenced functions are always accurate and it will
never go stale in the context is was committed.

> + */
> +void sparse_dmap_install(l4_pgentry_t *l4t, bool pv)
> +{
> +    unsigned int i;
> +
> +    ASSERT(sparse_dmap_root);
> +
> +    for ( i = l4_table_offset(SPARSE_DMAP_START);
> +          i < l4_table_offset(SPARSE_DMAP_END); i++ )
> +        l4t[i] = sparse_dmap_root[i];
> +
> +    if ( !pv )
> +        for ( ; i < l4_table_offset(DIRECTMAP_VIRT_END); i++ )
> +            l4t[i] = l4e_empty();
> +}
> +
> +static void __init sparse_dmap_install_mfn(mfn_t mfn, bool pv)
> +{
> +    l4_pgentry_t *l4t = map_domain_page(mfn);
> +
> +    sparse_dmap_install(l4t, pv);
> +    unmap_domain_page(l4t);
> +}
> +
> +/*
> + * Let guest contexts run on the sparse view.  Root page-tables built from
> + * here on carry it (init_xen_l4_slots()); those of the domains built so far
> + * (dom0, which has not run yet) are switched over here.  Its construction ran
> + * on the full directmap, and needed to.
> + */
> +void __init sparse_dmap_activate(void)
> +{
> +    struct domain *d;
> +    struct vcpu *v;
> +    unsigned long va, mapped = 0;
> +    unsigned int i, switched = 0;
> +
> +    if ( !sparse_dmap_root )
> +        return;
> +
> +    /*
> +     * Nothing in the directmap is executed: as subarch_init_memory() does
> +     * for the full directmap, mark the sparse view's slots NX, before any
> +     * root page-table takes a copy of them.
> +     */
> +    if ( cpu_has_nx )
> +        for ( i = l4_table_offset(SPARSE_DMAP_START);
> +              i < l4_table_offset(SPARSE_DMAP_END); i++ )
> +            l4e_add_flags(sparse_dmap_root[i], _PAGE_NX_BIT);

Shouldn't this be done when the sparse directmap L4 are populated,
instead of here?

> +
> +    rcu_read_lock(&domlist_read_lock);
> +
> +    for_each_domain ( d )
> +    {
> +        if ( !d->arch.sparse_dmap )
> +            continue;

You should likely ensure the domain is paused here, otherwise nothing
good can came out of playing with its root page-table.

> +
> +        /*
> +         * A PV domain runs on shadow L4s only in shadow mode, which boot
> +         * does not enter before this point: dom0=shadow enables it later,
> +         * from a tasklet, and the shadow L4s then get the sparse view from
> +         * init_xen_l4_slots(), as for any PV domain entering shadow mode.
> +         */
> +        switched++;
> +        for_each_vcpu ( d, v )
> +        {
> +            if ( is_hvm_vcpu(v) )

nit: you could pull is_hvm_vcpu() out of the loop and store in a
boolean, so that you don't need to call it for each domain vCPU.

> +            {
> +                if ( !pagetable_is_null(v->arch.hvm.monitor_table) )
> +                    sparse_dmap_install_mfn(
> +                        pagetable_get_mfn(v->arch.hvm.monitor_table), false);
> +                continue;
> +            }
> +
> +            if ( !pagetable_is_null(v->arch.guest_table) )
> +                sparse_dmap_install_mfn(
> +                    pagetable_get_mfn(v->arch.guest_table), true);
> +            if ( !pagetable_is_null(v->arch.guest_table_user) )
> +                sparse_dmap_install_mfn(
> +                    pagetable_get_mfn(v->arch.guest_table_user), true);
> +        }
> +    }
> +
> +    rcu_read_unlock(&domlist_read_lock);
> +
> +    /*
> +     * Belt and braces: nothing runs on these page-tables yet, and PV dom0's
> +     * construction, which ran on its own, returned to the idle page-tables
> +     * through switch_cr3_cr4(), dropping what it left in the TLB.
> +     */
> +    flush_all(FLUSH_TLB_GLOBAL);
> +
> +    sparse_dmap_active = true;
> +
> +    for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END; )
> +    {
> +        pte_attr_t flags;
> +        unsigned long n;
> +
> +        if ( !mfn_eq(xen_mapping_lookup(sparse_dmap_root, va, &flags, &n),
> +                     INVALID_MFN) )
> +            mapped += n;
> +        va += n << PAGE_SHIFT;
> +    }

This looks to be extremely expensive just for the purposes of printing
the number of mapped pages in the sparse directmap, it should be under
CONFIG_DEBUG if anything.

Thanks, Roger.

[PATCH v3 18/18] x86/spec-ctrl: wire the sparse directmap view into the asi= option

[George Dunlap]

Let asi= choose which domains run on the sparse view of the directmap,
when the hypervisor is built with CONFIG_SPARSE_DIRECTMAP:

 * `asi` and `asi=<bool>` enable or disable the sparse view, along with
   vCPU-PT, for every domain: HVM and PV guests, and the hardware
   domain, PV or PVH.  HVM guests, a PVH hardware domain included, use
   the sparse view together with vCPU-PT, which gives them the mapcache
   it needs; PV guests all have one.  A PV guest on the sparse view
   keeps its domain-wide mapcache and per-domain mappings: the sparse
   view separates it from other domains, not its vCPUs from each other.

 * `asi=hvm=<bool>`, and new `asi=pv=<bool>` and `asi=dom0=<bool>`, do
   the same for HVM or PV guests other than the hardware domain, or for
   the hardware domain, whatever its type: `asi=1,dom0=0` leaves the
   hardware domain out of ASI altogether.

 * A new `asi=directmap=<bool>` selects the sparse view for all domains,
   as `asi=vcpu-pt=<bool>` selects vCPU-PT; `vcpu-pt=` alone leaves the
   sparse view off.  On a hypervisor built without
   CONFIG_SPARSE_DIRECTMAP the forms enabling all features leave the
   sparse view off, and `directmap=` is ignored with a warning.  A guest
   type the build lacks is ignored, so that the sparse view is not built
   for nothing.

 * Only one of XPTI and the sparse view are enabled.  Enabled
   explicitly, for dom0, domUs or both, XPTI takes precedence: the PV
   domains it covers keep the full directmap, and the boot log says
   so.  Left to its default, XPTI yields: it is disabled for the PV
   domains using the sparse view, with a boot-log warning where it
   would otherwise be enabled, as the sparse view does not replace it
   yet (it still maps the xenheap and Xen's stacks).  Each domain's
   case is settled at its creation.

The boot log gains a line with the directmap mode, "sparse" once a
sparse view is configured and "full" otherwise, and the "ASI features
for ..." lines, one for PV guests now among them, say which domains use
the sparse view.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v3:
- New in this version.
---
 docs/misc/xen-command-line.pandoc           |  35 +++++-
 xen/arch/x86/include/asm/sparse-directmap.h |  28 ++++-
 xen/arch/x86/sparse-directmap.c             |   4 +-
 xen/arch/x86/spec_ctrl.c                    | 130 ++++++++++++++++++--
 4 files changed, 171 insertions(+), 26 deletions(-)

diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 785e099c0a..0bbc8ebeff 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -203,7 +203,8 @@ to appropriate auditing by Xen.  Argo is disabled by default.
     buggy or malicious other domain spamming the ring.
 
 ### asi (x86)
-> `= List of [ <bool>, hvm=<bool>, vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, directmap=<bool>,
+>              vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
 
 > Default: `false`
 
@@ -214,12 +215,16 @@ mitigations for certain speculative related attacks, at the cost of mapping
 sensitive information on-demand.  It might also offer some protection against
 unmitigated speculation-related attacks.
 
-The mechanisms currently implemented apply to HVM guests, including a PVH
-hardware domain.  PV guests, including a PV hardware domain, are unaffected.
+The mechanisms currently implemented are `vcpu-pt`, for HVM guests including
+a PVH hardware domain, and `directmap`, for all guests including the hardware
+domain, PV or PVH.  The plain boolean enables or disables both, wherever they
+apply.
 
-* `hvm=` enables the features for HVM guests other than the hardware domain,
-  which follows the whole-feature forms (the plain boolean, or an un-suffixed
-  `vcpu-pt=<bool>`).
+* `pv=`, `hvm=` and `dom0=` enable the features for PV guests or HVM guests
+  other than the hardware domain, or for the hardware domain, PV or PVH.  The
+  plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
+  out.  The forms naming no guest type (an un-suffixed `vcpu-pt=<bool>`, and
+  `directmap=<bool>`) cover the hardware domain too.
 
 **WARNING: manual de-selection of enabled options will invalidate any
 protection offered by the feature.  The fine grained options provided below
@@ -228,6 +233,18 @@ are meant to be used for debugging purposes only.**
 * `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
   each vCPU's monitor table maps a region private to that vCPU.
 
+* `directmap` (only available when Xen is built with
+  `CONFIG_SPARSE_DIRECTMAP`) runs guest contexts on a sparse view of the
+  directmap, which maps the memory Xen needs to reach at all times (the Xen
+  heap, Xen's own image, firmware tables), while guest memory is mapped
+  transiently.  This selection is system-wide; `pv=`, `hvm=` and `dom0=`
+  make it for one guest type, along with the other features.  HVM guests use
+  the sparse view only together with `vcpu-pt`, and PV guests only without
+  XPTI, which yields to the sparse view unless enabled explicitly (see
+  `xpti`).  The whole-feature forms (`asi`, `asi=<bool>`, and `pv=`, `hvm=`
+  and `dom0=`) set this too, on builds that support it; on other builds they
+  leave it off, and `directmap` is ignored with a warning.
+
 ### asid (x86)
 > `= <boolean>`
 
@@ -3118,6 +3135,12 @@ Meltdown for all domains.
 With `dom0` and `domu` it is possible to control page table isolation
 for dom0 or guest domains only.
 
+With `asi`, XPTI and the sparse directmap view (`asi=directmap`) exclude each
+other for a PV guest.  Enabled explicitly, as a whole or with `dom0` or
+`domu`, XPTI keeps the PV guests it covers off the sparse view, and the boot
+log says so.  Left to its default, XPTI is disabled for the PV guests using
+the sparse view, with a boot-log warning where it would otherwise be enabled.
+
 ### xsave (x86)
 > `= <boolean>`
 
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 41866b77e9..5349b2ba62 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -17,7 +17,11 @@
 #define SPARSE_DMAP_END     HYPERVISOR_VIRT_END
 #define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
 
-extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
+/*
+ * Which domains use the sparse view: PV guests and HVM guests other than the
+ * hardware domain, and the hardware domain, PV or PVH.
+ */
+extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm, opt_sparse_dmap_hwdom;
 
 /* Root of the sparse view, once built (NULL before, or if not configured). */
 extern l4_pgentry_t *sparse_dmap_root;
@@ -30,13 +34,22 @@ extern bool sparse_dmap_active;
 
 static inline bool sparse_dmap_configured(void)
 {
-    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
+    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
 }
 
-/* Whether a new domain of the given type is to use the sparse view. */
-static inline bool sparse_dmap_wanted(bool pv)
+static inline bool sparse_dmap_pv(void)    { return opt_sparse_dmap_pv; }
+static inline bool sparse_dmap_hvm(void)   { return opt_sparse_dmap_hvm; }
+static inline bool sparse_dmap_hwdom(void) { return opt_sparse_dmap_hwdom; }
+
+/*
+ * Whether a new domain of the given type is to use the sparse view.  The
+ * hardware domain, PV or PVH, has a knob of its own.
+ */
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
 {
-    return sparse_dmap_root && (pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
+    return sparse_dmap_root &&
+           (hwdom ? opt_sparse_dmap_hwdom
+                  : pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
 }
 
 void sparse_dmap_init(void);
@@ -54,7 +67,10 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
 #define sparse_dmap_active false
 
 static inline bool sparse_dmap_configured(void) { return false; }
-static inline bool sparse_dmap_wanted(bool pv) { return false; }
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
+static inline bool sparse_dmap_pv(void)    { return false; }
+static inline bool sparse_dmap_hvm(void)   { return false; }
+static inline bool sparse_dmap_hwdom(void) { return false; }
 static inline void sparse_dmap_init(void) {}
 static inline void sparse_dmap_activate(void) {}
 static inline void sparse_dmap_install(l4_pgentry_t *l4t, bool pv) {}
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index 54a6e36132..4fe160dad3 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -55,6 +55,7 @@
 
 bool __ro_after_init opt_sparse_dmap_pv;
 bool __ro_after_init opt_sparse_dmap_hvm;
+bool __ro_after_init opt_sparse_dmap_hwdom;
 
 /* Root of the sparse view: only ever walked and copied from, never loaded. */
 static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
@@ -358,7 +359,8 @@ void __init sparse_dmap_init(void)
     {
         printk(XENLOG_WARNING
                "Sparse directmap: too many early heap ranges, disabled\n");
-        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = false;
+        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = opt_sparse_dmap_hwdom =
+            false;
         return;
     }
 
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index 46105ef185..65e1d14861 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -390,6 +390,19 @@ int8_t __ro_after_init opt_xpti_domu = -1;
 
 static __init void xpti_init_default(void)
 {
+    /*
+     * XPTI and the sparse directmap view exclude each other for a PV domain
+     * (see spec_ctrl_init_domain()).  Enabled explicitly, XPTI takes
+     * precedence: the PV domains it covers stay off the sparse view.  Left to
+     * its default, it yields: it is off where the sparse view is asked for.
+     */
+    if ( !opt_dom0_pvh && opt_xpti_hwdom == 1 && sparse_dmap_hwdom() )
+        printk(XENLOG_ERR
+               "XPTI enabled, disabling Dom0 sparse-directmap\n");
+    if ( opt_xpti_domu == 1 && sparse_dmap_pv() )
+        printk(XENLOG_ERR
+               "XPTI enabled, disabling PV DomU sparse-directmap\n");
+
     if ( (boot_cpu_data.vendor & (X86_VENDOR_AMD | X86_VENDOR_HYGON)) ||
          cpu_has_rdcl_no )
     {
@@ -400,10 +413,23 @@ static __init void xpti_init_default(void)
     }
     else
     {
+        bool yield_hwdom = opt_xpti_hwdom < 0 && !opt_dom0_pvh &&
+                           sparse_dmap_hwdom();
+        bool yield_domu = opt_xpti_domu < 0 && sparse_dmap_pv();
+
+        /*
+         * The default is on here, so alert the user when it yields.
+         */
+        if ( yield_hwdom || yield_domu )
+            printk(XENLOG_WARNING
+                   "XPTI defaults off for PV %s on the sparse directmap\n",
+                   yield_hwdom ? (yield_domu ? "Dom0 and DomU" : "Dom0")
+                               : "DomU");
+
         if ( opt_xpti_hwdom < 0 )
-            opt_xpti_hwdom = 1;
+            opt_xpti_hwdom = !sparse_dmap_hwdom();
         if ( opt_xpti_domu < 0 )
-            opt_xpti_domu = 1;
+            opt_xpti_domu = !sparse_dmap_pv();
     }
 }
 
@@ -494,6 +520,33 @@ static int __init cf_check parse_pv_l1tf(const char *s)
 }
 custom_param("pv-l1tf", parse_pv_l1tf);
 
+/*
+ * The sparse directmap view, for PV guests and HVM guests other than the
+ * hardware domain, and for the hardware domain, PV or PVH.  A guest type the
+ * build lacks can't use the sparse view, and doesn't get it built.  Without
+ * CONFIG_SPARSE_DIRECTMAP there is no sparse view: the forms enabling all
+ * features, generally or for one guest type, leave it off.
+ */
+static void __init set_sparse_dmap(bool pv, bool hvm, bool hwdom, bool val)
+{
+#ifdef CONFIG_SPARSE_DIRECTMAP
+    if ( pv && IS_ENABLED(CONFIG_PV) )
+        opt_sparse_dmap_pv = val;
+    if ( hvm && IS_ENABLED(CONFIG_HVM) )
+        opt_sparse_dmap_hvm = val;
+    if ( hwdom )
+        opt_sparse_dmap_hwdom = val;
+#endif
+}
+
+/*
+ * An explicit asi=directmap on a build without the sparse directmap view.
+ * Warned about from init_speculation_mitigations(): printed while the command
+ * line is parsed, the message would stay in the console ring, and not reach
+ * the console.
+ */
+static bool __initdata asi_directmap_ignored;
+
 static int __init cf_check parse_asi(const char *s)
 {
     const char *ss;
@@ -501,7 +554,10 @@ static int __init cf_check parse_asi(const char *s)
 
     /* Interpret 'asi' alone in its positive boolean form. */
     if ( *s == '\0' )
+    {
         opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+        set_sparse_dmap(true, true, true, true);
+    }
 
     do {
         ss = strchr(s, ',');
@@ -514,11 +570,29 @@ static int __init cf_check parse_asi(const char *s)
         case 0:
         case 1:
             opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+            set_sparse_dmap(true, true, true, val);
             break;
 
         default:
-            if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+            if ( (val = parse_boolean("pv", s, ss)) >= 0 )
+                set_sparse_dmap(true, false, false, val);
+            else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+            {
                 opt_vcpu_pt_hvm = val;
+                set_sparse_dmap(false, true, false, val);
+            }
+            else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
+            {
+                opt_vcpu_pt_hwdom = val;
+                set_sparse_dmap(false, false, true, val);
+            }
+            else if ( (val = parse_boolean("directmap", s, ss)) >= 0 )
+            {
+                /* System-wide: every guest type, the hardware domain too. */
+                set_sparse_dmap(true, true, true, val);
+                asi_directmap_ignored = val &&
+                                        !IS_ENABLED(CONFIG_SPARSE_DIRECTMAP);
+            }
             else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
             {
                 switch ( val )
@@ -744,13 +818,35 @@ static void __init print_details(enum ind_thunk thunk)
            opt_pv_l1tf_domu  ? "enabled"  : "disabled");
 #endif
 
-    /* vCPU-PT is implemented for HVM only: a PV Dom0 gets none of it. */
-    printk("  ASI features for Dom0:%s\n",
-           opt_dom0_pvh && opt_vcpu_pt_hwdom ? " vCPU-PT" : " None");
+    /*
+     * vCPU-PT is implemented for HVM only, and HVM guests use the sparse
+     * directmap only with it; PV guests for which XPTI is enabled don't use
+     * it (see xpti_init_default()).  Dom0 is listed as its type makes it.
+     */
+    {
+        bool hwdom_pt = opt_dom0_pvh && opt_vcpu_pt_hwdom;
+        bool hwdom_sd = sparse_dmap_hwdom() &&
+                        (opt_dom0_pvh ? hwdom_pt : !opt_xpti_hwdom);
+
+        printk("  ASI directmap mode: %s\n",
+               sparse_dmap_configured()   ? "sparse"            : "full");
+        printk("  ASI features for Dom0:%s%s%s\n",
+               hwdom_pt || hwdom_sd       ? ""                  : " None",
+               hwdom_pt                   ? " vCPU-PT"          : "",
+               hwdom_sd                   ? " sparse-directmap" : "");
 #ifdef CONFIG_HVM
-    printk("  ASI features for HVM VMs:%s\n",
-           opt_vcpu_pt_hvm                   ? " vCPU-PT" : " None");
+        printk("  ASI features for HVM VMs:%s%s%s\n",
+               opt_vcpu_pt_hvm            ? ""                  : " None",
+               opt_vcpu_pt_hvm            ? " vCPU-PT"          : "",
+               opt_vcpu_pt_hvm && sparse_dmap_hvm()
+                                          ? " sparse-directmap" : "");
 #endif
+#ifdef CONFIG_PV
+        printk("  ASI features for PV VMs:%s\n",
+               sparse_dmap_pv() && !opt_xpti_domu
+                                          ? " sparse-directmap" : " None");
+#endif
+    }
 }
 
 static bool __init check_smt_enabled(void)
@@ -1942,11 +2038,14 @@ void spec_ctrl_init_domain(struct domain *d)
                                                     : opt_vcpu_pt_hvm);
 
     /*
-     * The sparse directmap view needs a mapcache: PV domains all have one,
-     * HVM domains through per-vCPU page-tables.
+     * The sparse directmap view needs a mapcache: PV domains all have
+     * one, HVM domains through per-vCPU page-tables.  Left to its
+     * default, XPTI is off for the domains meant for the sparse view,
+     * so this happens only where it was enabled explicitly
+     * (xpti_init_default()).
      */
-    d->arch.sparse_dmap = pv ? sparse_dmap_wanted(true)
-                             : d->arch.vcpu_pt && sparse_dmap_wanted(false);
+    d->arch.sparse_dmap = (pv ? !d->arch.pv.xpti : d->arch.vcpu_pt) &&
+                          sparse_dmap_wanted(pv, is_hardware_domain(d));
 }
 
 void __init init_speculation_mitigations(void)
@@ -2239,7 +2338,12 @@ void __init init_speculation_mitigations(void)
          hw_smt_enabled && default_xen_spec_ctrl )
         setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
 
-    if ( opt_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
+    if ( asi_directmap_ignored )
+        warning_add(
+            "asi: directmap ignored, Xen built without\n"
+            "CONFIG_SPARSE_DIRECTMAP.\n");
+
+    if ( opt_vcpu_pt_hwdom || opt_vcpu_pt_hvm || sparse_dmap_configured() )
         warning_add(
             "Address Space Isolation is not functional, this option is\n"
             "intended to be used only for development purposes.\n");
-- 
2.55.0

Re: [PATCH v3 18/18] x86/spec-ctrl: wire the sparse directmap view into the asi= option

[Roger Pau Monné]

On Wed, Oct 07, 2026 at 11:40:51AM +0100, George Dunlap wrote:
> Let asi= choose which domains run on the sparse view of the directmap,
> when the hypervisor is built with CONFIG_SPARSE_DIRECTMAP:
> 
>  * `asi` and `asi=<bool>` enable or disable the sparse view, along with
>    vCPU-PT, for every domain: HVM and PV guests, and the hardware
>    domain, PV or PVH.  HVM guests, a PVH hardware domain included, use
>    the sparse view together with vCPU-PT, which gives them the mapcache
>    it needs; PV guests all have one.  A PV guest on the sparse view
>    keeps its domain-wide mapcache and per-domain mappings: the sparse
>    view separates it from other domains, not its vCPUs from each other.
> 
>  * `asi=hvm=<bool>`, and new `asi=pv=<bool>` and `asi=dom0=<bool>`, do
>    the same for HVM or PV guests other than the hardware domain, or for
>    the hardware domain, whatever its type: `asi=1,dom0=0` leaves the
>    hardware domain out of ASI altogether.
> 
>  * A new `asi=directmap=<bool>` selects the sparse view for all domains,
>    as `asi=vcpu-pt=<bool>` selects vCPU-PT; `vcpu-pt=` alone leaves the
>    sparse view off.  On a hypervisor built without
>    CONFIG_SPARSE_DIRECTMAP the forms enabling all features leave the
>    sparse view off, and `directmap=` is ignored with a warning.  A guest
>    type the build lacks is ignored, so that the sparse view is not built
>    for nothing.
> 
>  * Only one of XPTI and the sparse view are enabled.  Enabled
>    explicitly, for dom0, domUs or both, XPTI takes precedence: the PV
>    domains it covers keep the full directmap, and the boot log says
>    so.  Left to its default, XPTI yields: it is disabled for the PV
>    domains using the sparse view, with a boot-log warning where it
>    would otherwise be enabled, as the sparse view does not replace it
>    yet (it still maps the xenheap and Xen's stacks).  Each domain's
>    case is settled at its creation.
> 
> The boot log gains a line with the directmap mode, "sparse" once a
> sparse view is configured and "full" otherwise, and the "ASI features
> for ..." lines, one for PV guests now among them, say which domains use
> the sparse view.
> 
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v3:
> - New in this version.
> ---
>  docs/misc/xen-command-line.pandoc           |  35 +++++-
>  xen/arch/x86/include/asm/sparse-directmap.h |  28 ++++-
>  xen/arch/x86/sparse-directmap.c             |   4 +-
>  xen/arch/x86/spec_ctrl.c                    | 130 ++++++++++++++++++--
>  4 files changed, 171 insertions(+), 26 deletions(-)
> 
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index 785e099c0a..0bbc8ebeff 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -203,7 +203,8 @@ to appropriate auditing by Xen.  Argo is disabled by default.
>      buggy or malicious other domain spamming the ring.
>  
>  ### asi (x86)
> -> `= List of [ <bool>, hvm=<bool>, vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> +> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, directmap=<bool>,
> +>              vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
>  
>  > Default: `false`
>  
> @@ -214,12 +215,16 @@ mitigations for certain speculative related attacks, at the cost of mapping
>  sensitive information on-demand.  It might also offer some protection against
>  unmitigated speculation-related attacks.
>  
> -The mechanisms currently implemented apply to HVM guests, including a PVH
> -hardware domain.  PV guests, including a PV hardware domain, are unaffected.
> +The mechanisms currently implemented are `vcpu-pt`, for HVM guests including
> +a PVH hardware domain, and `directmap`, for all guests including the hardware
> +domain, PV or PVH.  The plain boolean enables or disables both, wherever they
> +apply.
>  
> -* `hvm=` enables the features for HVM guests other than the hardware domain,
> -  which follows the whole-feature forms (the plain boolean, or an un-suffixed
> -  `vcpu-pt=<bool>`).
> +* `pv=`, `hvm=` and `dom0=` enable the features for PV guests or HVM guests
> +  other than the hardware domain, or for the hardware domain, PV or PVH.  The
> +  plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
> +  out.  The forms naming no guest type (an un-suffixed `vcpu-pt=<bool>`, and
> +  `directmap=<bool>`) cover the hardware domain too.

Shouldn't this generic pv, hvm, dom0 set of values go in at the same
time that `hvm=` is defined?  That would avoid having to re-write the
paragraph that's been added on this same series.

>  
>  **WARNING: manual de-selection of enabled options will invalidate any
>  protection offered by the feature.  The fine grained options provided below
> @@ -228,6 +233,18 @@ are meant to be used for debugging purposes only.**
>  * `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
>    each vCPU's monitor table maps a region private to that vCPU.
>  
> +* `directmap` (only available when Xen is built with
> +  `CONFIG_SPARSE_DIRECTMAP`) runs guest contexts on a sparse view of the
> +  directmap, which maps the memory Xen needs to reach at all times (the Xen
> +  heap, Xen's own image, firmware tables), while guest memory is mapped
> +  transiently.  This selection is system-wide; `pv=`, `hvm=` and `dom0=`
> +  make it for one guest type, along with the other features.  HVM guests use
> +  the sparse view only together with `vcpu-pt`, and PV guests only without
> +  XPTI, which yields to the sparse view unless enabled explicitly (see
> +  `xpti`).  The whole-feature forms (`asi`, `asi=<bool>`, and `pv=`, `hvm=`
> +  and `dom0=`) set this too, on builds that support it; on other builds they
> +  leave it off, and `directmap` is ignored with a warning.
> +
>  ### asid (x86)
>  > `= <boolean>`
>  
> @@ -3118,6 +3135,12 @@ Meltdown for all domains.
>  With `dom0` and `domu` it is possible to control page table isolation
>  for dom0 or guest domains only.
>  
> +With `asi`, XPTI and the sparse directmap view (`asi=directmap`) exclude each
> +other for a PV guest.  Enabled explicitly, as a whole or with `dom0` or
> +`domu`, XPTI keeps the PV guests it covers off the sparse view, and the boot
> +log says so.  Left to its default, XPTI is disabled for the PV guests using
> +the sparse view, with a boot-log warning where it would otherwise be enabled.
> +
>  ### xsave (x86)
>  > `= <boolean>`
>  
> diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
> index 41866b77e9..5349b2ba62 100644
> --- a/xen/arch/x86/include/asm/sparse-directmap.h
> +++ b/xen/arch/x86/include/asm/sparse-directmap.h
> @@ -17,7 +17,11 @@
>  #define SPARSE_DMAP_END     HYPERVISOR_VIRT_END
>  #define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
>  
> -extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm;
> +/*
> + * Which domains use the sparse view: PV guests and HVM guests other than the
> + * hardware domain, and the hardware domain, PV or PVH.
> + */
> +extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm, opt_sparse_dmap_hwdom;
>  
>  /* Root of the sparse view, once built (NULL before, or if not configured). */
>  extern l4_pgentry_t *sparse_dmap_root;
> @@ -30,13 +34,22 @@ extern bool sparse_dmap_active;
>  
>  static inline bool sparse_dmap_configured(void)
>  {
> -    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm;
> +    return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;

The opt_sparse_dmap_hwdom definition should be part of the patch that
adds opt_sparse_dmap_pv and opt_sparse_dmap_hvm (I've already
commented there I think).

>  }
>  
> -/* Whether a new domain of the given type is to use the sparse view. */
> -static inline bool sparse_dmap_wanted(bool pv)
> +static inline bool sparse_dmap_pv(void)    { return opt_sparse_dmap_pv; }
> +static inline bool sparse_dmap_hvm(void)   { return opt_sparse_dmap_hvm; }
> +static inline bool sparse_dmap_hwdom(void) { return opt_sparse_dmap_hwdom; }
> +
> +/*
> + * Whether a new domain of the given type is to use the sparse view.  The
> + * hardware domain, PV or PVH, has a knob of its own.
> + */
> +static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
>  {
> -    return sparse_dmap_root && (pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
> +    return sparse_dmap_root &&
> +           (hwdom ? opt_sparse_dmap_hwdom
> +                  : pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
>  }
>  
>  void sparse_dmap_init(void);
> @@ -54,7 +67,10 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
>  #define sparse_dmap_active false
>  
>  static inline bool sparse_dmap_configured(void) { return false; }
> -static inline bool sparse_dmap_wanted(bool pv) { return false; }
> +static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
> +static inline bool sparse_dmap_pv(void)    { return false; }
> +static inline bool sparse_dmap_hvm(void)   { return false; }
> +static inline bool sparse_dmap_hwdom(void) { return false; }
>  static inline void sparse_dmap_init(void) {}
>  static inline void sparse_dmap_activate(void) {}
>  static inline void sparse_dmap_install(l4_pgentry_t *l4t, bool pv) {}
> diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
> index 54a6e36132..4fe160dad3 100644
> --- a/xen/arch/x86/sparse-directmap.c
> +++ b/xen/arch/x86/sparse-directmap.c
> @@ -55,6 +55,7 @@
>  
>  bool __ro_after_init opt_sparse_dmap_pv;
>  bool __ro_after_init opt_sparse_dmap_hvm;
> +bool __ro_after_init opt_sparse_dmap_hwdom;
>  
>  /* Root of the sparse view: only ever walked and copied from, never loaded. */
>  static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
> @@ -358,7 +359,8 @@ void __init sparse_dmap_init(void)
>      {
>          printk(XENLOG_WARNING
>                 "Sparse directmap: too many early heap ranges, disabled\n");
> -        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = false;
> +        opt_sparse_dmap_pv = opt_sparse_dmap_hvm = opt_sparse_dmap_hwdom =
> +            false;
>          return;
>      }
>  
> diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
> index 46105ef185..65e1d14861 100644
> --- a/xen/arch/x86/spec_ctrl.c
> +++ b/xen/arch/x86/spec_ctrl.c
> @@ -390,6 +390,19 @@ int8_t __ro_after_init opt_xpti_domu = -1;
>  
>  static __init void xpti_init_default(void)
>  {
> +    /*
> +     * XPTI and the sparse directmap view exclude each other for a PV domain
> +     * (see spec_ctrl_init_domain()).  Enabled explicitly, XPTI takes
> +     * precedence: the PV domains it covers stay off the sparse view.  Left to
> +     * its default, it yields: it is off where the sparse view is asked for.
> +     */
> +    if ( !opt_dom0_pvh && opt_xpti_hwdom == 1 && sparse_dmap_hwdom() )
> +        printk(XENLOG_ERR
> +               "XPTI enabled, disabling Dom0 sparse-directmap\n");
> +    if ( opt_xpti_domu == 1 && sparse_dmap_pv() )
> +        printk(XENLOG_ERR
> +               "XPTI enabled, disabling PV DomU sparse-directmap\n");
> +
>      if ( (boot_cpu_data.vendor & (X86_VENDOR_AMD | X86_VENDOR_HYGON)) ||
>           cpu_has_rdcl_no )
>      {
> @@ -400,10 +413,23 @@ static __init void xpti_init_default(void)
>      }
>      else
>      {
> +        bool yield_hwdom = opt_xpti_hwdom < 0 && !opt_dom0_pvh &&
> +                           sparse_dmap_hwdom();
> +        bool yield_domu = opt_xpti_domu < 0 && sparse_dmap_pv();
> +
> +        /*
> +         * The default is on here, so alert the user when it yields.
> +         */
> +        if ( yield_hwdom || yield_domu )
> +            printk(XENLOG_WARNING
> +                   "XPTI defaults off for PV %s on the sparse directmap\n",
> +                   yield_hwdom ? (yield_domu ? "Dom0 and DomU" : "Dom0")
> +                               : "DomU");
> +
>          if ( opt_xpti_hwdom < 0 )
> -            opt_xpti_hwdom = 1;
> +            opt_xpti_hwdom = !sparse_dmap_hwdom();
>          if ( opt_xpti_domu < 0 )
> -            opt_xpti_domu = 1;
> +            opt_xpti_domu = !sparse_dmap_pv();
>      }
>  }
>  
> @@ -494,6 +520,33 @@ static int __init cf_check parse_pv_l1tf(const char *s)
>  }
>  custom_param("pv-l1tf", parse_pv_l1tf);
>  
> +/*
> + * The sparse directmap view, for PV guests and HVM guests other than the
> + * hardware domain, and for the hardware domain, PV or PVH.  A guest type the
> + * build lacks can't use the sparse view, and doesn't get it built.  Without
> + * CONFIG_SPARSE_DIRECTMAP there is no sparse view: the forms enabling all
> + * features, generally or for one guest type, leave it off.
> + */
> +static void __init set_sparse_dmap(bool pv, bool hvm, bool hwdom, bool val)
> +{
> +#ifdef CONFIG_SPARSE_DIRECTMAP
> +    if ( pv && IS_ENABLED(CONFIG_PV) )
> +        opt_sparse_dmap_pv = val;
> +    if ( hvm && IS_ENABLED(CONFIG_HVM) )
> +        opt_sparse_dmap_hvm = val;
> +    if ( hwdom )
> +        opt_sparse_dmap_hwdom = val;
> +#endif
> +}
> +
> +/*
> + * An explicit asi=directmap on a build without the sparse directmap view.
> + * Warned about from init_speculation_mitigations(): printed while the command
> + * line is parsed, the message would stay in the console ring, and not reach
> + * the console.
> + */
> +static bool __initdata asi_directmap_ignored;
> +
>  static int __init cf_check parse_asi(const char *s)
>  {
>      const char *ss;
> @@ -501,7 +554,10 @@ static int __init cf_check parse_asi(const char *s)
>  
>      /* Interpret 'asi' alone in its positive boolean form. */
>      if ( *s == '\0' )
> +    {
>          opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
> +        set_sparse_dmap(true, true, true, true);
> +    }
>  
>      do {
>          ss = strchr(s, ',');
> @@ -514,11 +570,29 @@ static int __init cf_check parse_asi(const char *s)
>          case 0:
>          case 1:
>              opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> +            set_sparse_dmap(true, true, true, val);
>              break;
>  
>          default:
> -            if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> +            if ( (val = parse_boolean("pv", s, ss)) >= 0 )
> +                set_sparse_dmap(true, false, false, val);
> +            else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> +            {
>                  opt_vcpu_pt_hvm = val;
> +                set_sparse_dmap(false, true, false, val);
> +            }
> +            else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
> +            {
> +                opt_vcpu_pt_hwdom = val;
> +                set_sparse_dmap(false, false, true, val);
> +            }
> +            else if ( (val = parse_boolean("directmap", s, ss)) >= 0 )
> +            {
> +                /* System-wide: every guest type, the hardware domain too. */
> +                set_sparse_dmap(true, true, true, val);
> +                asi_directmap_ignored = val &&
> +                                        !IS_ENABLED(CONFIG_SPARSE_DIRECTMAP);

I think we should return error here, rather than print a warning: we
don't do this for any other command line option that's not parsed
because the required support has been compiled out.

> +            }
>              else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
>              {
>                  switch ( val )
> @@ -744,13 +818,35 @@ static void __init print_details(enum ind_thunk thunk)
>             opt_pv_l1tf_domu  ? "enabled"  : "disabled");
>  #endif
>  
> -    /* vCPU-PT is implemented for HVM only: a PV Dom0 gets none of it. */
> -    printk("  ASI features for Dom0:%s\n",
> -           opt_dom0_pvh && opt_vcpu_pt_hwdom ? " vCPU-PT" : " None");
> +    /*
> +     * vCPU-PT is implemented for HVM only, and HVM guests use the sparse
> +     * directmap only with it; PV guests for which XPTI is enabled don't use
> +     * it (see xpti_init_default()).  Dom0 is listed as its type makes it.

I'm not sure this comment is very useful, and it will keep growing as
more features are added.  The set of available options is clearly seen
in the logic below, and I don't think it deserves the extra comment.

> +     */
> +    {
> +        bool hwdom_pt = opt_dom0_pvh && opt_vcpu_pt_hwdom;
> +        bool hwdom_sd = sparse_dmap_hwdom() &&
> +                        (opt_dom0_pvh ? hwdom_pt : !opt_xpti_hwdom);
> +
> +        printk("  ASI directmap mode: %s\n",
> +               sparse_dmap_configured()   ? "sparse"            : "full");
> +        printk("  ASI features for Dom0:%s%s%s\n",
> +               hwdom_pt || hwdom_sd       ? ""                  : " None",
> +               hwdom_pt                   ? " vCPU-PT"          : "",
> +               hwdom_sd                   ? " sparse-directmap" : "");
>  #ifdef CONFIG_HVM
> -    printk("  ASI features for HVM VMs:%s\n",
> -           opt_vcpu_pt_hvm                   ? " vCPU-PT" : " None");
> +        printk("  ASI features for HVM VMs:%s%s%s\n",
> +               opt_vcpu_pt_hvm            ? ""                  : " None",
> +               opt_vcpu_pt_hvm            ? " vCPU-PT"          : "",
> +               opt_vcpu_pt_hvm && sparse_dmap_hvm()
> +                                          ? " sparse-directmap" : "");
>  #endif
> +#ifdef CONFIG_PV
> +        printk("  ASI features for PV VMs:%s\n",
> +               sparse_dmap_pv() && !opt_xpti_domu
> +                                          ? " sparse-directmap" : " None");
> +#endif
> +    }
>  }
>  
>  static bool __init check_smt_enabled(void)
> @@ -1942,11 +2038,14 @@ void spec_ctrl_init_domain(struct domain *d)
>                                                      : opt_vcpu_pt_hvm);
>  
>      /*
> -     * The sparse directmap view needs a mapcache: PV domains all have one,
> -     * HVM domains through per-vCPU page-tables.
> +     * The sparse directmap view needs a mapcache: PV domains all have
> +     * one, HVM domains through per-vCPU page-tables.   Left to its

In principle HVM domains could also use the per-domain mapcache, I
don't think we should tie the sparse dmap to HVM domains use per-vCPU
mapcaches.  The point of having the fine-grained options is to enable
them on such fine grained basis.

Thanks, Roger.